Multiple interpreters in a Python process¶

ほとんどの場合は埋め込む Python インタプリタは1つだけですが、いくつかの場合に同一プロセス内、あるいは同一スレッド内で、複数の独立したインタプリタを作成する必要があります。これを可能にするのがサブインタプリタです。

"メイン" インタプリタとは、ランタイムが初期化を行ったときに最初に作成されたインタプリタのことです。サブインタプリタと違い、メインインタプリタにはシグナルハンドリングのような、プロセス全域で唯一な責務があります。メインインタプリタにはランタイムの初期化中の処理実行という責務もあり、通常はランタイムの終了処理中に動いているランタイムでもあります。 PyInterpreterState_Main() 関数は、メインインタプリタの状態へのポインタを返します。

サブインタプリタを切り替えが PyThreadState_Swap() 関数でできます。次の関数を使ってサブインタプリタの作成と削除が行えます:

type PyInterpreterConfig¶

Structure containing most parameters to configure a sub-interpreter. Its values are used only in Py_NewInterpreterFromConfig() and never modified by the runtime.

Added in version 3.12.

構造体フィールド:

int use_main_obmalloc¶

If this is 0 then the sub-interpreter will use its own "object" allocator state. Otherwise it will use (share) the main interpreter's.

If this is 0 then check_multi_interp_extensions must be 1 (non-zero). If this is 1 then gil must not be PyInterpreterConfig_OWN_GIL.

int allow_fork¶

If this is 0 then the runtime will not support forking the process in any thread where the sub-interpreter is currently active. Otherwise fork is unrestricted.

Note that the subprocess module still works when fork is disallowed.

int allow_exec¶

If this is 0 then the runtime will not support replacing the current process via exec (e.g. os.execv()) in any thread where the sub-interpreter is currently active. Otherwise exec is unrestricted.

Note that the subprocess module still works when exec is disallowed.

int allow_threads¶: If this is 0 then the sub-interpreter's threading module won't create threads. Otherwise threads are allowed.

int allow_daemon_threads¶: If this is 0 then the sub-interpreter's threading module won't create daemon threads. Otherwise daemon threads are allowed (as long as allow_threads is non-zero).

int check_multi_interp_extensions¶

If this is 0 then all extension modules may be imported, including legacy (single-phase init) modules, in any thread where the sub-interpreter is currently active. Otherwise only multi-phase init extension modules (see PEP 489) may be imported. (Also see Py_mod_multiple_interpreters.)

This must be 1 (non-zero) if use_main_obmalloc is 0.

int gil¶

This determines the operation of the GIL for the sub-interpreter. It may be one of the following:

PyInterpreterConfig_DEFAULT_GIL¶: Use the default selection (PyInterpreterConfig_SHARED_GIL).

PyInterpreterConfig_SHARED_GIL¶: Use (share) the main interpreter's GIL.

PyInterpreterConfig_OWN_GIL¶: Use the sub-interpreter's own GIL.

If this is PyInterpreterConfig_OWN_GIL then PyInterpreterConfig.use_main_obmalloc must be 0.

PyStatus Py_NewInterpreterFromConfig(PyThreadState **tstate_p, const PyInterpreterConfig *config)¶

新しいサブインタプリタ (sub-interpreter) を生成します。サブインタプリタとは、(ほぼ完全に) 個別に分割された Python コードの実行環境です。特に、新しいサブインタプリタは、 import されるモジュール全てについて個別のバージョンを持ち、これには基盤となるモジュール builtins, __main__ および sys も含まれます。ロード済みのモジュールからなるテーブル (sys.modules) およびモジュール検索パス (sys.path) もサブインタプリタ毎に別個のものになります。新たなサブインタプリタ環境には sys.argv 変数がありません。また、サブインタプリタは新たな標準 I/O ストリーム sys.stdin, sys.stdout, sys.stderr を持ちます (とはいえ、これらのストリームは根底にある同じファイル記述子を参照しています)。

The given config controls the options with which the interpreter is initialized.

Upon success, tstate_p will be set to the first thread state created in the new sub-interpreter. This thread state is attached. Note that no actual thread is created; see the discussion of thread states below. If creation of the new interpreter is unsuccessful, tstate_p is set to NULL; no exception is set since the exception state is stored in the attached thread state, which might not exist.

Like all other Python/C API functions, an attached thread state must be present before calling this function, but it might be detached upon returning. On success, the returned thread state will be attached. If the sub-interpreter is created with its own GIL then the attached thread state of the calling interpreter will be detached. When the function returns, the new interpreter's thread state will be attached to the current thread and the previous interpreter's attached thread state will remain detached.

Added in version 3.12.

Sub-interpreters are most effective when isolated from each other, with certain functionality restricted:

PyInterpreterConfig config = {
    .use_main_obmalloc = 0,
    .allow_fork = 0,
    .allow_exec = 0,
    .allow_threads = 1,
    .allow_daemon_threads = 0,
    .check_multi_interp_extensions = 1,
    .gil = PyInterpreterConfig_OWN_GIL,
};
PyThreadState *tstate = NULL;
PyStatus status = Py_NewInterpreterFromConfig(&tstate, &config);
if (PyStatus_Exception(status)) {
    Py_ExitStatusException(status);
}

Note that the config is used only briefly and does not get modified. During initialization the config's values are converted into various PyInterpreterState values. A read-only copy of the config may be stored internally on the PyInterpreterState.

Extension modules are shared between (sub-)interpreters as follows:

For modules using multi-phase initialization, e.g. PyModule_FromDefAndSpec(), a separate module object is created and initialized for each interpreter. Only C-level static and global variables are shared between these module objects.
For modules using legacy single-phase initialization, e.g. PyModule_Create(), the first time a particular extension is imported, it is initialized normally, and a (shallow) copy of its module's dictionary is squirreled away. When the same extension is imported by another (sub-)interpreter, a new module is initialized and filled with the contents of this copy; the extension's init function is not called. Objects in the module's dictionary thus end up shared across (sub-)interpreters, which might cause unwanted behavior (see Bugs and caveats below).

Note that this is different from what happens when an extension is imported after the interpreter has been completely re-initialized by calling Py_FinalizeEx() and Py_Initialize(); in that case, the extension's initmodule function is called again. As with multi-phase initialization, this means that only C-level static and global variables are shared between these modules.

PyThreadState *Py_NewInterpreter(void)¶: 次に属します: Stable ABI.
Create a new sub-interpreter. This is essentially just a wrapper around Py_NewInterpreterFromConfig() with a config that preserves the existing behavior. The result is an unisolated sub-interpreter that shares the main interpreter's GIL, allows fork/exec, allows daemon threads, and allows single-phase init modules.

void Py_EndInterpreter(PyThreadState *tstate)¶

次に属します: Stable ABI.

Destroy the (sub-)interpreter represented by the given thread state. The given thread state must be attached. When the call returns, there will be no attached thread state. All thread states associated with this interpreter are destroyed.

Py_FinalizeEx() will destroy all sub-interpreters that haven't been explicitly destroyed at that point.

A per-interpreter GIL¶

Added in version 3.12.

Using Py_NewInterpreterFromConfig() you can create a sub-interpreter that is completely isolated from other interpreters, including having its own GIL. The most important benefit of this isolation is that such an interpreter can execute Python code without being blocked by other interpreters or blocking any others. Thus a single Python process can truly take advantage of multiple CPU cores when running Python code. The isolation also encourages a different approach to concurrency than that of just using threads. (See PEP 554 and PEP 684.)

Using an isolated interpreter requires vigilance in preserving that isolation. That especially means not sharing any objects or mutable state without guarantees about thread-safety. Even objects that are otherwise immutable (e.g. None, (1, 5)) can't normally be shared because of the refcount. One simple but less-efficient approach around this is to use a global lock around all use of some state (or object). Alternately, effectively immutable objects (like integers or strings) can be made safe in spite of their refcounts by making them immortal. In fact, this has been done for the builtin singletons, small integers, and a number of other builtin objects.

If you preserve isolation then you will have access to proper multi-core computing without the complications that come with free-threading. Failure to preserve isolation will expose you to the full consequences of free-threading, including races and hard-to-debug crashes.

Aside from that, one of the main challenges of using multiple isolated interpreters is how to communicate between them safely (not break isolation) and efficiently. The runtime and stdlib do not provide any standard approach to this yet. A future stdlib module would help mitigate the effort of preserving isolation and expose effective tools for communicating (and sharing) data between interpreters.

バグと注意事項¶

Because sub-interpreters (and the main interpreter) are part of the same process, the insulation between them isn't perfect --- for example, using low-level file operations like os.close() they can (accidentally or maliciously) affect each other's open files. Because of the way extensions are shared between (sub-)interpreters, some extensions may not work properly; this is especially likely when using single-phase initialization or (static) global variables. It is possible to insert objects created in one sub-interpreter into a namespace of another (sub-)interpreter; this should be avoided if possible.

Special care should be taken to avoid sharing user-defined functions, methods, instances or classes between sub-interpreters, since import operations executed by such objects may affect the wrong (sub-)interpreter's dictionary of loaded modules. It is equally important to avoid sharing objects from which the above are reachable.

サブインタプリタを PyGILState_* API と組み合わせるのが難しいことにも注意してください。これらのAPIはPythonのスレッド状態とOSレベルスレッドが1対1で対応していることを前提にしていて、サブインタプリタが存在するとその前提が崩れるからです。対応する PyGILState_Ensure() と PyGILState_Release() の呼び出しのペアの間では、サブインタプリタの切り替えを行わないことを強く推奨します。さらに、(ctypes のような)これらのAPIを使ってPythonの外で作られたスレッドから Pythonコードを実行している拡張モジュールはサブインタプリタを使うと壊れる可能性があります。

高水準 API¶

type PyInterpreterState¶

次に属します: Limited API (不透明な構造体として).

このデータ構造体は、協調動作する多数のスレッド間で共有されている状態を表現します。同じインタプリタに属するスレッドはモジュール管理情報やその他いくつかの内部的な情報を共有しています。この構造体には公開 (public) のメンバはありません。

異なるインタプリタに属するスレッド間では、利用可能なメモリ、開かれているファイルデスクリプタなどといったプロセス状態を除いて、初期状態では何も共有されていません。GILもまた、スレッドがどのインタプリタに属しているかに関わらずすべてのスレッドで共有されています。

バージョン 3.12 で変更: PEP 684 introduced the possibility of a per-interpreter GIL. See Py_NewInterpreterFromConfig().

PyInterpreterState *PyInterpreterState_Get(void)¶

次に属します: Stable ABI (バージョン 3.9 より).

Get the current interpreter.

Issue a fatal error if there is no attached thread state. It cannot return NULL.

Added in version 3.9.

int64_t PyInterpreterState_GetID(PyInterpreterState *interp)¶

次に属します: Stable ABI (バージョン 3.7 より).

インタプリタの一意な ID を返します。処理中に何かエラーが起きたら、 -1 が返され、エラーがセットされます。

The caller must have an attached thread state.

Added in version 3.7.

PyObject *PyInterpreterState_GetDict(PyInterpreterState *interp)¶

戻り値: 借用参照。 次に属します: Stable ABI (バージョン 3.8 より).

インタプリタ固有のデータを保持している辞書を返します。この関数が NULL を返した場合は、ここまでで例外は送出されておらず、呼び出し側はインタプリタ固有の辞書は利用できないと考えなければなりません。

この関数は PyModule_GetState() を置き換えるものではなく、拡張モジュールがインタプリタ固有の状態情報を格納するのに使うべきものです。

The returned dictionary is borrowed from the interpreter and is valid until interpreter shutdown.

Added in version 3.8.

typedef PyObject *(*_PyFrameEvalFunction)(PyThreadState *tstate, _PyInterpreterFrame *frame, int throwflag)¶

Type of a frame evaluation function.

The throwflag parameter is used by the throw() method of generators: if non-zero, handle the current exception.

バージョン 3.9 で変更: The function now takes a tstate parameter.

バージョン 3.11 で変更: The frame parameter changed from PyFrameObject* to _PyInterpreterFrame*.

_PyFrameEvalFunction _PyInterpreterState_GetEvalFrameFunc(PyInterpreterState *interp)¶

Get the frame evaluation function.

See the PEP 523 "Adding a frame evaluation API to CPython".

Added in version 3.9.

void _PyInterpreterState_SetEvalFrameFunc(PyInterpreterState *interp, _PyFrameEvalFunction eval_frame)¶

Set the frame evaluation function.

See the PEP 523 "Adding a frame evaluation API to CPython".

Added in version 3.9.

低水準 API¶

次の全ての関数は Py_Initialize() の後に呼び出さなければなりません。

バージョン 3.7 で変更: Py_Initialize() now initializes the GIL and sets an attached thread state.

PyInterpreterState *PyInterpreterState_New()¶

次に属します: Stable ABI.

Create a new interpreter state object. An attached thread state is not needed, but may optionally exist if it is necessary to serialize calls to this function.

引数無しで監査イベント cpython.PyInterpreterState_New を送出します。

void PyInterpreterState_Clear(PyInterpreterState *interp)¶

次に属します: Stable ABI.

Reset all information in an interpreter state object. There must be an attached thread state for the interpreter.

引数無しで監査イベント cpython.PyInterpreterState_Clear を送出します。

void PyInterpreterState_Delete(PyInterpreterState *interp)¶: 次に属します: Stable ABI.
Destroy an interpreter state object. There should not be an attached thread state for the target interpreter. The interpreter state must have been reset with a previous call to PyInterpreterState_Clear().

Advanced debugger support¶

以下の関数は高度なデバッグツールでの使用のためだけのものです。

PyInterpreterState *PyInterpreterState_Head()¶: インタプリタ状態オブジェクトからなるリストのうち、先頭にあるものを返します。

PyInterpreterState *PyInterpreterState_Main()¶: メインインタプリタの状態オブジェクトを返します。

PyInterpreterState *PyInterpreterState_Next(PyInterpreterState *interp)¶: インタプリタ状態オブジェクトからなるリストのうち、interp の次にあるものを返します。

PyThreadState *PyInterpreterState_ThreadHead(PyInterpreterState *interp)¶: インタプリタ interp に関連付けられているスレッドからなるリストのうち、先頭にある PyThreadState オブジェクトを返します。

PyThreadState *PyThreadState_Next(PyThreadState *tstate)¶: tstate と同じ PyInterpreterState オブジェクトに属しているスレッド状態オブジェクトのうち、 tstate の次にあるものを返します。

Multiple interpreters in a Python process¶

A per-interpreter GIL¶

バグと注意事項¶

高水準 API¶

低水準 API¶

Advanced debugger support¶

目次

前のトピックへ

次のトピックへ

This page