Skip to content

Add Python repl_prepare runtime preparation#135

Draft
t-kalinowski wants to merge 20 commits into
mainfrom
add-tool-repl_prepare
Draft

Add Python repl_prepare runtime preparation#135
t-kalinowski wants to merge 20 commits into
mainfrom
add-tool-repl_prepare

Conversation

@t-kalinowski

@t-kalinowski t-kalinowski commented Jun 21, 2026

Copy link
Copy Markdown
Member

Summary

Adds a Python-only repl_prepare tool for cases where a caller wants to change, select, or preflight the Python environment before later repl calls. The Python REPL itself still starts ready for evaluation; repl_prepare is for environment preparation, not for making repl usable.

Tool schema

repl_prepare is advertised only by Python servers when uv is available. R servers continue to advertise repl_reset; Python servers no longer advertise repl_reset.

repl_prepare({
  requirements?: {
    packages?: string[];
    python_version?: string;
    action?: "add" | "remove" | "set";      // default: "add"
    restart?: "if_needed" | "no" | "yes";   // default: "if_needed"
  };
  python?: {
    executable?: string; // absolute path
    venv?: string;       // absolute path to a virtual environment
  };
})

Contract details:

  • Do not call repl_prepare just to make repl available; the REPL starts ready.
  • Use requirements when the caller needs managed packages or a managed Python version before evaluation.
  • Use python when the caller needs an existing interpreter or virtual environment.
  • Omit all arguments only to realize the current managed requirements manifest without changing it.
  • The managed manifest starts as packages=["numpy"] and python_version=null.
  • Provide either requirements or python, not both.
  • Inside python, provide exactly one of executable or venv.
  • requirements.action updates the persistent managed manifest with add, remove, or set.
  • requirements.restart controls whether preparation may restart the active Python session.
  • requirements.python_version is forwarded to uv as supplied.
  • Explicit python selections use the requested interpreter or venv as-is and do not mutate the managed manifest.
  • The tool does not execute user code, import packages, attach modules, or define aliases.
  • There is no timeout_ms argument.

Scenario examples

The snippets below use pseudo tool-call syntax. Each repl(...) call is a later Python REPL evaluation after repl_prepare(...) returns.

Add a package before importing it:

repl_prepare(requirements = {packages: ["pandas"]})

repl("""
import pandas as pd
print(pd.DataFrame({"x": [1, 2]}).shape)
""")
# (2, 1)

Build up a persistent managed manifest:

repl_prepare(requirements = {packages: ["pandas"]})
repl_prepare(requirements = {packages: ["plotnine"]})

repl("""
import pandas
import plotnine
print("ready")
""")
# ready

Replace the whole managed manifest and request a Python version constraint:

repl_prepare(requirements = {
  packages: ["numpy", "pandas"],
  python_version: ">=3.11,<3.13",
  action: "set"
})

repl("""
import sys
import numpy
import pandas
print(sys.version_info >= (3, 11))
""")
# True

Preserve existing session state if preparation would require a restart:

repl("_marker = 'kept'")

repl_prepare(requirements = {
  packages: ["plotnine"],
  restart: "no"
})
# returns an error if satisfying the request requires a restart
# session unchanged; no user state discarded

repl("print(_marker)")
# kept

Force a fresh session for the current managed manifest:

repl("_marker = 'discarded'")

repl_prepare(requirements = {restart: "yes"})
# session restarted; user state discarded

repl("print('_marker' in globals())")
# False

Switch to an existing project virtual environment:

repl_prepare(python = {venv: "/absolute/path/to/project/.venv"})

repl("""
import sys
print(sys.prefix)
""")
# /absolute/path/to/project/.venv

Switch to an explicit Python executable:

repl_prepare(python = {executable: "/absolute/path/to/python"})

repl("""
import sys
print(sys.executable)
""")
# /absolute/path/to/python

Remove a package from the managed manifest before returning to that managed environment:

repl_prepare(requirements = {packages: ["plotnine"], action: "remove"})

repl("""
import importlib.util
print(importlib.util.find_spec("plotnine") is None)
""")
# True

Reset Python without environment preparation:

repl("<Ctrl-D>")
# new session started

R keeps the existing reset tool:

repl_reset()
# new session started

Internal changes

  • Adds Python runtime preparation and requirement resolution through uv.
  • Propagates prepared Python executable and module search-path overrides into embedded Python workers.
  • Adds worker replacement support and lightweight user-state tracking for restart decisions.
  • Splits server tool surfaces into R reset, Python prepare, and Python repl-only variants.
  • Adds public MCP coverage for tool listing, schema validation, manifest updates, restart policy behavior, active-work replacement, and explicit Python selection.

Diff composition

Measured against the merge base with main, this PR is 2079 insertions and 107 deletions across 22 files.

  • runtime src/: +1058/-95 (52.7% of churn)
  • inline tests inside src/: +3/-0 (0.1% of churn)
  • tests in tests/: +924/-8 (42.6% of churn)
  • docs: +94/-4 (4.5% of churn)

Largest files:

  • tests/python_repl_prepare.rs: +904/-0
  • src/server.rs: +387/-49
  • src/python_prepare.rs: +399/-0
  • src/worker_supervisor.rs: +88/-25
  • src/python_runtime.rs: +71/-17

@t-kalinowski t-kalinowski changed the title Add repl_prepare for Python session initialization Add Python repl_prepare runtime preparation Jun 23, 2026
Explain that Python repl is self-starting and that repl_prepare should not be called preemptively.
Address Codex review findings for Python repl_prepare.

Finding 1:
<<<FINDING 1 START>>>
[P1] Pass sandbox metadata into prepare restarts — /Users/tomasz/.codex/worktrees/92b2/mcp-repl/src/server.rs:948-951: When the server runs with `--sandbox inherit`, `repl_prepare` receives call metadata but drops it before replacing the Python worker. Prepares that need to spawn or replace the worker can fail with missing sandbox metadata or restart under stale policy; handle `meta` the same way as `repl`/`repl_reset` before replacing the worker.
<<<FINDING 1 END>>>

Finding 2:
<<<FINDING 2 START>>>
[P2] Check full requirement specifiers before preserving Python — /Users/tomasz/.codex/worktrees/92b2/mcp-repl/src/python_prepare.rs:360-363: The probe strips requirements to distribution names and only checks metadata exists. Requirements like `numpy>=2`, extras, or direct references can be treated as satisfied when they are not, allowing `repl_prepare` to commit an unsatisfied environment without invoking uv; validate full requirements with packaging/uv or only short-circuit bare names.
<<<FINDING 2 END>>>

Finding 3:
<<<FINDING 3 START>>>
[P2] Clear stale timeout bundles after prepare replacement — /Users/tomasz/.codex/worktrees/92b2/mcp-repl/src/server.rs:552-555: When `repl_prepare` replaces the worker while a request is timed out, worker state is cleared but `ResponseState` active/staged timeout output remains. The next normal `repl` call can be finalized as a timeout follow-up and reuse or append to the discarded request's bundle; retire or clear the timeout bundle when prepare discards pending work.
<<<FINDING 3 END>>>

Finding 4:
<<<FINDING 4 START>>>
[P2] Reset user-state tracking in files mode — /Users/tomasz/.codex/worktrees/92b2/mcp-repl/src/worker_process/output_state.rs:237: `user_state_may_exist` is cleared only on the pager reset path. Files-mode resets/replacements leave the flag set after the old Python process is discarded, so later `repl_prepare` with `restart: "no"` can be rejected as if it would discard user state even when the fresh session has not run user code; apply the same reset in files mode.
<<<FINDING 4 END>>>
[P1] Stage sandbox metadata before probing executables — /Users/tomasz/.codex/worktrees/92b2/mcp-repl/src/server.rs:597-599. When the server runs with `--sandbox inherit`, it resolves the requested Python before `stage_prepare_sandbox_state` is called, so a `repl_prepare` call with missing or invalid sandbox metadata still executes the supplied executable on the host via `query_python_runtime_config` before being rejected. This bypasses the inherited-metadata gate used by the other tool paths; validate/stage the sandbox state before running the probe.

[P2] Avoid treating supersets as satisfying removals — /Users/tomasz/.codex/worktrees/92b2/mcp-repl/src/python_prepare.rs:220. When the current/default interpreter already has an extra package that was just removed or omitted by `action: "set"`, this shortcut returns the current interpreter because it satisfies the remaining packages. `active_matches` then leaves the session unchanged while the result reports the package removed, even though it remains importable; narrowed manifests need to force a managed uv target instead of accepting any superset.
[P1] Validate sandbox metadata before invoking uv — /Users/tomasz/.codex/worktrees/92b2/mcp-repl/src/server.rs:519-522
Under `--sandbox inherit`, requirements prepares should fail closed before side effects when `_meta["codex/sandbox-state-meta"]` is missing or malformed. The patch resolves the manifest first, which can spawn `uv tool run --with ...` for user-supplied requirements before `sandbox_state_update()` is called, so a rejected tool call can still perform package resolution/downloads outside the accepted sandbox state. Stage or validate the sandbox update before calling the requirements resolver, as the explicit Python path already does.

[P2] Check the active Python before falling back to uv — /Users/tomasz/.codex/worktrees/92b2/mcp-repl/src/python_prepare.rs:233-234
After `repl_prepare` has selected an explicit venv or executable, the active session can satisfy a later `requirements` add/no-op even when the original server Python does not. This helper probes `resolve_python_runtime_config()` from the server environment, not the worker launch, so `restart: "no"` can fail or `if_needed` can restart and discard state even though the current venv already has the manifest packages. Use the active worker executable for the shortcut, or skip the shortcut when it is unavailable.
[P1] Run explicit Python probes inside the sandbox — /Users/tomasz/.codex/worktrees/92b2/mcp-repl/src/python_prepare.rs:136-138
When `repl_prepare` receives `python.executable`, this path runs the caller-supplied absolute executable in the server process via `query_python_runtime_config` before the sandboxed worker is spawned. Under `read-only`, `workspace-write`, or `inherit` sandbox modes, a caller can point this at any executable that accepts or ignores the probe args and execute it with server permissions, bypassing the worker sandbox. Probe the interpreter under the effective worker sandbox or otherwise avoid executing arbitrary paths in the parent process.

[P1] Run uv resolution under the sandbox policy — /Users/tomasz/.codex/worktrees/92b2/mcp-repl/src/python_prepare.rs:340-341
For `requirements` requests, this launches `uv` directly from the server process, so a package request can access the network, write uv caches, and potentially run package build code outside the configured worker sandbox and managed-network policy. This affects restricted `read-only`/`workspace-write`/`inherit` sessions where `repl` code would otherwise be contained. Resolve/install requirements through a sandboxed helper or explicitly enforce the same filesystem and network policy here.
Finding addressed:

[P2] Track auto-selected Python executable for prepare matching — /Users/tomasz/.codex/worktrees/92b2/mcp-repl/src/worker_process.rs:306-309
When the Python server is started normally without `MCP_REPL_PYTHON_EXECUTABLE`, the worker still has an active auto-selected `sys.executable`, but this returns `None` for `WorkerLaunch::Builtin`. As a result, `repl_prepare` cannot detect that `python.executable` points at the current interpreter, so it restarts and discards user state even though the session already matches; the same missing hint also prevents requirements shortcuts for an already-satisfying auto-selected runtime.

Response:
Built-in Python workers now report their resolved runtime executable in worker_ready metadata, and WorkerManager stores that live hint for repl_prepare executable matching. A public repl_prepare regression covers the auto-selected Python launch path.
[P2] Preserve Python prefixes for prepared workers — /Users/tomasz/.codex/worktrees/92b2/mcp-repl/src/python_ffi.rs:433-433
When `repl_prepare` restarts into a prepared Python, `module_search_paths` is non-empty and this calls `Py_SetPath`, which bypasses CPython's normal path/prefix calculation; after a default prepare, `sys.prefix` and `sys.exec_prefix` are empty. Code and packages that use `sys.prefix`/`sysconfig` to locate the active environment will see a broken Python environment, so the prepared worker should initialize with config/home/prefix information instead of only overriding `sys.path`.
Finding 1:
[P2] Persist prepared Python target on no-op prepares — /Users/tomasz/.codex/worktrees/92b2/mcp-repl/src/server.rs:598-604
When the resolved target already matches the live process, this commits the manifest but leaves `worker_launch` as the previous builtin auto-discovery. If a later Ctrl-D reset or sandbox-cwd change respawns the worker, it can discover a different Python and lose the prepared manifest even though `repl_prepare` reported success; the explicit-python `active_matches` branch has the same issue. Record the resolved `PythonExecutable` target for future spawns even when reusing the current session.

Finding 2:
[P2] Avoid concatenating bare Python version requests — /Users/tomasz/.codex/worktrees/92b2/mcp-repl/src/python_prepare.rs:162-164
When a user prepares one Python version and later asks for another with the default `action: "add"`, this builds a value like `3.11,3.12` and passes it to `uv --python`. `uv` treats that as a single executable name/invalid request, so the second prepare fails instead of switching versions while keeping the package manifest; store or require replacing the version request rather than appending bare versions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant