Databricks RLM REPL

Execute Python on a Databricks cluster through a stateful REPL that follows the Recursive Language Model (RLM) pattern. The root LM (Claude Code) never sees auth tokens, connection boilerplate, raw polling, or large dataframes — it interacts with Databricks through a clean CLI that returns JSON metadata and file paths.

Design Principles

Constraints

• Classic all-purpose clusters only — the Command Execution API requires them
• Max 145 execution contexts per cluster — destroy unused sessions to free slots
• Idle contexts evicted permanently — the wrapper detects eviction and Claude replays steps
• Use %pip install for packages — %pip works in execution contexts; %%pip (double percent) does not
• Sub-LM calls are blocking — sub_llm_batch() provides parallelism within a single command

When Invoked

Arguments: $0 = Databricks CLI profile from ~/.databrickscfg, $1 = cluster name (or substring).

1. Resolve the cluster ID from the cluster name:

   python3 .claude/skills/databricks-repl/find_cluster.py --profile $0 --name "$1"
   

   Returns JSON with cluster_id, cluster_name, state. If multiple matches, ask the user to pick. If the cluster is not running, inform the user.

2. Check for an active session — look for session.json in the project dir. If one exists and is not destroyed, reuse it. If destroyed or missing, create a new one:

   python3 .claude/skills/databricks-repl/dbx_repl.py create \
     --cluster-id <RESOLVED_ID> --session-name <task-name> --profile $0
   

3. Execute the user's code via exec. Keep the session alive for follow-up commands.

4. Read output selectively — use stdout_preview for small output, read the file for larger output. Present results to the user.

5. Only destroy when the user says they're done, or when switching to a different task.

Quick Start

# 1. Create a session (injects bootstrap helpers into the REPL)
python .claude/skills/databricks-repl/dbx_repl.py create --cluster-id $CLUSTER_ID --session-name my-task --profile $0

# 2. Execute code (submit + wait, returns JSON metadata)
python .claude/skills/databricks-repl/dbx_repl.py exec \
  --command "df = spark.read.parquet('s3://bucket/data')" \
  --tag "load_data"

# 3. Read output selectively (never dump full stdout)
head -20 repl_outputs/001_load_data.stdout

# 4. Iterate — execute more commands, read results, refine
python .claude/skills/databricks-repl/dbx_repl.py exec \
  --command "print(df.describe().toPandas().to_string())" \
  --tag "explore"

# 5. Destroy when done
python .claude/skills/databricks-repl/dbx_repl.py destroy

Core Workflow

1. Create a session — one per task. The wrapper creates an execution context and injects bootstrap helpers.
2. Exec commands sequentially — each returns JSON metadata with stdout_file, stderr_file, and stdout_preview. Never see raw output in the response.
3. Read output files selectively — use head, grep, or partial reads. Control what enters your context.
4. Persist results to Volumes using standard Python (shutil.copy2, df.write.parquet, open().write()) when you need outputs to survive eviction.
5. Handle eviction if it occurs — the wrapper detects it and returns a structured response. Replay steps using idempotent patterns.
6. Destroy the session when finished.

Core Principles

• You are the root LM. Reason about the task, decide what code to run, read outputs selectively. The wrapper handles auth tokens, SDK internals, and polling so they stay out of your context — no need to inspect them.
• Project directory = session scope. All commands use --project-dir (defaults to cwd) to locate session.json and repl_outputs/. Each project directory is an independent session. Use --project-dir when managing multiple tasks from a single location.
• Append-only execution. Each exec is a new cell in an ordered log. Append corrections rather than editing previous commands — the ordered log enables lineage tracking and eviction replay.
• Output via files. The wrapper writes stdout/stderr to disk. Read only what you need.
• sub_llm() for heavy lifting. When you need to classify rows, summarize partitions, or process text at scale, use sub_llm() inside the REPL — it calls a Databricks serving endpoint.
• Use standard Python for persistence. Write files to Volumes with shutil.copy2, df.write.parquet(), open().write(), etc. Track paths yourself for eviction recovery.

CLI Reference

All commands accept these common options:

create

python .claude/skills/databricks-repl/dbx_repl.py create \
  --cluster-id <CLUSTER_ID> \
  --session-name <NAME>

exec

# Inline code
python .claude/skills/databricks-repl/dbx_repl.py exec \
  --command "df = spark.read.table('catalog.schema.table')" \
  --tag "load_data"

# Code from file (prefix with @)
python .claude/skills/databricks-repl/dbx_repl.py exec \
  --command @step3_train.py \
  --tag "train"

Side effects per exec:

• Command saved to {project-dir}/repl_outputs/{step}_{tag}.cmd.py
• stdout/stderr written to .stdout / .stderr files
• session.json steps array updated with the new step

cancel

# Auto-detect active command from session.json (preferred)
python .claude/skills/databricks-repl/dbx_repl.py cancel

# Or specify a command ID explicitly
python .claude/skills/databricks-repl/dbx_repl.py cancel --run-id <COMMAND_ID>

await

Re-poll a command that is still running after a previous exec or await timed out.

# Re-poll with default timeout (600s)
python .claude/skills/databricks-repl/dbx_repl.py await

# Re-poll with custom timeout
python .claude/skills/databricks-repl/dbx_repl.py await --timeout 1200

Reads active_command from session.json. If the command has finished, writes output files and records the step. If still running, returns a Running response with a tip for next actions.

destroy

python .claude/skills/databricks-repl/dbx_repl.py destroy

Response Schemas

Exec success:

{
  "cmd_id": "003_train",
  "status": "Finished",
  "stdout_file": "repl_outputs/003_train.stdout",
  "stderr_file": "repl_outputs/003_train.stderr",
  "stdout_bytes": 1247,
  "stderr_bytes": 342,
  "stdout_preview": "Training complete. F1: 0.87...",
  "elapsed_seconds": 47.3
}

Exec error:

{
  "cmd_id": "003_train",
  "status": "Error",
  "stdout_file": "repl_outputs/003_train.stdout",
  "stderr_file": "repl_outputs/003_train.stderr",
  "stdout_bytes": 0,
  "stderr_bytes": 891,
  "stderr_preview": "Traceback (most recent call last):\n  File...",
  "elapsed_seconds": 2.1
}

Exec interrupted (Ctrl+C):

{
  "cmd_id": "003_train",
  "status": "Cancelled",
  "stdout_file": "repl_outputs/003_train.stdout",
  "stderr_file": "repl_outputs/003_train.stderr",
  "stdout_bytes": 0,
  "stderr_bytes": 0,
  "elapsed_seconds": 12.4,
  "message": "Command interrupted by user."
}

Exec timeout (command still running):

{
  "cmd_id": "003_train",
  "status": "Running",
  "command_id": "abc123-def456",
  "elapsed_seconds": 600.0,
  "message": "Command still running after 600s client-side timeout.",
  "tip": {
    "await": "dbx_repl.py await --timeout 600 --project-dir .",
    "cancel": "dbx_repl.py cancel --project-dir ."
  }
}

Cancel success:

{
  "status": "cancelled",
  "command_id": "abc123-def456"
}

Exec eviction detected: See references/eviction-replay.md.

Decision guide after exec:

• stdout_bytes small (< 500) → read the full file
• stdout_bytes large → use stdout_preview or head/grep on the file
• status: "Error" → read stderr file, fix the code, exec again

Error Handling

Best Practices

• Tag every command with a descriptive --tag (e.g., load_data, train, evaluate). Tags appear in the manifest and output filenames.
• One logical step per exec. Keep commands focused. Avoid multi-paragraph scripts in a single exec — split into steps.
• Persist expensive results to Volumes. If it takes more than a few seconds to compute, write it to a Volume with standard Python. This is both a safety net and a replay optimization.
• Use sub_llm() instead of manual iteration. When processing rows, classifying text, or summarizing — use the on-cluster LM, not your own context.
• Handle timeouts gracefully. If an exec returns status: "Running", the command is still executing on the cluster. Use await to re-poll with a fresh timeout, or cancel to abort. Do not run another exec while a command is running — the execution context is single-threaded and the new command will queue behind the old one.
• Read before you act. After an exec, read the output file before deciding the next step. Don't chain multiple execs blindly.
• Execute from file for long code. Write complex code to a .py file first, then use --command @filename.py.
• Use --project-dir for multi-task workflows. Run independent sessions in separate project directories:

  python .claude/skills/databricks-repl/dbx_repl.py exec --project-dir ./examples/categorization --command "..." --tag "classify"
  python .claude/skills/databricks-repl/dbx_repl.py exec --project-dir ./examples/reranking --command "..." --tag "rerank"
  

Reference Files

Read these as needed for detailed documentation:

• references/bootstrap-helpers.md — Read when using sub_llm() or sub_llm_batch() inside the REPL. Includes signatures and examples.
• references/eviction-replay.md — Read when a command fails due to context eviction. Covers detection, the eviction response format, and idempotent replay strategies.