
dspy-agent-skills
Production-grade DSPy 3.2.x skills: fundamentals, evaluation, GEPA, BetterTogether, RLM, and end-to-end workflow.
- version
- 0.2.3
- GitHub stars
- 280
Install
/plugin install dspy-agent-skills@dspy-agent-skillsRun it inside Claude Code after adding the marketplace with /plugin marketplace add intertwine/dspy-agent-skills.
Contents
Discovered from sourceSkills 5
DSPy Advanced Workflow (2026)
This skill runs the seven-step loop that turns a natural-language task description into an optimized, saved, deployable DSPy program. Use the relevant steps in order. Stop at a validated baseline for a prototype; optimizer runs require an appropriate authorized budget and evidence of need. Exporting a local artifact does not authorize deployment.
The seven steps
1. Spec
Rephrase the user's task in one sentence. Identify inputs, outputs, the quality axis that matters, and any constraints (latency, cost, tool access, context size). Pick predictor shape:
| Task shape | Predictor |
|---|---|
| Single-step structured I/O | dspy.Predict / dspy.ChainOfThought |
| Tool use / multi-step | dspy.ReAct |
| Code execution | dspy.ProgramOfThought |
| Long context / codebase | dspy.RLM → dspy-rlm-module |
2. Program
Write the typed dspy.Signature + dspy.Module subclass per dspy-fundamentals. No hard-coded prompts. Keep predictors named so GEPA can target them.
3. Data
Build trainset and separate valset as dspy.Example(...).with_inputs(...). For GEPA, maximize trainset size and keep validation just large enough to represent downstream behavior; held-out testset is reported on at the end only. See dspy-evaluation-harness.
4. Rich metric
Write rich_metric(gold, pred, trace=None, pred_name=None, pred_trace=None) returning dspy.Prediction(score=0..1, feedback="natural-language critique"). The feedback is load-bearing — it's what GEPA's reflection LM learns from. A dict with the same fields crashes dspy.Evaluate; only dspy.Prediction aggregates correctly. See dspy-evaluation-harness.
5. Baseline
evaluator = dspy.Evaluate(devset=valset, metric=rich_metric,
num_threads=8, display_progress=True,
provide_traceback=True,
save_as_json="runs/baseline.json")
baseline = evaluator(program)
print("Baseline:", baseline.score)
6. GEPA optimize
reflection_lm = dspy.LM("openai/gpt-5", temperature=1.0, max_tokens=32000)
optimizer = dspy.GEPA(
metric=rich_metric,
auto="medium",
reflection_lm=reflection_lm,
candidate_selection_strategy="pareto",
track_stats=True,
track_best_outputs=True,
log_dir="./gepa_logs",
num_threads=8,
seed=0,
)
optimized = optimizer.compile(student=program, trainset=trainset, valset=valset)
print("Optimized:", evaluator(optimized).score)
Run auto="light" first as a sanity check; move to auto="medium"/"heavy" for the final run. See dspy-gepa-optimizer.
If you need a deliberate multi-stage compile loop, DSPy 3.2.x also exposes dspy.BetterTogether(metric=..., bootstrap=..., gepa=...) for chaining named optimizers after you have a clean baseline GEPA setup.
7. Export & deploy
optimized.save("artifacts/program.json", save_program=False) # state, portable
# or for full deployment artifact:
optimized.save("artifacts/program_dir/", save_program=True)
Deploy:
- Load with
dspy.load("artifacts/program_dir/")or reconstruct +.load("program.json"). - Wrap in FastAPI/CLI.
- Enable
track_usage=Truefor cost/latency observability. - Log with MLflow (
mlflow.dspy.autolog()) or W&B in CI. - Keep an offline regression test that runs the
evaluatoragainst the saved program and fails CI below a threshold.
Full orchestration template
"""DSPy end-to-end pipeline — spec → optimize → deploy."""
import dspy
from pathlib import Path
# ----- 1–2. Spec & program (dspy-fundamentals) -----
class MyTask(dspy.Signature):
"""<one-line instruction from the spec>."""
input_field: str = dspy.InputField()
output_field: str = dspy.OutputField()
class MyProgram(dspy.Module):
def __init__(self):
super().__init__()
self.step = dspy.ChainOfThought(MyTask)
def forward(self, **kw):
return self.step(**kw)
# ----- 3. Data (dspy-evaluation-harness) -----
trainset = [...] # list[dspy.Example(...).with_inputs(...)]
valset = [...]
# ----- 4. Rich metric (dspy-evaluation-harness) -----
def rich_metric(gold, pred, trace=None, pred_name=None, pred_trace=None):
score = ... # compute 0..1
feedback = ... # detailed critique
return dspy.Prediction(score=score, feedback=feedback) # NOT a dict
# ----- 5. Baseline -----
dspy.configure(lm=dspy.LM("openai/gpt-4o"), track_usage=True)
evaluator = dspy.Evaluate(devset=valset, metric=rich_metric, num_threads=8,
display_progress=True, provide_traceback=True,
save_as_json="runs/baseline.json")
program = MyProgram()
print("Baseline:", evaluator(program).score)
# ----- 6. GEPA optimize (dspy-gepa-optimizer) -----
optimizer = dspy.GEPA(
metric=rich_metric,
auto="medium",
reflection_lm=dspy.LM("openai/gpt-5", temperature=1.0, max_tokens=32000),
candidate_selection_strategy="pareto",
track_stats=True, track_best_outputs=True,
log_dir="./gepa_logs", num_threads=8, seed=0,
)
optimized = optimizer.compile(student=program, trainset=trainset, valset=valset)
print("Optimized:", evaluator(optimized).score)
# ----- 7. Export (dspy-fundamentals) -----
Path("artifacts").mkdir(exist_ok=True)
optimized.save("artifacts/program.json", save_program=False)
Guardrails
- Define the metric in step 4 before optimization; use informative feedback appropriate to the task.
- Always baseline before optimizing — no baseline, no claim.
- Save both pre- and post-optimization metrics to JSON for auditability.
- If held-out test score drops, preserve that result and diagnose using training/validation evidence rather than assuming a cause. After test-informed changes, use a new untouched final holdout or label subsequent results exploratory; do not repeatedly tune against the original test set.
- Freeze optimized program with
module._compiled = Truebefore multi-stage re-compilation.
Runnable scaffold → example_pipeline.py
DSPy Evaluation Harness (3.2.x)
The metric is usually more important than the program. For dspy.GEPA especially, the quality of textual feedback in your metric determines whether optimization converges.
Two rules
- Return a
dspy.Prediction(score=..., feedback=...), not a dict.dspy.Evaluate's parallel executor aggregates scores via sum, which breaks on dict outputs (TypeError: unsupported operand type(s) for +: 'int' and 'dict').dspy.Predictionsupports__float__/__add__and is what GEPA's adapter natively unwraps. A bare float still works for puredspy.Evaluatescoring, but GEPA needs the score+feedback pair. - Separate valset. Never optimize and evaluate on the same examples. Optimizers overfit fast.
Canonical rich-feedback metric
import dspy
def rich_metric(gold: dspy.Example, pred: dspy.Prediction, trace=None,
pred_name: str | None = None, pred_trace=None):
# 1. Compute sub-scores — multi-axis beats scalar
correctness = 1.0 if _normalize(pred.answer) == _normalize(gold.answer) else 0.0
cited = _has_citation(pred.answer, gold.sources) if hasattr(gold, "sources") else 1.0
concise = 1.0 if len(pred.answer.split()) <= 50 else 0.5
score = 0.6 * correctness + 0.25 * cited + 0.15 * concise
# 2. Write feedback that teaches the optimizer
parts = []
if correctness < 1.0:
parts.append(
f"Answer mismatch. Predicted: {pred.answer!r}. Expected: {gold.answer!r}. "
f"Likely cause: reasoning skipped the units/quantity in the question."
)
if cited < 1.0:
parts.append("Did not ground the claim in the provided sources. Quote a source fragment.")
if concise < 1.0:
parts.append("Answer exceeded 50 words — tighten to one sentence.")
if not parts:
parts.append("Correct, grounded, and concise.")
feedback = " ".join(parts)
return dspy.Prediction(score=score, feedback=feedback)
Canonical harness
evaluator = dspy.Evaluate(
devset=valset,
metric=rich_metric,
num_threads=8,
display_progress=True,
display_table=10, # pretty-print first 10 rows
provide_traceback=True, # surface exceptions, don't swallow them
max_errors=5,
failure_score=0.0,
save_as_json="eval_runs/baseline.json",
)
result = evaluator(program)
print("Overall:", result.score)
for example_result in result.results[:3]:
print(example_result)
dspy.Evaluate returns an EvaluationResult with .score (aggregate float) and .results (list of (example, pred, score) tuples).
Dataset hygiene
- Size: 20–50 examples is enough for GEPA's reflective loop; 100–500 for MIPROv2-style bootstrapping.
- Split: hand-curate two disjoint sets —
trainset(for optimization) andvalset(for metric-on-optimized-program). A test set you never look at during development is gold. - Representativeness beats size. Include edge cases, ambiguity, adversarial inputs.
- Build
dspy.Example(...).with_inputs("question", "context")— thewith_inputscall marks which fields are inputs vs. gold outputs.
trainset = [
dspy.Example(question="…", answer="…").with_inputs("question"),
...
]
Multi-axis metrics (recommended)
Combine correctness, faithfulness, format adherence, latency, and cost. Each axis should be a 0–1 float with a written definition. Weight them explicitly; don't hide weights inside magic numbers — make them constants so optimizers can be told to trade off.
CI-ready eval suite
# tests/test_dspy_eval.py
import dspy, pytest
from my_program import program, valset, rich_metric
@pytest.fixture(scope="module")
def evaluator():
return dspy.Evaluate(devset=valset, metric=rich_metric, num_threads=8,
display_progress=False, provide_traceback=True)
def test_program_meets_threshold(evaluator):
result = evaluator(program)
assert result.score >= 0.75, f"Regression: {result.score:.3f}"
Run offline in CI with a cached LM (dspy.LM(..., cache=True)) + pre-populated DSPY_CACHEDIR.
Tracing & observability
track_usage=Trueondspy.configureaccumulates token counts on predictions (pred.get_lm_usage()).- MLflow:
import mlflow; mlflow.dspy.autolog()→ traces every prediction. - W&B: pass
use_wandb=Truetodspy.GEPAto log Pareto fronts. - Always log eval results to a versioned JSON file (
save_as_json=...) so you can diff runs.
Anti-patterns
- Scalar-only metrics (float but no feedback) when using GEPA — wasted signal.
return {"score": s, "feedback": f}(dict) — crashesdspy.Evaluate's parallel aggregator. Usedspy.Prediction(score=s, feedback=f).- Exact-match metrics on open-ended generation tasks — use semantic or LM-as-judge scoring.
- Evaluating on the trainset — optimistic by 10–30 points.
- Silently swallowing exceptions (
provide_traceback=False) — you'll blame the LM for a KeyError. - Changing the metric mid-experiment without re-baselining — prior numbers become incomparable.
Next
- Feed this metric into
dspy-gepa-optimizer. - Full harness pattern → reference.md.
- Runnable example → example_metric.py.
DSPy Fundamentals (3.2.x)
DSPy is the "PyTorch for prompts" — you declare Signatures (typed I/O contracts), compose them into Modules, and let optimizers (not you) tune the instructions and few-shot examples. Never write raw prompts.
The one-paragraph model
Configure a single LM globally with dspy.configure(lm=...). Define a dspy.Signature subclass with dspy.InputField() / dspy.OutputField() (docstring becomes the instruction). Wrap it in a predictor — dspy.Predict (direct), dspy.ChainOfThought (adds reasoning), dspy.ReAct (tool-using agent), dspy.ProgramOfThought (code-executing), or dspy.RLM (long-context). Subclass dspy.Module to compose multi-step programs. For built-in providers, use dspy.LM("provider/model"); for a truly custom backend, subclass dspy.BaseLM. Optimize later with GEPA.
Canonical template
import dspy
dspy.configure(lm=dspy.LM("openai/gpt-4o"), track_usage=True)
class QuestionAnswer(dspy.Signature):
"""Answer questions with rigorous step-by-step reasoning."""
question: str = dspy.InputField()
answer: str = dspy.OutputField(desc="concise final answer")
class QAProgram(dspy.Module):
def __init__(self):
super().__init__()
self.solve = dspy.ChainOfThought(QuestionAnswer)
def forward(self, question: str) -> dspy.Prediction:
return self.solve(question=question)
program = QAProgram()
pred = program(question="What is 2 + 2?")
print(pred.reasoning, pred.answer)
Predictor cheatsheet (DSPy 3.2.x)
| Predictor | When to use | Adds |
|---|---|---|
dspy.Predict(sig) | Simple structured I/O | nothing — just the signature |
dspy.ChainOfThought(sig) | Reasoning tasks | a reasoning output field |
dspy.ReAct(sig, tools=[...], max_iters=20) | Tool-using agent | Thought/Action/Observation loop |
dspy.ProgramOfThought(sig, max_iters=3) | Math/data tasks | generates & runs Python (needs Deno) |
dspy.RLM(sig, ...) | Long context / codebases | recursive REPL exploration (see dspy-rlm-module) |
Typed outputs — use Pydantic on fields, not TypedPredictor
dspy.TypedPredictor is superseded; dspy.Predict now handles Pydantic types natively via field annotations.
from pydantic import BaseModel
from typing import Literal
class Entity(BaseModel):
name: str
kind: Literal["person", "org", "place"]
class ExtractEntities(dspy.Signature):
"""Extract named entities from text."""
text: str = dspy.InputField()
entities: list[Entity] = dspy.OutputField()
extractor = dspy.Predict(ExtractEntities)
Save & load
Two modes — know the difference:
# State-only (portable JSON; you must rebuild the architecture to load)
program.save("program.json", save_program=False)
new = QAProgram(); new.load("program.json")
# Full program (cloudpickle into a directory; restores everything)
program.save("./program_dir/", save_program=True)
restored = dspy.load("./program_dir/")
Prefer state-only for version control; full-program for deployment artifacts.
Ten anti-patterns to refuse
- Hard-coded prompt strings (
"You are a helpful assistant...") — write a Signature. dspy.TypedPredictor(...)in new code — usedspy.Predictwith Pydantic fields.dspy.OpenAI(...)/dspy.settings.configure(...)— usedspy.configure(lm=dspy.LM(...)).- Provider-specific LM classes for built-in providers — use
dspy.LM("provider/model"). If DSPy doesn't ship your backend, subclassdspy.BaseLM. - Giant monolithic predictors that do five jobs — decompose into a
Modulewith named sub-predictors. - Mutating
signature.instructionsby hand — let the optimizer do it. - In-lining few-shot demos in the Signature docstring — bootstrap/optimize them.
- Using
pickle.dump(program)— useprogram.save(...). - Setting an LM per module at construction time without reason — configure globally, override only when you need model mixing.
- Vague metrics (yes/no, exact-match only) when training an optimizer — see
dspy-evaluation-harness.
Configuring the LM
dspy.configure(
lm=dspy.LM("openai/gpt-4o", temperature=0.0, max_tokens=2000),
track_usage=True, # accumulate token counts on predictions
async_max_workers=4, # for .acall / batch
)
DSPy 3.2.x warns by default when a module call passes extra input fields or values that don't match the signature's declared types. Treat those warnings as a callsite bug first; if you're intentionally passing pre-serialized values, disable them with dspy.configure(warn_on_type_mismatch=False).
Common provider prefixes: openai/, anthropic/, azure/, vertex_ai/, bedrock/, ollama/. For local Ollama: dspy.LM("ollama_chat/llama3.1:8b", api_base="http://localhost:11434").
Where to go next
- Measuring quality →
dspy-evaluation-harness - Automatic optimization →
dspy-gepa-optimizer - Context >100k tokens →
dspy-rlm-module - Full pipeline →
dspy-advanced-workflow - Full API reference → reference.md
- Runnable example → example_qa.py
DSPy GEPA Optimizer (3.2.x)
GEPA (Genetic-Pareto) is a reflective optimizer: it mutates a program's instructions and few-shots using an LM that reads your metric's textual feedback and proposes improvements. It maintains a Pareto frontier across validation tasks and is the default recommendation for complex DSPy workloads in 2026.
The expansion "Genetic-Evolutionary Prompt Adaptation" that appears in some AI-generated summaries is an LLM-hallucinated backronym. The paper defines GEPA as Genetic-Pareto; the "Pareto" is load-bearing (GEPA keeps a frontier of candidates rather than collapsing to one).
Prerequisites — do these first or GEPA wastes rollouts
- A
dspy.Modulethat runs end-to-end (seedspy-fundamentals). - A rich-feedback metric returning
dspy.Prediction(score=float, feedback=str)(seedspy-evaluation-harness). Informative feedback can support reflection; evaluate optimizer benefit on the task rather than assuming superiority. A dict with the same fields still crashesdspy.Evaluateunder DSPy 3.2.1 — usedspy.Prediction. trainsetand a separatevalset. For GEPA, maximize training examples and keep validation just large enough to represent the downstream distribution; do not reuse the same examples for both.- A
reflection_lm— a strong LM (often the same or stronger than the task LM) set totemperature=1.0for creative proposals. Current DSPy docs use a GPT-5-class reflection model with a large output budget.
Canonical call
import dspy
dspy.configure(lm=dspy.LM("openai/gpt-5-mini"))
reflection_lm = dspy.LM("openai/gpt-5", temperature=1.0, max_tokens=32000)
optimizer = dspy.GEPA(
metric=rich_metric,
auto="medium", # "light" / "medium" / "heavy"
reflection_lm=reflection_lm,
reflection_minibatch_size=3,
candidate_selection_strategy="pareto", # or "current_best"
skip_perfect_score=True,
use_merge=True,
num_threads=8,
track_stats=True,
track_best_outputs=True, # enables inference-time best-of selection
log_dir="./gepa_logs", # resume/checkpoint
seed=0,
)
optimized = optimizer.compile(
student=program,
trainset=trainset,
valset=valset,
)
# Pareto inspection
pareto = optimized.detailed_results.val_aggregate_scores
print("Pareto frontier:", sorted(pareto, reverse=True)[:5])
optimized.save("optimized_program.json", save_program=False)
Import paths
Either works; use the top-level in new code:
import dspy
dspy.GEPA(...) # preferred
# equivalently:
from dspy.teleprompt import GEPA
Metric contract (precise)
import dspy
def rich_metric(gold, pred, trace=None, pred_name=None, pred_trace=None):
score = ... # 0.0..1.0
feedback = ... # detailed natural-language critique
return dspy.Prediction(score=score, feedback=feedback)
Return dspy.Prediction, not a dict. Some upstream GEPA prose describes score/feedback as a dict-like shape, but dspy.Evaluate in DSPy 3.2.1 still crashes on a literal dict metric (TypeError: unsupported operand type(s) for +: 'int' and 'dict'). GEPA uses dspy.Evaluate internally for candidate scoring, so a dict return can fail inside GEPA too, not just in your explicit Evaluate(...) calls.
pred_name/pred_traceare set during reflection on a specific predictor inside your module — write per-predictor feedback when possible (credit assignment). If you cannot localize feedback, return program-level feedback rather than a vague score-only critique.- Feedback quality is the load-bearing part: specifics about why it failed and what good looks like are what the reflection LM acts on.
Budget knobs
Use either auto=... or explicit budget — not both.
| Mode | Rough rollouts | When to use |
|---|---|---|
auto="light" | ~20–40 full evals | Sanity-check GEPA works on your metric |
auto="medium" | ~80–150 full evals | Everyday optimization |
auto="heavy" | ~300–600 full evals | Final run before ship |
max_full_evals=N | Explicit | Deterministic budget |
max_metric_calls=N | Explicit | Hard cap on metric invocations (more predictable cost) |
Each "full eval" ≈ len(valset) metric calls. Budget accordingly for cost.
Constructor parameters (every one, DSPy 3.2.x)
dspy.GEPA(
metric, # required
auto=None, # Literal["light","medium","heavy"] | None
max_full_evals=None,
max_metric_calls=None,
reflection_minibatch_size=3,
candidate_selection_strategy="pareto", # or "current_best"
reflection_lm=None, # required in practice
skip_perfect_score=True,
add_format_failure_as_feedback=False,
instruction_proposer=None, # custom ProposalFn
component_selector="round_robin", # or a callable
use_merge=True,
max_merge_invocations=5,
num_threads=None,
failure_score=0.0,
perfect_score=1.0,
log_dir=None,
track_stats=False,
use_wandb=False,
wandb_api_key=None, # overrides WANDB_API_KEY env var
wandb_init_kwargs=None, # dict forwarded to wandb.init(...)
track_best_outputs=False,
warn_on_score_mismatch=True,
use_mlflow=False,
seed=0,
gepa_kwargs=None, # e.g. {"use_cloudpickle": True} for dynamic signatures
)
.compile(student, *, trainset, valset=None, teacher=None) — teacher is not currently used.
Data split guidance
DSPy's general prompt-optimizer docs often recommend a validation-heavy split, such as 20% train / 80% validation, because small prompt optimizers can overfit tiny trainsets. GEPA is different: maximize the training set and reserve only enough validation examples to represent downstream behavior. The Pareto frontier still needs a real valset, but GEPA learns from traces and textual feedback on training examples, so starving trainset hurts.
BetterTogether in DSPy 3.2.x
If you want a multi-stage optimizer loop, DSPy 3.2.0's BetterTogether now accepts arbitrary named optimizers instead of the older fixed prompt_optimizer / weight_optimizer pair:
optimizer = dspy.BetterTogether(
metric=rich_metric,
bootstrap=dspy.BootstrapFewShotWithRandomSearch(metric=rich_metric),
gepa=dspy.GEPA(metric=rich_metric, auto="light", reflection_lm=reflection_lm),
)
optimized = optimizer.compile(
student=program,
trainset=trainset,
valset=valset,
strategy="bootstrap -> gepa",
)
Pass strategy= explicitly when you use named stages like bootstrap=... and gepa=.... DSPy 3.2.0's default strategy is still "p -> w -> p", which only works if your optimizer keys are literally p and w.
Keep plain GEPA as the default first pass. Reach for BetterTogether only when you have a specific reason to chain optimizers and want the valset to pick the best intermediate program.
When GEPA > MIPROv2
- Your metric can produce specific, teachable critiques (GEPA's superpower).
- The program has multiple predictors that need targeted improvements (GEPA gives per-predictor feedback; MIPRO doesn't).
- Rollout budget is small (GEPA converges faster with rich feedback).
When MIPROv2 > GEPA
- Metric is scalar-only (no signal to reflect on) — use
dspy.MIPROv2. - You want pure few-shot bootstrapping with no instruction mutation.
- Very large trainset (500+) where Bayesian search over demos pays off.
When SIMBA is worth trying
dspy.SIMBA is a lighter reflective optimizer. Try it when you want a cheaper reflective pass than GEPA, your program is simple, or you need quick exploration before committing to a full GEPA run. Keep GEPA as the default for multi-predictor programs where per-predictor feedback and Pareto candidate selection matter.
Resume & checkpointing
log_dir writes candidate programs + scores per round. To resume an interrupted run, point log_dir at the same directory — GEPA picks up from the last checkpoint. Inspect <log_dir>/candidates/ to see every proposed program.
Inference-time best-of with track_best_outputs
With track_best_outputs=True, GEPA records, per task, the best prediction seen across all candidates. At inference time on held-out data, you can ensemble or select among the top-Pareto programs for robustness. Access via optimized.detailed_results.best_outputs_valset.
Anti-patterns
- Float-only metric ("score is 0.7") with no feedback — GEPA collapses to random search.
- Same set used for train and val — Pareto selection overfits.
reflection_lm= small model — it can't critique; use the strongest LM you can afford for this role.- Running
auto="heavy"on an untested metric — burn money to learn the metric was bugged. Runauto="light"first. - Ignoring
log_dir— losing a 4-hour run to a disconnect is very painful.
Gotcha: reflection_lm is required at construction, not compile
dspy.GEPA(...) asserts reflection_lm is not None (or a custom instruction_proposer) at init time — you cannot defer it to .compile(). If you see
AssertionError: GEPA requires a reflection language model...
add reflection_lm=dspy.LM("openai/gpt-5", temperature=1.0, max_tokens=32000) to the constructor, or substitute the strongest instruction-following model available on your provider. dspy.LM(...) is a cheap stub until you actually call it, so constructing one doesn't hit the network.
Next
- Build the metric →
dspy-evaluation-harness. - End-to-end pipeline →
dspy-advanced-workflow. - Parameter reference → reference.md.
- Runnable example → example_gepa.py.
- BetterTogether chaining example → example_bettertogether.py.
dspy.RLM — Recursive Language Model
dspy.RLM runs the LLM in a sandboxed Python REPL (Pyodide/WASM via Deno) with access to the full context as variables. The LLM writes code to slice, grep, summarize, and recursively sub-query the data, iterating until it can answer. Use it when the context is too large to cram into a single prompt.
Prerequisites
- Deno installed (for the default
PythonInterpreter):brew install denoor see https://deno.land. The interpreter is a Pyodide-in-WASM sandbox spawned by Deno. - A sub-LM for inner calls — usually a cheaper model than the outer LM. Defaults to
dspy.settings.lm.
Canonical usage
import dspy
dspy.configure(lm=dspy.LM("openai/gpt-4o"))
sub_lm = dspy.LM("openai/gpt-4o-mini") # cheap inner model
rlm = dspy.RLM(
"context, query -> answer",
max_iterations=20,
max_llm_calls=50,
max_output_chars=10_000,
sub_lm=sub_lm,
tools=[],
verbose=False,
)
result = rlm(
context=open("huge_log.txt").read(), # can be 500k+ tokens
query="Summarize every unique error class and how many times each appeared.",
)
print(result.answer)
Full constructor
dspy.RLM(
signature: type[Signature] | str,
max_iterations: int = 20, # REPL loop cap
max_llm_calls: int = 50, # sub-LM call cap (stops runaway recursion)
max_output_chars: int = 10_000, # truncate REPL stdout per step
verbose: bool = False, # print the REPL trace
tools: list[Callable] | None = None,
sub_lm: dspy.LM | None = None,
interpreter: CodeInterpreter | None = None, # custom sandbox
)
When to reach for RLM vs. alternatives
| Situation | Use |
|---|---|
| Context <100k, answer fits one LM call | dspy.Predict / dspy.ChainOfThought |
| Need external tools (web, db) | dspy.ReAct(tools=[...]) |
| Math/code that must run | dspy.ProgramOfThought |
| Huge context, recursive chunking, or data-exploration loop | dspy.RLM |
| Entire-codebase reasoning where the LM should grep/read files | dspy.RLM with file-reading tools=[...] |
Composition — RLM as a module inside a larger program
Wrap the RLM in your own dspy.Module and optimize the enclosing program with GEPA. GEPA can tune both the RLM's outer signature instruction and the surrounding predictors.
class RepoAuditor(dspy.Module):
def __init__(self):
super().__init__()
self.explore = dspy.RLM("repo_tree, question -> findings",
max_iterations=30, sub_lm=dspy.LM("openai/gpt-4o-mini"))
self.synth = dspy.ChainOfThought("findings, question -> report")
def forward(self, repo_tree, question):
f = self.explore(repo_tree=repo_tree, question=question).findings
return self.synth(findings=f, question=question)
Then: dspy.GEPA(metric=..., ...).compile(student=RepoAuditor(), trainset=..., valset=...).
Practical tips
- Budget carefully. A single RLM call can issue dozens of sub-LM calls. Keep
max_llm_callstight (20–50) in production; raise for research. - The default stdout cap is smaller in DSPy 3.2.x.
max_output_charsnow defaults to10_000; raise it deliberately if your REPL tools print large tables or document slices. - Use a cheap
sub_lm. The outer LM orchestrates; inner calls (summarize, filter, score) don't need the flagship model. - Pass data as kwargs, not in the instruction.
rlm(context=huge_string, query="...")lets the REPL treatcontextas a Python variable. Avoid concatenating it into the prompt. verbose=Truewhile debugging. Prints every REPL step — invaluable when the RLM appears to hang or loop.- Custom tools are regular Python callables passed via
tools=[...]; they are exposed inside the sandbox. Useful forread_file,grep,vector_search, etc. In DSPy 3.2.x they are invoked by keyword, so give them named, typed parameters rather than positional-only signatures. - Deno install is required. Missing Deno is the #1 RLM error. Check
which denobefore reporting bugs.
Security note
The default interpreter is a Deno-sandboxed Pyodide WASM runtime — no filesystem, network, or subprocess access by default. If you pass custom tools that do I/O, your tools' security posture is yours. Never hand raw subprocess.run to the RLM.
Anti-patterns
- Using RLM when a 32k-token prompt would fit — overhead is not worth it.
- Missing Deno → hard-to-diagnose failures. Install it.
max_llm_callsleft at default in a production path — runaway cost.- Passing secrets in the
contextstring — they get echoed into REPL state.
Next
- Wrap-and-optimize with GEPA →
dspy-gepa-optimizer. - Full reference → reference.md.
Sponsors
Details
- Marketplace
- intertwine/dspy-agent-skills
- Author
- Bryan Young
- Category
- Llm Frameworks
- License
- MIT
- Keywords
- dspy, gepa, rlm, optimization, llm, evaluation