Production-grade DSPy 3.2.x skills: fundamentals, evaluation, GEPA, BetterTogether, RLM, and end-to-end workflow.

version
0.2.3
GitHub stars
280

Install

/plugin install dspy-agent-skills@dspy-agent-skills

Run it inside Claude Code after adding the marketplace with /plugin marketplace add intertwine/dspy-agent-skills.

Contents

Discovered from source

Skills 5

DSPy Advanced Workflow (2026)

This skill runs the seven-step loop that turns a natural-language task description into an optimized, saved, deployable DSPy program. Use the relevant steps in order. Stop at a validated baseline for a prototype; optimizer runs require an appropriate authorized budget and evidence of need. Exporting a local artifact does not authorize deployment.

The seven steps

1. Spec

Rephrase the user's task in one sentence. Identify inputs, outputs, the quality axis that matters, and any constraints (latency, cost, tool access, context size). Pick predictor shape:

Task shapePredictor
Single-step structured I/Odspy.Predict / dspy.ChainOfThought
Tool use / multi-stepdspy.ReAct
Code executiondspy.ProgramOfThought
Long context / codebasedspy.RLM → dspy-rlm-module

2. Program

Write the typed dspy.Signature + dspy.Module subclass per dspy-fundamentals. No hard-coded prompts. Keep predictors named so GEPA can target them.

3. Data

Build trainset and separate valset as dspy.Example(...).with_inputs(...). For GEPA, maximize trainset size and keep validation just large enough to represent downstream behavior; held-out testset is reported on at the end only. See dspy-evaluation-harness.

4. Rich metric

Write rich_metric(gold, pred, trace=None, pred_name=None, pred_trace=None) returning dspy.Prediction(score=0..1, feedback="natural-language critique"). The feedback is load-bearing — it's what GEPA's reflection LM learns from. A dict with the same fields crashes dspy.Evaluate; only dspy.Prediction aggregates correctly. See dspy-evaluation-harness.

5. Baseline

evaluator = dspy.Evaluate(devset=valset, metric=rich_metric,
                          num_threads=8, display_progress=True,
                          provide_traceback=True,
                          save_as_json="runs/baseline.json")
baseline = evaluator(program)
print("Baseline:", baseline.score)

6. GEPA optimize

reflection_lm = dspy.LM("openai/gpt-5", temperature=1.0, max_tokens=32000)
optimizer = dspy.GEPA(
    metric=rich_metric,
    auto="medium",
    reflection_lm=reflection_lm,
    candidate_selection_strategy="pareto",
    track_stats=True,
    track_best_outputs=True,
    log_dir="./gepa_logs",
    num_threads=8,
    seed=0,
)
optimized = optimizer.compile(student=program, trainset=trainset, valset=valset)
print("Optimized:", evaluator(optimized).score)

Run auto="light" first as a sanity check; move to auto="medium"/"heavy" for the final run. See dspy-gepa-optimizer.

If you need a deliberate multi-stage compile loop, DSPy 3.2.x also exposes dspy.BetterTogether(metric=..., bootstrap=..., gepa=...) for chaining named optimizers after you have a clean baseline GEPA setup.

7. Export & deploy

optimized.save("artifacts/program.json", save_program=False)     # state, portable
# or for full deployment artifact:
optimized.save("artifacts/program_dir/", save_program=True)

Deploy:

  • Load with dspy.load("artifacts/program_dir/") or reconstruct + .load("program.json").
  • Wrap in FastAPI/CLI.
  • Enable track_usage=True for cost/latency observability.
  • Log with MLflow (mlflow.dspy.autolog()) or W&B in CI.
  • Keep an offline regression test that runs the evaluator against the saved program and fails CI below a threshold.

Full orchestration template

"""DSPy end-to-end pipeline — spec → optimize → deploy."""

import dspy
from pathlib import Path

# ----- 1–2. Spec & program (dspy-fundamentals) -----
class MyTask(dspy.Signature):
    """<one-line instruction from the spec>."""
    input_field: str = dspy.InputField()
    output_field: str = dspy.OutputField()

class MyProgram(dspy.Module):
    def __init__(self):
        super().__init__()
        self.step = dspy.ChainOfThought(MyTask)
    def forward(self, **kw):
        return self.step(**kw)

# ----- 3. Data (dspy-evaluation-harness) -----
trainset = [...]   # list[dspy.Example(...).with_inputs(...)]
valset   = [...]

# ----- 4. Rich metric (dspy-evaluation-harness) -----
def rich_metric(gold, pred, trace=None, pred_name=None, pred_trace=None):
    score = ...          # compute 0..1
    feedback = ...       # detailed critique
    return dspy.Prediction(score=score, feedback=feedback)  # NOT a dict

# ----- 5. Baseline -----
dspy.configure(lm=dspy.LM("openai/gpt-4o"), track_usage=True)
evaluator = dspy.Evaluate(devset=valset, metric=rich_metric, num_threads=8,
                          display_progress=True, provide_traceback=True,
                          save_as_json="runs/baseline.json")
program = MyProgram()
print("Baseline:", evaluator(program).score)

# ----- 6. GEPA optimize (dspy-gepa-optimizer) -----
optimizer = dspy.GEPA(
    metric=rich_metric,
    auto="medium",
    reflection_lm=dspy.LM("openai/gpt-5", temperature=1.0, max_tokens=32000),
    candidate_selection_strategy="pareto",
    track_stats=True, track_best_outputs=True,
    log_dir="./gepa_logs", num_threads=8, seed=0,
)
optimized = optimizer.compile(student=program, trainset=trainset, valset=valset)
print("Optimized:", evaluator(optimized).score)

# ----- 7. Export (dspy-fundamentals) -----
Path("artifacts").mkdir(exist_ok=True)
optimized.save("artifacts/program.json", save_program=False)

Guardrails

  • Define the metric in step 4 before optimization; use informative feedback appropriate to the task.
  • Always baseline before optimizing — no baseline, no claim.
  • Save both pre- and post-optimization metrics to JSON for auditability.
  • If held-out test score drops, preserve that result and diagnose using training/validation evidence rather than assuming a cause. After test-informed changes, use a new untouched final holdout or label subsequent results exploratory; do not repeatedly tune against the original test set.
  • Freeze optimized program with module._compiled = True before multi-stage re-compilation.

Runnable scaffold → example_pipeline.py

DSPy Evaluation Harness (3.2.x)

The metric is usually more important than the program. For dspy.GEPA especially, the quality of textual feedback in your metric determines whether optimization converges.

Two rules

  1. Return a dspy.Prediction(score=..., feedback=...), not a dict. dspy.Evaluate's parallel executor aggregates scores via sum, which breaks on dict outputs (TypeError: unsupported operand type(s) for +: 'int' and 'dict'). dspy.Prediction supports __float__/__add__ and is what GEPA's adapter natively unwraps. A bare float still works for pure dspy.Evaluate scoring, but GEPA needs the score+feedback pair.
  2. Separate valset. Never optimize and evaluate on the same examples. Optimizers overfit fast.

Canonical rich-feedback metric

import dspy

def rich_metric(gold: dspy.Example, pred: dspy.Prediction, trace=None,
                pred_name: str | None = None, pred_trace=None):
    # 1. Compute sub-scores — multi-axis beats scalar
    correctness = 1.0 if _normalize(pred.answer) == _normalize(gold.answer) else 0.0
    cited = _has_citation(pred.answer, gold.sources) if hasattr(gold, "sources") else 1.0
    concise = 1.0 if len(pred.answer.split()) <= 50 else 0.5
    score = 0.6 * correctness + 0.25 * cited + 0.15 * concise

    # 2. Write feedback that teaches the optimizer
    parts = []
    if correctness < 1.0:
        parts.append(
            f"Answer mismatch. Predicted: {pred.answer!r}. Expected: {gold.answer!r}. "
            f"Likely cause: reasoning skipped the units/quantity in the question."
        )
    if cited < 1.0:
        parts.append("Did not ground the claim in the provided sources. Quote a source fragment.")
    if concise < 1.0:
        parts.append("Answer exceeded 50 words — tighten to one sentence.")
    if not parts:
        parts.append("Correct, grounded, and concise.")
    feedback = " ".join(parts)

    return dspy.Prediction(score=score, feedback=feedback)

Canonical harness

evaluator = dspy.Evaluate(
    devset=valset,
    metric=rich_metric,
    num_threads=8,
    display_progress=True,
    display_table=10,          # pretty-print first 10 rows
    provide_traceback=True,    # surface exceptions, don't swallow them
    max_errors=5,
    failure_score=0.0,
    save_as_json="eval_runs/baseline.json",
)
result = evaluator(program)
print("Overall:", result.score)
for example_result in result.results[:3]:
    print(example_result)

dspy.Evaluate returns an EvaluationResult with .score (aggregate float) and .results (list of (example, pred, score) tuples).

Dataset hygiene

  • Size: 20–50 examples is enough for GEPA's reflective loop; 100–500 for MIPROv2-style bootstrapping.
  • Split: hand-curate two disjoint sets — trainset (for optimization) and valset (for metric-on-optimized-program). A test set you never look at during development is gold.
  • Representativeness beats size. Include edge cases, ambiguity, adversarial inputs.
  • Build dspy.Example(...).with_inputs("question", "context") — the with_inputs call marks which fields are inputs vs. gold outputs.
trainset = [
    dspy.Example(question="…", answer="…").with_inputs("question"),
    ...
]

Multi-axis metrics (recommended)

Combine correctness, faithfulness, format adherence, latency, and cost. Each axis should be a 0–1 float with a written definition. Weight them explicitly; don't hide weights inside magic numbers — make them constants so optimizers can be told to trade off.

CI-ready eval suite

# tests/test_dspy_eval.py
import dspy, pytest
from my_program import program, valset, rich_metric

@pytest.fixture(scope="module")
def evaluator():
    return dspy.Evaluate(devset=valset, metric=rich_metric, num_threads=8,
                         display_progress=False, provide_traceback=True)

def test_program_meets_threshold(evaluator):
    result = evaluator(program)
    assert result.score >= 0.75, f"Regression: {result.score:.3f}"

Run offline in CI with a cached LM (dspy.LM(..., cache=True)) + pre-populated DSPY_CACHEDIR.

Tracing & observability

  • track_usage=True on dspy.configure accumulates token counts on predictions (pred.get_lm_usage()).
  • MLflow: import mlflow; mlflow.dspy.autolog() → traces every prediction.
  • W&B: pass use_wandb=True to dspy.GEPA to log Pareto fronts.
  • Always log eval results to a versioned JSON file (save_as_json=...) so you can diff runs.

Anti-patterns

  • Scalar-only metrics (float but no feedback) when using GEPA — wasted signal.
  • return {"score": s, "feedback": f} (dict) — crashes dspy.Evaluate's parallel aggregator. Use dspy.Prediction(score=s, feedback=f).
  • Exact-match metrics on open-ended generation tasks — use semantic or LM-as-judge scoring.
  • Evaluating on the trainset — optimistic by 10–30 points.
  • Silently swallowing exceptions (provide_traceback=False) — you'll blame the LM for a KeyError.
  • Changing the metric mid-experiment without re-baselining — prior numbers become incomparable.

Next

DSPy Fundamentals (3.2.x)

DSPy is the "PyTorch for prompts" — you declare Signatures (typed I/O contracts), compose them into Modules, and let optimizers (not you) tune the instructions and few-shot examples. Never write raw prompts.

The one-paragraph model

Configure a single LM globally with dspy.configure(lm=...). Define a dspy.Signature subclass with dspy.InputField() / dspy.OutputField() (docstring becomes the instruction). Wrap it in a predictor — dspy.Predict (direct), dspy.ChainOfThought (adds reasoning), dspy.ReAct (tool-using agent), dspy.ProgramOfThought (code-executing), or dspy.RLM (long-context). Subclass dspy.Module to compose multi-step programs. For built-in providers, use dspy.LM("provider/model"); for a truly custom backend, subclass dspy.BaseLM. Optimize later with GEPA.

Canonical template

import dspy

dspy.configure(lm=dspy.LM("openai/gpt-4o"), track_usage=True)

class QuestionAnswer(dspy.Signature):
    """Answer questions with rigorous step-by-step reasoning."""
    question: str = dspy.InputField()
    answer: str = dspy.OutputField(desc="concise final answer")

class QAProgram(dspy.Module):
    def __init__(self):
        super().__init__()
        self.solve = dspy.ChainOfThought(QuestionAnswer)

    def forward(self, question: str) -> dspy.Prediction:
        return self.solve(question=question)

program = QAProgram()
pred = program(question="What is 2 + 2?")
print(pred.reasoning, pred.answer)

Predictor cheatsheet (DSPy 3.2.x)

PredictorWhen to useAdds
dspy.Predict(sig)Simple structured I/Onothing — just the signature
dspy.ChainOfThought(sig)Reasoning tasksa reasoning output field
dspy.ReAct(sig, tools=[...], max_iters=20)Tool-using agentThought/Action/Observation loop
dspy.ProgramOfThought(sig, max_iters=3)Math/data tasksgenerates & runs Python (needs Deno)
dspy.RLM(sig, ...)Long context / codebasesrecursive REPL exploration (see dspy-rlm-module)

Typed outputs — use Pydantic on fields, not TypedPredictor

dspy.TypedPredictor is superseded; dspy.Predict now handles Pydantic types natively via field annotations.

from pydantic import BaseModel
from typing import Literal

class Entity(BaseModel):
    name: str
    kind: Literal["person", "org", "place"]

class ExtractEntities(dspy.Signature):
    """Extract named entities from text."""
    text: str = dspy.InputField()
    entities: list[Entity] = dspy.OutputField()

extractor = dspy.Predict(ExtractEntities)

Save & load

Two modes — know the difference:

# State-only (portable JSON; you must rebuild the architecture to load)
program.save("program.json", save_program=False)
new = QAProgram(); new.load("program.json")

# Full program (cloudpickle into a directory; restores everything)
program.save("./program_dir/", save_program=True)
restored = dspy.load("./program_dir/")

Prefer state-only for version control; full-program for deployment artifacts.

Ten anti-patterns to refuse

  1. Hard-coded prompt strings ("You are a helpful assistant...") — write a Signature.
  2. dspy.TypedPredictor(...) in new code — use dspy.Predict with Pydantic fields.
  3. dspy.OpenAI(...) / dspy.settings.configure(...) — use dspy.configure(lm=dspy.LM(...)).
  4. Provider-specific LM classes for built-in providers — use dspy.LM("provider/model"). If DSPy doesn't ship your backend, subclass dspy.BaseLM.
  5. Giant monolithic predictors that do five jobs — decompose into a Module with named sub-predictors.
  6. Mutating signature.instructions by hand — let the optimizer do it.
  7. In-lining few-shot demos in the Signature docstring — bootstrap/optimize them.
  8. Using pickle.dump(program) — use program.save(...).
  9. Setting an LM per module at construction time without reason — configure globally, override only when you need model mixing.
  10. Vague metrics (yes/no, exact-match only) when training an optimizer — see dspy-evaluation-harness.

Configuring the LM

dspy.configure(
    lm=dspy.LM("openai/gpt-4o", temperature=0.0, max_tokens=2000),
    track_usage=True,        # accumulate token counts on predictions
    async_max_workers=4,     # for .acall / batch
)

DSPy 3.2.x warns by default when a module call passes extra input fields or values that don't match the signature's declared types. Treat those warnings as a callsite bug first; if you're intentionally passing pre-serialized values, disable them with dspy.configure(warn_on_type_mismatch=False).

Common provider prefixes: openai/, anthropic/, azure/, vertex_ai/, bedrock/, ollama/. For local Ollama: dspy.LM("ollama_chat/llama3.1:8b", api_base="http://localhost:11434").

Where to go next

  • Measuring quality → dspy-evaluation-harness
  • Automatic optimization → dspy-gepa-optimizer
  • Context >100k tokens → dspy-rlm-module
  • Full pipeline → dspy-advanced-workflow
  • Full API reference → reference.md
  • Runnable example → example_qa.py

DSPy GEPA Optimizer (3.2.x)

GEPA (Genetic-Pareto) is a reflective optimizer: it mutates a program's instructions and few-shots using an LM that reads your metric's textual feedback and proposes improvements. It maintains a Pareto frontier across validation tasks and is the default recommendation for complex DSPy workloads in 2026.

The expansion "Genetic-Evolutionary Prompt Adaptation" that appears in some AI-generated summaries is an LLM-hallucinated backronym. The paper defines GEPA as Genetic-Pareto; the "Pareto" is load-bearing (GEPA keeps a frontier of candidates rather than collapsing to one).

Prerequisites — do these first or GEPA wastes rollouts

  1. A dspy.Module that runs end-to-end (see dspy-fundamentals).
  2. A rich-feedback metric returning dspy.Prediction(score=float, feedback=str) (see dspy-evaluation-harness). Informative feedback can support reflection; evaluate optimizer benefit on the task rather than assuming superiority. A dict with the same fields still crashes dspy.Evaluate under DSPy 3.2.1 — use dspy.Prediction.
  3. trainset and a separate valset. For GEPA, maximize training examples and keep validation just large enough to represent the downstream distribution; do not reuse the same examples for both.
  4. A reflection_lm — a strong LM (often the same or stronger than the task LM) set to temperature=1.0 for creative proposals. Current DSPy docs use a GPT-5-class reflection model with a large output budget.

Canonical call

import dspy

dspy.configure(lm=dspy.LM("openai/gpt-5-mini"))
reflection_lm = dspy.LM("openai/gpt-5", temperature=1.0, max_tokens=32000)

optimizer = dspy.GEPA(
    metric=rich_metric,
    auto="medium",                       # "light" / "medium" / "heavy"
    reflection_lm=reflection_lm,
    reflection_minibatch_size=3,
    candidate_selection_strategy="pareto",  # or "current_best"
    skip_perfect_score=True,
    use_merge=True,
    num_threads=8,
    track_stats=True,
    track_best_outputs=True,             # enables inference-time best-of selection
    log_dir="./gepa_logs",               # resume/checkpoint
    seed=0,
)

optimized = optimizer.compile(
    student=program,
    trainset=trainset,
    valset=valset,
)

# Pareto inspection
pareto = optimized.detailed_results.val_aggregate_scores
print("Pareto frontier:", sorted(pareto, reverse=True)[:5])

optimized.save("optimized_program.json", save_program=False)

Import paths

Either works; use the top-level in new code:

import dspy
dspy.GEPA(...)                              # preferred
# equivalently:
from dspy.teleprompt import GEPA

Metric contract (precise)

import dspy

def rich_metric(gold, pred, trace=None, pred_name=None, pred_trace=None):
    score = ...      # 0.0..1.0
    feedback = ...   # detailed natural-language critique
    return dspy.Prediction(score=score, feedback=feedback)

Return dspy.Prediction, not a dict. Some upstream GEPA prose describes score/feedback as a dict-like shape, but dspy.Evaluate in DSPy 3.2.1 still crashes on a literal dict metric (TypeError: unsupported operand type(s) for +: 'int' and 'dict'). GEPA uses dspy.Evaluate internally for candidate scoring, so a dict return can fail inside GEPA too, not just in your explicit Evaluate(...) calls.

  • pred_name / pred_trace are set during reflection on a specific predictor inside your module — write per-predictor feedback when possible (credit assignment). If you cannot localize feedback, return program-level feedback rather than a vague score-only critique.
  • Feedback quality is the load-bearing part: specifics about why it failed and what good looks like are what the reflection LM acts on.

Budget knobs

Use either auto=... or explicit budget — not both.

ModeRough rolloutsWhen to use
auto="light"~20–40 full evalsSanity-check GEPA works on your metric
auto="medium"~80–150 full evalsEveryday optimization
auto="heavy"~300–600 full evalsFinal run before ship
max_full_evals=NExplicitDeterministic budget
max_metric_calls=NExplicitHard cap on metric invocations (more predictable cost)

Each "full eval" ≈ len(valset) metric calls. Budget accordingly for cost.

Constructor parameters (every one, DSPy 3.2.x)

dspy.GEPA(
    metric,                                  # required
    auto=None,                               # Literal["light","medium","heavy"] | None
    max_full_evals=None,
    max_metric_calls=None,
    reflection_minibatch_size=3,
    candidate_selection_strategy="pareto",   # or "current_best"
    reflection_lm=None,                      # required in practice
    skip_perfect_score=True,
    add_format_failure_as_feedback=False,
    instruction_proposer=None,               # custom ProposalFn
    component_selector="round_robin",        # or a callable
    use_merge=True,
    max_merge_invocations=5,
    num_threads=None,
    failure_score=0.0,
    perfect_score=1.0,
    log_dir=None,
    track_stats=False,
    use_wandb=False,
    wandb_api_key=None,                      # overrides WANDB_API_KEY env var
    wandb_init_kwargs=None,                  # dict forwarded to wandb.init(...)
    track_best_outputs=False,
    warn_on_score_mismatch=True,
    use_mlflow=False,
    seed=0,
    gepa_kwargs=None,                        # e.g. {"use_cloudpickle": True} for dynamic signatures
)

.compile(student, *, trainset, valset=None, teacher=None) — teacher is not currently used.

Data split guidance

DSPy's general prompt-optimizer docs often recommend a validation-heavy split, such as 20% train / 80% validation, because small prompt optimizers can overfit tiny trainsets. GEPA is different: maximize the training set and reserve only enough validation examples to represent downstream behavior. The Pareto frontier still needs a real valset, but GEPA learns from traces and textual feedback on training examples, so starving trainset hurts.

BetterTogether in DSPy 3.2.x

If you want a multi-stage optimizer loop, DSPy 3.2.0's BetterTogether now accepts arbitrary named optimizers instead of the older fixed prompt_optimizer / weight_optimizer pair:

optimizer = dspy.BetterTogether(
    metric=rich_metric,
    bootstrap=dspy.BootstrapFewShotWithRandomSearch(metric=rich_metric),
    gepa=dspy.GEPA(metric=rich_metric, auto="light", reflection_lm=reflection_lm),
)

optimized = optimizer.compile(
    student=program,
    trainset=trainset,
    valset=valset,
    strategy="bootstrap -> gepa",
)

Pass strategy= explicitly when you use named stages like bootstrap=... and gepa=.... DSPy 3.2.0's default strategy is still "p -> w -> p", which only works if your optimizer keys are literally p and w.

Keep plain GEPA as the default first pass. Reach for BetterTogether only when you have a specific reason to chain optimizers and want the valset to pick the best intermediate program.

When GEPA > MIPROv2

  • Your metric can produce specific, teachable critiques (GEPA's superpower).
  • The program has multiple predictors that need targeted improvements (GEPA gives per-predictor feedback; MIPRO doesn't).
  • Rollout budget is small (GEPA converges faster with rich feedback).

When MIPROv2 > GEPA

  • Metric is scalar-only (no signal to reflect on) — use dspy.MIPROv2.
  • You want pure few-shot bootstrapping with no instruction mutation.
  • Very large trainset (500+) where Bayesian search over demos pays off.

When SIMBA is worth trying

dspy.SIMBA is a lighter reflective optimizer. Try it when you want a cheaper reflective pass than GEPA, your program is simple, or you need quick exploration before committing to a full GEPA run. Keep GEPA as the default for multi-predictor programs where per-predictor feedback and Pareto candidate selection matter.

Resume & checkpointing

log_dir writes candidate programs + scores per round. To resume an interrupted run, point log_dir at the same directory — GEPA picks up from the last checkpoint. Inspect <log_dir>/candidates/ to see every proposed program.

Inference-time best-of with track_best_outputs

With track_best_outputs=True, GEPA records, per task, the best prediction seen across all candidates. At inference time on held-out data, you can ensemble or select among the top-Pareto programs for robustness. Access via optimized.detailed_results.best_outputs_valset.

Anti-patterns

  • Float-only metric ("score is 0.7") with no feedback — GEPA collapses to random search.
  • Same set used for train and val — Pareto selection overfits.
  • reflection_lm = small model — it can't critique; use the strongest LM you can afford for this role.
  • Running auto="heavy" on an untested metric — burn money to learn the metric was bugged. Run auto="light" first.
  • Ignoring log_dir — losing a 4-hour run to a disconnect is very painful.

Gotcha: reflection_lm is required at construction, not compile

dspy.GEPA(...) asserts reflection_lm is not None (or a custom instruction_proposer) at init time — you cannot defer it to .compile(). If you see

AssertionError: GEPA requires a reflection language model...

add reflection_lm=dspy.LM("openai/gpt-5", temperature=1.0, max_tokens=32000) to the constructor, or substitute the strongest instruction-following model available on your provider. dspy.LM(...) is a cheap stub until you actually call it, so constructing one doesn't hit the network.

Next

dspy.RLM — Recursive Language Model

dspy.RLM runs the LLM in a sandboxed Python REPL (Pyodide/WASM via Deno) with access to the full context as variables. The LLM writes code to slice, grep, summarize, and recursively sub-query the data, iterating until it can answer. Use it when the context is too large to cram into a single prompt.

Prerequisites

  • Deno installed (for the default PythonInterpreter): brew install deno or see https://deno.land. The interpreter is a Pyodide-in-WASM sandbox spawned by Deno.
  • A sub-LM for inner calls — usually a cheaper model than the outer LM. Defaults to dspy.settings.lm.

Canonical usage

import dspy

dspy.configure(lm=dspy.LM("openai/gpt-4o"))
sub_lm = dspy.LM("openai/gpt-4o-mini")    # cheap inner model

rlm = dspy.RLM(
    "context, query -> answer",
    max_iterations=20,
    max_llm_calls=50,
    max_output_chars=10_000,
    sub_lm=sub_lm,
    tools=[],
    verbose=False,
)

result = rlm(
    context=open("huge_log.txt").read(),   # can be 500k+ tokens
    query="Summarize every unique error class and how many times each appeared.",
)
print(result.answer)

Full constructor

dspy.RLM(
    signature: type[Signature] | str,
    max_iterations: int = 20,       # REPL loop cap
    max_llm_calls: int = 50,        # sub-LM call cap (stops runaway recursion)
    max_output_chars: int = 10_000, # truncate REPL stdout per step
    verbose: bool = False,          # print the REPL trace
    tools: list[Callable] | None = None,
    sub_lm: dspy.LM | None = None,
    interpreter: CodeInterpreter | None = None,  # custom sandbox
)

When to reach for RLM vs. alternatives

SituationUse
Context <100k, answer fits one LM calldspy.Predict / dspy.ChainOfThought
Need external tools (web, db)dspy.ReAct(tools=[...])
Math/code that must rundspy.ProgramOfThought
Huge context, recursive chunking, or data-exploration loopdspy.RLM
Entire-codebase reasoning where the LM should grep/read filesdspy.RLM with file-reading tools=[...]

Composition — RLM as a module inside a larger program

Wrap the RLM in your own dspy.Module and optimize the enclosing program with GEPA. GEPA can tune both the RLM's outer signature instruction and the surrounding predictors.

class RepoAuditor(dspy.Module):
    def __init__(self):
        super().__init__()
        self.explore = dspy.RLM("repo_tree, question -> findings",
                                max_iterations=30, sub_lm=dspy.LM("openai/gpt-4o-mini"))
        self.synth = dspy.ChainOfThought("findings, question -> report")

    def forward(self, repo_tree, question):
        f = self.explore(repo_tree=repo_tree, question=question).findings
        return self.synth(findings=f, question=question)

Then: dspy.GEPA(metric=..., ...).compile(student=RepoAuditor(), trainset=..., valset=...).

Practical tips

  • Budget carefully. A single RLM call can issue dozens of sub-LM calls. Keep max_llm_calls tight (20–50) in production; raise for research.
  • The default stdout cap is smaller in DSPy 3.2.x. max_output_chars now defaults to 10_000; raise it deliberately if your REPL tools print large tables or document slices.
  • Use a cheap sub_lm. The outer LM orchestrates; inner calls (summarize, filter, score) don't need the flagship model.
  • Pass data as kwargs, not in the instruction. rlm(context=huge_string, query="...") lets the REPL treat context as a Python variable. Avoid concatenating it into the prompt.
  • verbose=True while debugging. Prints every REPL step — invaluable when the RLM appears to hang or loop.
  • Custom tools are regular Python callables passed via tools=[...]; they are exposed inside the sandbox. Useful for read_file, grep, vector_search, etc. In DSPy 3.2.x they are invoked by keyword, so give them named, typed parameters rather than positional-only signatures.
  • Deno install is required. Missing Deno is the #1 RLM error. Check which deno before reporting bugs.

Security note

The default interpreter is a Deno-sandboxed Pyodide WASM runtime — no filesystem, network, or subprocess access by default. If you pass custom tools that do I/O, your tools' security posture is yours. Never hand raw subprocess.run to the RLM.

Anti-patterns

  • Using RLM when a 32k-token prompt would fit — overhead is not worth it.
  • Missing Deno → hard-to-diagnose failures. Install it.
  • max_llm_calls left at default in a production path — runaway cost.
  • Passing secrets in the context string — they get echoed into REPL state.

Next

  • Wrap-and-optimize with GEPA → dspy-gepa-optimizer.
  • Full reference → reference.md.

Sponsors

Details

Marketplace
intertwine/dspy-agent-skills
Author
Bryan Young
Category
Llm Frameworks
License
MIT
Keywords
dspy, gepa, rlm, optimization, llm, evaluation