
deciding-with-confidence
Routes, triages, flags and rates a piece of text with a probability for every option: which department or queue a ticket goes to, which intent a message expresses, whether a yes/no condition holds (is this tool call grounded, should the agent ask first, is the customer angry), and how severe or urgent it is on a rubric, each with a calibrated confidence. Runs Claude Haiku in the request and response shapes of OpenAI's Decisions API, or of TypeSafe's System One API (Jev's), for when no purpose-built decision model is at hand. Use for "route this ticket", "triage these", "classify with a confidence", "give me a probability that", "rate the severity", "score against this rubric", guardrail checks before a tool call, "a Decisions API without OpenAI", or a stand-in for Jev.
- version
- 0.4.0
- GitHub stars
- 404
Install
/plugin install deciding-with-confidence@oaustegard-claude-skillsRun it inside Claude Code after adding the marketplace with /plugin marketplace add oaustegard/claude-skills.
Contents
Discovered from sourceSkills 1
Deciding with confidence
scripts/decide.py takes an OpenAI Decisions API request (input plus
questions of type predicate, choice or score) and returns its response
shape: a probability per option, the chosen value or score, and confidence.
Claude Haiku 5.5 answers. Its logits are not exposed, so the distribution is
estimated: a sample states a probability per option, and the result is
temperature-scaled from a labelled eval.
The default is one sample at low effort: one call, the answer, and the
probability that call stated. That confidence is weaker than a pooled one,
because nothing checks it. When the confidence will gate an action, run
--stack: three parallel samples, each under a different angle (plain,
literal, sceptic) and option order, averaged. Their disagreement lowers
confidence and shows up in diagnostics.agreement.
Every sample answers each question with a cue first: the few words of the
evidence that decide it, quoted. Quoting before committing to probabilities
was worth more than any other prompt change tested.
Measured 2026-11-11 through the cli transport at low effort, on four
labelled sets: 104 short banking-intent, guardrail and severity items; the
held-out half of two life-science sets (77 whole research articles, 289
labelled questions, and 204 article sections, 1,457); and the 2,000 test
decisions of the public Typed Decisions benchmark, sent as System One
requests. Jev, TypeSafe's purpose-built decision model, is scored on the same
items, or taken from the benchmark's card:
| short items | whole articles | sections | Typed Decisions | |
|---|---|---|---|---|
| one sample (default) | 0.856 | 0.969 | 0.870 | 0.741 |
--stack | 0.894 | 0.962 | 0.906 | 0.7465 |
| Jev | 0.894 | 0.972 | 0.894 | 0.727 |
| one sample without the cue | 0.827 | 0.931 | 0.857 | — |
Among the 43 general models the Typed Decisions card scores zero-shot, the
stack would rank 10th and one sample 13th; Jev is 18th. The life-science labels come from Sonnet 5.5 and
Opus 5.5 agreeing, which may favour Haiku, and half of the short items were
used to choose the prompt. The stack's main value is its agreement signal:
answers its three samples agreed on were 83% to 98% right, split ones 52% to
73%. One sample costs $0.0005 to $0.0024 a request at list price, the stack
three times that; a stacked request took a median 8.8 s through the CLI, where
Jev answers in under a second. references/method.md has the full tables.
When NOT to use this skill
| Situation | Use instead |
|---|---|
| The label set is too large to list in a prompt (a taxonomy, hundreds of tags) | hallucinating-labels |
| The answer needs reasoning, multi-step work or a written explanation | a normal Claude call; this skill forbids explanation |
| You need extracted fields or free-form JSON | structured outputs on the Messages API |
A real decision model is reachable (OpenAI /v1/decisions, Strands Decider, Jev) and latency matters | that model: 0.1–0.3 s against ~0.7–4 s here |
| Picking which Claude tier runs a subagent | agent-routing |
| The evidence is an image | not supported; the transports send text only |
Abandon a run when diagnostics.agreement is below 1 on most of a batch: the
questions or option descriptions are ambiguous, and more samples will not fix
them. Rewrite the options (distinct, observable criteria) before sampling more.
Procedure
-
Write the request as JSON in the OpenAI shape. Give every question a unique
name. Options needdescriptions that say when each applies; add anotherchoice when the set is not exhaustive. Scorelevelsgo lowest first. When the evidence is a file on disk (a paper, a log, a transcript), give"input_file": "path"in place of"input": the script or the subagent reads it, and you do not write the document out again. When the evidence is this conversation, give"input_transcript": true(or{"last_turns": 20, "max_chars": 60000}): the script reads the transcript Claude Code keeps on disk for the session, renders the last turns with tool calls clipped, and redacts secrets. The file can trail the turn in progress, so do not ask it about a tool result from this same turn.{"input": "I was charged twice for my order.", "questions": [ {"type": "choice", "name": "department", "instructions": "Which department should handle this?", "choices": [{"value": "billing", "description": "Payments, invoices, refunds."}, {"value": "shipping", "description": "Delivery and tracking."}, {"value": "other", "description": "Anything else."}]}, {"type": "predicate", "name": "angry", "instructions": "Is the customer expressing anger?"}, {"type": "score", "name": "severity", "instructions": "How severe is this?", "levels": [{"label": "Cosmetic", "description": "No lost functionality."}, {"label": "Workaround", "description": "Fails, another way works."}, {"label": "Blocked", "description": "Fails with no workaround."}]}]}A predicate may carry
"criteria": {"true": "...", "false": "..."}to say what each answer means; without it the defaults are "the condition holds" and "does not hold".A System One request (the body of TypeSafe's
POST /v1/systemone, which Jev and its clones answer) works as it is:state(text, or any JSON, which the prompt shows as indented JSON) plusquestionskeyed by id, of typenoul,choiceorscore, each withinstructionsandcriteria. The response then comes back in System One's shape: per question id,probabilitieskeyed by option,label,confidence, andprobability_true(noul) orscore(score), plusdiagnostics. A request written for Jev needs no change. -
Pick the transport.
--transport auto(the default) takes the first that works:Transport Needs Wall time per request apipip install anthropicandANTHROPIC_API_KEYorANTHROPIC_AUTH_TOKEN~0.7 s expected (the API time measured inside claude -p)clithe claudeCLI, logged in~4 s, measured bedrockpip install anthropic,AWS_REGION, AWS credentials; explicit onlyas apiWith none of them, use the subagent path in step 3b.
-
Run it.
a. With a transport:
S=/mnt/skills/user/deciding-with-confidence/scripts # or wherever the skill lives python3 $S/decide.py run request.json # one low-effort sample python3 $S/decide.py run request.json --stack # three angles, pooled: use when confidence gates an action echo '{...}' | python3 $S/decide.py run - python3 $S/decide.py batch requests.jsonl --out responses.jsonl # one request per line, resumableFrom Python:
sys.path.insert(0, S); import decide; decide.decide(req)returns the response dict, or None when no sample was usable; passk=3, angles=decide.STACKfor the stack.--anglestakes any comma-separated list, and a request's"angles"may hold instruction text of your own.b. In a Claude Code session, the plugin's
decidersubagent is the cheaper path: it is pinned toclaude-haiku-5-5(the model the calibration was fitted on) and carries its instructions inline, so it answers without a tool call. Install thedeciding-with-confidenceplugin, or copyagents/decider.mdto.claude/agents/; agent definitions load at session start. Then:python3 $S/decide.py emit request.json --stack --agent deciding-with-confidence:decider > batch.jsonUse the name the Agent tool lists:
deciding-with-confidence:deciderfrom the plugin,deciderfrom.claude/agents/. Send eachsamples[i].promptverbatim as an Agent call with thatsubagent_type, all in one message. Collect the reply texts into a JSON list inreplies.json, thenpython3 $S/decide.py aggregate batch.json replies.json. A single reply carries no agreement signal and no calibration untilaggregatehas run. Withinput_filethe emitted prompt names the file and the subagent reads it, so you send a short prompt however long the document is.Without the agent installed, omit
--agent: the system prompt then rides inside each prompt for a stock general-purpose subagent withmodel: haiku. That works and costs about 58K tokens a sample against 9K. -
Read the answer. Predicates give
probability(P true). Choices givechoice,probabilitiesandconfidence. Scores givescore(the probability-weighted level index, so 1.4 sits between levels 1 and 2),probabilitiesandconfidence. Every answer carriesdiagnostics:agreement,spread, the pre-calibrationrawpool,per_sample, andcues, each sample's quote of the evidence it decided on. Read the cues when an answer looks wrong: a cue of "none", or a quote from the wrong part of the evidence, says the sample was guessing. -
Act on it with thresholds, never on the top answer alone.
- With one sample there is no
agreement; use the top-probability rule alone, or rerun the doubtful ones with--stack. agreement < 1(the samples' top answers differ): escalate. With--stack, unanimous answers were 90% to 98% right across the three sets and split ones 63% to 73%.- Top probability under 0.7 (after calibration): escalate. On article sections that sent 17% of one-sample answers onward and left 5.6% of all answers wrong; with the stack, adding the split rule sent 19% and left 3.0% wrong. Both assume the escalation target gets every escalated answer right. Treat 0.7 as a starting point.
- Set the final thresholds from labelled examples of your own traffic, weighing the cost of a false positive against a false negative.
- With one sample there is no
-
Calibrate on your own labels when you have 50 or more per question type:
python3 $S/decide.py eval my_labelled.jsonl --out results.jsonl # {id, input, questions, labels} python3 $S/decide.py calibrate results.jsonl --out my-cal.json export DECIDER_CALIBRATION=$PWD/my-cal.jsonRun
evalwith--stackso the file holds both temperature sets. One sample and a pool of three need different temperatures, because averaging three samples flattens the distribution.calibratefits single-sample temperatures from each request's first sample and pooled ones from the pools, and a type with fewer than 50 labels keeps T = 1. The bundledassets/calibration.jsonwas fitted through theclitransport on the 1,798 labelled answers above. Every temperature is below 1, so Haiku's stated probabilities are underconfident and the calibration sharpens them. Single-sample T is 0.74 for predicates, 0.60 for choices and 0.81 for scores; the pooled T values are 0.44, 0.52 and 0.50.
Earned exceptions
| Rule | Banned when | Earned when |
|---|---|---|
| One sample | the confidence decides whether something is escalated, blocked or sent on | you need the answer, and a wrong one is cheap or gets reviewed anyway |
--stack | cost is the constraint and nothing reads the confidence | the confidence gates an action: parallel samples cost no wall time and 3x the tokens |
| Bundled calibration | your transport or domain differs from the eval's (cli; banking, support and life-science text) | you have not yet labelled 50 items of your own |
Escalate on agreement < 1 | nothing stronger sits behind it | a stronger model or a human can take the split items |
Common failure modes
decide: no usable sample; first error: TypeError: "Could not resolve authentication method..."→ theapitransport found no key. SetANTHROPIC_API_KEY, or pass--transport cli.decide: no transport→ no key and noclaudeon PATH. Use step 3b.diagnostics.samplesbelow k → some replies did not parse. Usually a transport timeout under heavy concurrency; lowerDECIDER_WORKERS(default 8) or raiseDECIDER_TIMEOUT(default 60 s).- Every answer near uniform (confidence around 0.1–0.2 throughout) → options without descriptions, or descriptions that overlap. Rewrite them so each has distinct, observable criteria; more samples will not help.
decide: input_transcript: no session idorno transcript for session→ the script is not running inside a Claude Code session, or the transcript lives elsewhere. Pass{"path": "/path/to/session.jsonl"}.- A subagent reply with prose around the JSON →
aggregatetakes the outermost{...}and copes. A reply with no JSON at all counts as a failed sample. - Transcript text showing up as evidence in the subagent path → the
[no-context]line was dropped from the prompt. Send the emitted prompt verbatim.
Verification
Run the tests, then one live request:
python3 -m pytest /mnt/skills/user/deciding-with-confidence/tests -q # 56 collected, no network
echo '{"input":"Nobody can log in since the 2pm deploy.","questions":[{"type":"predicate","name":"outage","instructions":"Is a service down for users?"}]}' \
| python3 $S/decide.py run -
The live answer should have probability above 0.9 and diagnostics.samples
equal to 1 (3 with --stack). A bad success is an answer with samples: 1
from a --stack run: it looks normal but has lost the agreement signal. Check
samples when a batch runs under load.
Diagnosed failures
- 2026-10-09, first eval: 14 of 520 samples were discarded because Haiku left the options it ruled out of 8-way choices. Missing keys now share the probability mass the sample left unassigned, and the re-run lost none. The prompt also asks for ruled-out options at 0.001.
- 2026-10-09:
claude -p --json-schemadoubled API time (2.2 s against 0.8 s) with a second turn and three times the output tokens. Theclitransport sends the JSON contract in the system prompt instead. - 2026-10-09: the claude-workspace PreToolUse hook that appends
parent-transcript chunks to Agent prompts would have put them in the
evidence.
emitappends[no-context], which that hook honours. - 2026-11-10: the
clitransport passed the sample prompt as a command-line argument, which caps a single argument at about 128 KB and so cannot carry a long document. The prompt now goes in on stdin. - 2026-11-10: a session that pasted a document into each subagent prompt
paid for the document in its own output tokens, once per sample. A request
with
input_fileis emitted as a path for the subagent to read. - 2026-11-10: a decision about the conversation itself meant restating the
conversation in the request.
input_transcriptreads it from the session's transcript file; on a long session one call over the last six turns read 38,580 characters and cost $0.0036. - 2026-11-11: one temperature per question type, fitted on pools, applied to single samples and pools alike. With the cue prompt the stack's pools were further from calibrated (ECE 0.10 to 0.15) than single samples were, because averaging flattens them. Pools now have their own temperatures, and cross-validated ECE is 0.01 to 0.05 for both.
Agents 1
You are a decision model. You read evidence and questions and answer each question with a probability distribution over the options it offers. You never write prose and never ask for clarification. You use a tool in one case only: when the prompt names an evidence file, read that file once, then answer.
Answer at once, the way a trained classifier would: take the reading of the evidence that most people who know the domain would take. Do not reason step by step and do not explain.
For each question:
- Start each question's answer with a
cue: the few words of the evidence that decide the question, quoted exactly, or "none" when nothing in the evidence decides it. The cue is a quotation, not an explanation. The probabilities follow from it; with "none", spread them. - Give every offered option key a probability, including the ones you rule out (give those 0.001). The probabilities for one question sum to 1.
- A probability is how often you would be right if you gave this answer to many inputs like this one. Use 0.97 or higher only when the evidence states the answer outright. Spread probability over two or three options when the input is ambiguous, underspecified, or could fairly be read more than one way. Give an option 0.01 or less only when the evidence rules it out.
- A
predicatequestion has the optionstrueandfalse;truemeans the condition in the instructions holds for the evidence. - A
scorequestion lists ordered levels. Put your probability on the levels whose criteria the evidence meets; adjacent levels can share it. - The order in which options are listed means nothing. Judge each option by its description.
- Set
refusalto true only when answering would itself cause harm. Difficulty or ambiguity is never a reason to refuse; spread the probability instead.
Reply with only this JSON object, no code fence:
{"answers": {"": {"cue": "", "probabilities": {"": , ...}, "refusal": false}, ...}}
Sponsors
Details
- Marketplace
- oaustegard/claude-skills
- Source
- plugins/deciding-with-confidence
- Category
- Agents
- License
- MIT
- Keywords
- deciding-with-confidence, agents
More from oaustegard/claude-skills
All 14- declaudingv0.14.0
- delegating-with-contextv0.3.0
- development-toolsv1.2.1
- environment-and-configv2.0.0
- github-and-gitv2.0.1
- knowledge-and-memoryv1.3.0
- media-processingv2.3.0
- skill-developmentv2.6.0
- social-and-lifestylev1.0.0
- web-and-retrievalv1.0.4
- ai-and-reasoningv2.9.1
- code-intelligencev2.6.0
- data-and-visualizationv1.0.0