
outputai
Output.ai Workflow Assistants
- version
- 0.3.0
- GitHub stars
- 445
Install
/plugin install outputai@outputaiRun it inside Claude Code after adding the marketplace with /plugin marketplace add growthxai/output.
Contents
Discovered from sourceSkills 51
Your task is to implement an Output.ai workflow based on a provided plan document.
The workflow directory is provided as an argument (the workflow directory path). The workflow skeleton should already have been created there; if it has not, create it first.
Please read the plan file and implement the workflow according to its specifications.
Use the todo tool to track your progress through the implementation process.
Implementation Rules
Overview
Implement the workflow described in the plan document, following Output SDK patterns and best practices.
<pre_flight_check>
EXECUTE: Claude Skill: output-meta-pre-flight
</pre_flight_check>
<process_flow>
Step 1: Plan Analysis
Read and understand the plan document.
- Read the plan file from the provided plan file path
- Identify the workflow name, description, and purpose
- Extract input and output schema definitions
- List all required steps and their relationships
- Note any LLM-based steps that require prompt templates
- Understand error handling and retry requirements
Step 2: Workflow Implementation
Update workflow.ts in the workflow directory with the workflow definition.
<implementation_checklist>
- Import required dependencies (workflow, z from '@outputai/core')
- Define inputSchema based on plan specifications
- Define outputSchema based on plan specifications
- Import step functions from steps.ts
- Implement workflow function with proper orchestration
- Handle conditional logic if specified in plan
- Add proper error handling
- When catching a specific step or evaluator error, use
hasErrorType(error, ErrorClass)instead ofinstanceof(seeoutput-error-try-catch) </implementation_checklist>
<workflow_template>
import { workflow, z } from '@outputai/core';
import { stepName } from './steps.js';
const inputSchema = z.object( {
// Define based on plan
} );
const outputSchema = z.object( {
// Define based on plan
} );
export default workflow( {
name: 'workflow-name-from-plan',
description: 'Description from plan',
inputSchema,
outputSchema,
fn: async input => {
// Implement orchestration logic from plan
const result = await stepName( input );
return { result };
}
} );
</workflow_template>
Step 3: Steps Implementation
Update steps.ts in the workflow directory with all step definitions from the plan.
<implementation_checklist>
- Import required dependencies (step, z from '@outputai/core')
- Implement each step with proper schema validation
- Add error handling and retry logic as specified
- Ensure step names match plan specifications
- Add descriptive comments for complex logic </implementation_checklist>
<step_template>
import { step, z } from '@outputai/core';
export const stepName = step( {
name: 'stepName',
description: 'Description from plan',
inputSchema: z.object( {
// Define based on plan
} ),
outputSchema: z.object( {
// Define based on plan
} ),
fn: async input => {
// Implement step logic from plan
return output;
}
} );
</step_template>
Step 3.5: Evaluators Implementation (if needed)
If the plan includes evaluator functions, implement them in evaluators.ts in the workflow directory.
<decision_tree> IF plan_includes_evaluators: CREATE evaluators.ts IMPLEMENT evaluator functions per plan ELSE: SKIP to step 4 </decision_tree>
<implementation_checklist>
- Import required dependencies (evaluator, z, result types from '@outputai/core')
- Import
generateTextandaiSdkfrom@outputai/core/llmif using LLM-powered evaluators - Implement each evaluator with proper schema validation
- Use appropriate result types (EvaluationBooleanResult, EvaluationNumberResult, EvaluationStringResult)
- Include confidence scores (0.0-1.0)
- Add reasoning for transparency
- All imports use .js extension
- Consider offline eval tests for dataset-driven verification (see
output-dev-eval-testingskill) </implementation_checklist>
<evaluator_template>
import { evaluator, z, EvaluationBooleanResult } from '@outputai/core';
export const evaluateName = evaluator( {
name: 'evaluate_name',
description: 'Description from plan',
inputSchema: z.object( {
// Define based on plan
} ),
fn: async input => {
// Implement evaluation logic from plan
return new EvaluationBooleanResult( {
value: true,
confidence: 0.95,
reasoning: 'Explanation of evaluation'
} );
}
} );
</evaluator_template>
Step 4: Prompt Templates (if needed)
If the plan includes LLM-based steps, create prompt templates in the prompts/ subdirectory of the workflow directory.
<decision_tree> IF plan_includes_llm_steps: CREATE prompt_templates UPDATE steps.ts to use loadPrompt and generateText ELSE: SKIP to step 6 </decision_tree>
<llm_step_template>
import { step, z } from '@outputai/core';
import { generateText } from '@outputai/core/llm';
export const llmStep = step( {
name: 'llmStep',
description: 'LLM-based step',
inputSchema: z.object( {
param: z.string()
} ),
outputSchema: z.string(),
fn: async ( { param } ) => {
const { result } = await generateText( {
prompt: 'prompt_name@v1',
variables: { param }
} );
return result;
}
} );
</llm_step_template>
<prompt_file_template>
---
provider: anthropic
# current as of 2026-05-04 — run output-dev-model-selection for the latest
model: claude-sonnet-4-6
temperature: 0.7
---
<assistant>
You are a helpful assistant.
</assistant>
<user>
</user>
</prompt_file_template>
Step 5: README Update
Update README.md in the workflow directory with workflow-specific documentation.
<documentation_requirements>
- Update workflow name and description
- Document input schema with examples
- Document output schema with examples
- Explain each step's purpose
- Provide usage examples
- Document any prerequisites or setup requirements
- Include testing instructions </documentation_requirements>
Step 6: Scenario File Creation
Create at least one scenario file in the scenarios/ subdirectory of the workflow directory for testing the workflow.
<scenario_requirements>
- Create
scenarios/directory if it doesn't exist - Create
test_input.jsonwith valid example input matching the inputSchema - Input values should be realistic and demonstrate the workflow's purpose
- JSON must be valid and parseable </scenario_requirements>
<scenario_template>
{
// Populate with example values matching inputSchema
// Use realistic test data that demonstrates the workflow
}
</scenario_template>
For a workflow with inputSchema: ```typescript z.object( { topic: z.string(), maxLength: z.number().optional() } ) ```Create scenarios/test_input.json:
{
"topic": "The history of artificial intelligence",
"maxLength": 500
}
Step 7: Implementation Validation
Verify the implementation is complete and correct.
<validation_checklist>
- All steps from plan are implemented
- Input/output schemas match plan specifications
- Workflow orchestration logic is correct
- Error handling is in place
- LLM prompts are created (if needed)
- Evaluators are implemented (if specified in plan)
- Evaluators use correct result types and confidence scores
- README is updated with accurate information
- Code follows Output SDK patterns
- TypeScript types are properly defined
- Scenario file exists with valid example input
- Offline eval tests created (if applicable) </validation_checklist>
Step 8: Post-Flight Check
Verify the implementation is ready for use.
<post_flight_check>
EXECUTE: Claude Skill: output-meta-post-flight
</post_flight_check>
</process_flow>
---- START ----
Use the workflow name, workflow directory, and plan file path provided as arguments, along with any additional instructions the user provided.
Viewing and Editing Credentials
When to Use This Skill
- Adding or updating API keys and secrets
- Verifying what credentials are currently stored
- Retrieving a specific credential value
- Checking credential structure before referencing in code
Commands
Edit (Opens $EDITOR)
# Edit global credentials
npx output credentials edit
# Edit environment-specific
npx output credentials edit -e production
npx output credentials edit -e development
# Edit per-workflow credentials
npx output credentials edit -w my_workflow
The file is decrypted to a temp file, opened in $EDITOR, then re-encrypted on save. The temp file is securely wiped (overwritten with null bytes) after closing.
Only production and development are valid environments: the worker picks the file from NODE_ENV and ignores any other name.
Set (Single value, no editor)
# Set a single credential by dot-notation path
npx output credentials set anthropic.api_key sk-ant-...
npx output credentials set openai.api_key sk-... -e production
npx output credentials set stripe.secret_key sk_live_... -w payment_processing
Prefer set in non-interactive sessions, where $EDITOR can't be used. Intermediate keys are created as needed. When the new value would replace an object with a string (or the reverse), set asks for confirmation; pass -y to skip it.
Show (Print to stdout)
# Show global credentials (plaintext — use carefully)
npx output credentials show
# Show environment-specific
npx output credentials show -e production
# Show per-workflow
npx output credentials show -w my_workflow
Get (Single value)
# Get a single credential by dot-notation path
npx output credentials get anthropic.api_key
npx output credentials get aws.region
npx output credentials get stripe.secret_key -w payment_processing
Returns the raw string value (or JSON for nested objects).
YAML Format
Credentials are stored as structured YAML with dot-notation access:
anthropic:
api_key: sk-ant-...
openai:
api_key: sk-...
aws:
region: us-east-1
access_key_id: AKIA...
secret_access_key: ...
stripe:
secret_key: sk_live_...
webhook_secret: whsec_...
Accessing Credentials in Code
import { credentials } from '@outputai/core/credentials';
// Safe read — returns undefined if not found
const region = credentials.get('aws.region');
// Read with default
const region = credentials.get('aws.region', 'us-east-1');
// Strict read — throws MissingCredentialError if not found
const apiKey = credentials.require('anthropic.api_key');
Credential Merging
Per-workflow credentials deep-merge over global credentials at runtime. Workflow values take precedence:
# Global: anthropic.api_key = "sk-ant-global"
# Workflow: anthropic.api_key = "sk-ant-workflow"
# Runtime result: credentials.get('anthropic.api_key') → "sk-ant-workflow"
Troubleshooting
| Error | Cause | Fix |
|---|---|---|
MissingKeyError | Key file not found and env var not set | Run output credentials init or set OUTPUT_CREDENTIALS_KEY |
MissingCredentialError | Path not found in credentials | Add the value with npx output credentials set or edit |
InvalidCredentialsKeyError | Key doesn't match the encrypted file, or the file is corrupted | Use the key the file was encrypted with, or re-init with --force (discards existing values) |
MalformedCredentialsKeyError | Key isn't exactly 64 hex characters | Check the key for typos, whitespace, or truncation |
Verification Checklist
-
npx output credentials showprints expected values -
npx output credentials get anthropic.api_keyreturns the correct key - No empty string values remain for required secrets
- Per-workflow credentials merge correctly with global
Related Skills
output-credentials-init— Create credentials files for the first timeoutput-credentials-env-vars— Automatically wire credentials to env varsoutput-dev-credentials— Full credentials system reference
Credentials as Environment Variables
When to Use This Skill
- Setting up
ANTHROPIC_API_KEYorOPENAI_API_KEYfrom encrypted credentials - Wiring any credential path to a
process.envvariable automatically - Migrating from plaintext
.envsecrets to encrypted credentials - Understanding why an env var is being resolved at worker startup
The credential: Convention
Any env var whose value starts with credential: is resolved from encrypted credentials at worker startup. The format is:
ENV_VAR_NAME=credential:<dot.path>
Example .env
# These are resolved automatically from the global credentials
ANTHROPIC_API_KEY=credential:anthropic.api_key
OPENAI_API_KEY=credential:openai.api_key
# Any credential path works
MY_SERVICE_TOKEN=credential:my_service.token
DATABASE_URL=credential:postgres.url
Encrypted credentials (config/credentials.yml.enc)
anthropic:
api_key: sk-ant-... # → resolves ANTHROPIC_API_KEY
openai:
api_key: sk-... # → resolves OPENAI_API_KEY
my_service:
token: tok_live_... # → resolves MY_SERVICE_TOKEN
postgres:
url: postgres://... # → resolves DATABASE_URL
How It Works
When the worker starts, every env var set to credential:<path> is replaced with the decrypted value at that path. By the time a workflow runs, ANTHROPIC_API_KEY holds the real key and LLM SDKs read it as usual.
Values come from the global credentials for the current environment: config/credentials/<NODE_ENV>.yml.enc when it exists, otherwise config/credentials.yml.enc. Workflow-scoped credentials are never used for env vars, so keep these paths in the global file.
Precedence Rules
Real env var values always take precedence. If ANTHROPIC_API_KEY is already set to a non-credential: value (e.g. from the shell or a CI secret), it is never overwritten:
# Real value - never replaced
ANTHROPIC_API_KEY=sk-ant-real-override
# Placeholder — gets replaced at startup
ANTHROPIC_API_KEY=credential:anthropic.api_key
This means you can override any credential ref at deploy time without changing files.
Setting Up the Convention
Step 1: Initialize credentials (if not done)
npx output credentials init
npx output credentials edit # Add anthropic.api_key, openai.api_key
Step 2: Update .env
# Replace plaintext secrets with credential references
ANTHROPIC_API_KEY=credential:anthropic.api_key
OPENAI_API_KEY=credential:openai.api_key
Step 3: Verify
Start the worker and look for the log line:
[info] Credentials: Resolved credential env vars { vars: [ "ANTHROPIC_API_KEY", "OPENAI_API_KEY" ] }
If the log line lists your env vars, credentials are wired correctly.
Verification Checklist
-
config/credentials.yml.enccontains the target credential paths -
.envusescredential:<path>values for the relevant env vars - Worker startup log shows
Resolved credential env varslisting the expected env vars - First LLM workflow run succeeds (confirming
ANTHROPIC_API_KEYis set correctly) - Setting a real env var in the shell overrides the credential ref
Related Skills
output-credentials-init— Create the encrypted credentials fileoutput-credentials-edit— Add/update credential valuesoutput-dev-credentials— Full credentials system reference
Initializing Credentials
When to Use This Skill
- First time setting up credentials for a project
- Adding production/development environment-specific credentials
- Adding per-workflow credentials that override global ones
- Re-initializing credentials after losing a key file
Overview
The npx output credentials init command generates two files:
- A key file (
.key) — the decryption secret. Never commit this. - An encrypted YAML file (
.yml.enc) — the credentials store. Safe to commit.
Commands
# Global credentials (most common)
npx output credentials init
# Environment-specific (only production and development are read by the worker, based on NODE_ENV)
npx output credentials init -e production
npx output credentials init -e development
# Per-workflow credentials (overrides globals for that workflow)
npx output credentials init -w my_workflow
# Force overwrite existing files
npx output credentials init --force
What Gets Created
Global (default)
config/
├── credentials.key ← Add to .gitignore
└── credentials.yml.enc ← Safe to commit
Environment-specific
config/credentials/
├── production.key ← Add to .gitignore
└── production.yml.enc ← Safe to commit
Per-workflow
src/workflows/{name}/
├── credentials.key ← Add to .gitignore
└── credentials.yml.enc ← Safe to commit
Default Template
After init, the encrypted YAML contains this template:
anthropic:
api_key: ""
openai:
api_key: ""
After Init: Add Your Secrets
npx output credentials edit # Opens $EDITOR with decrypted YAML
Fill in the empty values, save, and close. The file is re-encrypted automatically.
Gitignore Setup
echo "*.key" >> .gitignore
echo "config/credentials.key" >> .gitignore
Or add to your .gitignore:
# Credentials decryption keys — never commit
*.key
config/credentials.key
config/credentials/*.key
src/workflows/*/credentials.key
CI/CD: Key Distribution
In CI/CD pipelines, pass the key as an environment variable instead of committing the file:
# Set in your CI/CD environment
OUTPUT_CREDENTIALS_KEY=<key-value>
# Environment-specific
OUTPUT_CREDENTIALS_KEY_PRODUCTION=<key-value>
# Per-workflow
OUTPUT_CREDENTIALS_KEY_MY_WORKFLOW=<key-value>
The key value is the contents of the .key file.
Verification Checklist
-
config/credentials.keycreated (or env-specific variant) -
config/credentials.yml.enccreated -
.keyfiles added to.gitignore -
npx output credentials editrun to fill in secret values -
npx output credentials showverifies decryption works
Related Skills
output-credentials-edit— Fill in and manage credential valuesoutput-credentials-env-vars— Wire credentials to environment variablesoutput-dev-credentials— Full credentials system reference
Your task is to systematically debug an Output SDK workflow issue in a local development environment.
The arguments the user provided describe the problem they're experiencing, and may include a specific workflow ID.
Use the todo tool to track your progress through the debugging process.
Debugging Process
Overview
Follow a systematic approach to identify and resolve workflow execution issues: verify infrastructure, gather evidence, analyze traces, and apply targeted fixes.
<pre_flight_check>
EXECUTE: Claude Skill: output-meta-pre-flight
</pre_flight_check>
<process_flow>
Step 1: Verify Services Running
Before debugging, confirm that all required services are operational. The output-services-check skill provides comprehensive guidance.
<verification_commands>
# Check Docker containers are running
docker ps | grep output
# Verify Output services respond
curl -s http://localhost:3001/health || echo "API not responding"
# Check Temporal UI is accessible
curl -s http://localhost:8080 > /dev/null && echo "Temporal UI accessible" || echo "Temporal UI not accessible"
</verification_commands>
<decision_tree> IF docker_not_running: RUN: docker compose up -d WAIT: for services to start (30-60 seconds) IF output_dev_not_running: RUN: npx output dev WAIT: for services to initialize IF all_services_running: PROCEED: to step 2 </decision_tree>
Expected State:
- Docker containers for
outputare running - API server responds at
http://localhost:3001 - Temporal UI accessible at
http://localhost:8080
Step 2: List Workflow Runs
Identify the failing workflow execution by listing recent runs. The output-workflow-runs-list skill provides detailed filtering guidance.
<list_commands>
# List all recent workflow runs
npx output workflow runs list
# Filter by specific workflow type (if known)
npx output workflow runs list <workflowName>
# Get detailed JSON output for analysis
npx output workflow runs list --json
# Limit results to most recent
npx output workflow runs list --limit 10
</list_commands>
<identification_criteria> Look for:
- Status: FAILED or TERMINATED
- Recent timestamp matching when the issue occurred
- Workflow type matching the problem description </identification_criteria>
<decision_tree>
IF user_provided_workflow_id:
USE: provided workflow ID
PROCEED: to step 3
IF failed_runs_found:
SELECT: most recent failed run
NOTE: workflow ID from output
PROCEED: to step 3
IF no_runs_found:
CHECK: workflow exists with npx output workflow list
IF workflow_not_found:
REPORT: workflow doesn't exist
SUGGEST: verify workflow name and location
ELSE:
SUGGEST: run the workflow with npx output workflow run <name>
</decision_tree>
Step 3: Debug Specific Workflow
Read and analyze the Temporal history of the identified workflow. The output-workflow-trace skill provides analysis techniques.
<debug_commands>
# One JSON document: an entry per workflow (the run, then its children) with its Temporal history events
npx output workflow debug <workflowId>
# A specific run, or another Temporal
npx output workflow debug <workflowId> --run-id <runId>
npx output workflow debug <workflowId> --profile <name>
</debug_commands>
Tip: The output is always JSON, so pipe it to jq. Check complete first (false means part of the tree was not read, see warnings). Long values are truncated with a marker; the full payload is in the Temporal UI.
<analysis_checklist>
- Identify which step failed
- Examine the error message and stack trace
- Check input data passed to the failing step
- Check output data from preceding steps
- Look for patterns matching common error types </analysis_checklist>
<temporal_ui_guidance> For visual workflow inspection, open the Temporal Web UI at http://localhost:8080:
- Find your workflow execution by ID
- View the event history timeline
- Inspect individual step inputs and outputs </temporal_ui_guidance>
Step 4: Suggest Fixes
Based on the history analysis, identify the error pattern and suggest targeted fixes. Claude will invoke the relevant error skill based on symptoms.
<error_matching>
| Symptom | Skill |
|---|---|
| "incompatible schema" errors, type errors | output-error-zod-import |
| Replay failures, inconsistent results | output-error-nondeterminism |
| Retries not working, errors swallowed | output-error-try-catch |
| Type errors, undefined properties at step boundaries | output-error-missing-schemas |
| Workflow hangs, determinism errors | output-error-direct-io |
| Untraced requests, axios errors | output-error-http-client |
</error_matching>
<decision_tree> IF error_matches_known_pattern: INVOKE: relevant error skill for detailed fix ELSE: CONSULT: workflow-quality subagent for additional patterns SUGGEST: Manual trace inspection in Temporal UI </decision_tree>
After applying fix: ```bash # Re-run the workflow to verify npx output workflow run --input ''Or start asynchronously and check result
npx output workflow start --input '' npx output workflow status npx output workflow result
Or, if the fix only affects a specific step and earlier steps succeeded,
re-run from after the last known-good step (skips re-executing earlier work)
npx output workflow reset --step --reason ""
For targeted rerun after fixing a downstream step, see the `output-workflow-reset` skill.
</verification>
</step>
</process_flow>
<post_flight_check>
EXECUTE: Claude Skill: `output-meta-post-flight`
</post_flight_check>
---- START ----
Use the problem description and any optional workflow ID the user provided.
Using the Agent Class
Overview
The Agent class uses an internal AI SDK ToolLoopAgent through composition with Output prompt files and the skills system. It does not inherit from ToolLoopAgent. Use it when you need multi-step tool execution, conversation history, or a reusable agent instance. For single-shot LLM calls without tools, generateText is simpler.
When to Use This Skill
- Building multi-step agents that call tools in a loop
- Using skills (lazy-loaded instructions) with an agent
- Creating agents with structured output via
aiSdk.Output.object() - Implementing stateful conversations with
messageStore - Streaming Agent progress with
onChunk - Deciding between
AgentandgenerateText
Import Pattern
import { Agent, aiSdk } from '@outputai/core/llm';
import type { MessageStore } from '@outputai/core/llm';
import { z } from '@outputai/core';
Agent comes from @outputai/core/llm. Use aiSdk.Output for structured output. Import z from @outputai/core (never from zod directly). MessageStore is the type for a pluggable getMessages / addMessages store; implement it yourself.
Construction
The prompt file is loaded and rendered at construction time. Variables and tools are fixed at construction. Skills and maxSteps come from the prompt file. The agent is ready to call generate(), generateWithStreaming(), or stream() immediately.
const agent = new Agent( {
prompt: 'writing_assistant@v1',
variables: {
content_type: input.contentType,
focus: input.focus,
content: input.content
},
output: aiSdk.Output.object( { schema: reviewSchema } )
} );
Constructor Options
| Option | Type | Default | Description |
|---|---|---|---|
prompt | string | (required) | Prompt file name (e.g. 'writing_assistant@v1') |
promptDir | string | - | Override the stack-resolved prompt directory |
variables | PromptVariables | - | Template variables rendered at construction |
tools | AI SDK tools | - | Caller tools; merged with prompt YAML tools (load_skill last) |
stopWhen | function or function[] | - | Custom stop condition (overrides prompt maxSteps when tools exist) |
output | aiSdk.Output | - | Structured output spec (e.g. aiSdk.Output.object({ schema })) |
messageStore | MessageStore | - | Pluggable store for multi-turn history |
generate()
Run the agent and return when complete:
const result = await agent.generate();
console.log( result.text ); // Generated text
console.log( result.output ); // Structured output (when using aiSdk.Output.object)
console.log( result.usage ); // Token counts
The result has the same shape as generateText: text, result (alias for text), output, usage, finishReason, toolCalls, etc.
Passing Additional Messages
Extend the conversation with extra messages:
const result = await agent.generate( {
messages: [ { role: 'user', content: 'Focus on the introduction section.' } ]
} );
Messages are appended after the initial prompt messages (and any message-store history). You can also pass abortSignal and toolChoice.
generateWithStreaming()
Use generateWithStreaming() when you need progress callbacks and a complete result:
const result = await agent.generateWithStreaming( {
onChunk( { chunk } ) {
if ( chunk.type === 'text-delta' ) {
process.stdout.write( chunk.text );
}
}
} );
The method behaves like generate() while using streaming internally. It returns the complete response, rejects on stream errors, and automatically appends messages to the configured message store. It accepts the same messages, abortSignal, and toolChoice as generate(), plus onChunk. Prefer it over stream() in Temporal activity steps unless direct access to the stream result is required.
stream()
Use stream() when direct control over textStream or stream is required. It accepts the same messages, abortSignal, and toolChoice as generate(), plus onChunk, onEnd, and onError:
const stream = await agent.stream();
for await ( const chunk of stream.textStream ) {
process.stdout.write( chunk );
}
Like streamText, the stream result provides textStream and stream iterables, plus promise-based properties (text, usage, finishReason) that resolve on completion.
stream() appends messages to the message store in its wrapped onEnd when finishReason is not 'error'. See output-dev-llm-streaming for streaming and error-handling guidance.
Structured Output
Use aiSdk.Output.object() to get typed responses:
const reviewSchema = z.object( {
issues: z.array( z.string() ).describe( 'List of issues found' ),
suggestions: z.array( z.string() ).describe( 'Actionable suggestions' ),
score: z.number().describe( 'Quality score 0-100' ),
summary: z.string().describe( 'Brief overall assessment' )
} );
const agent = new Agent( {
prompt: 'writing_assistant@v1',
variables: { content_type: 'documentation', focus: 'clarity', content: markdownContent },
output: aiSdk.Output.object( { schema: reviewSchema } )
} );
const { output } = await agent.generate();
// output: { issues: string[], suggestions: string[], score: number, summary: string }
Use .describe() on schema fields instead of .min()/.max() for number constraints. Anthropic does not support minimum/maximum JSON Schema constraints in tool definitions.
Message Store
By default, Agent is stateless. Each generate() call starts fresh with only the initial prompt messages. Pass a messageStore to maintain history across calls:
import { Agent } from '@outputai/core/llm';
import type { MessageStore } from '@outputai/core/llm';
const messages: Parameters<MessageStore['addMessages']>[0] = [];
const messageStore: MessageStore = {
getMessages: () => messages,
addMessages: incoming => {
messages.push( ...incoming );
}
};
const chatbot = new Agent( {
prompt: 'chatbot@v1',
messageStore
} );
const r1 = await chatbot.generate( {
messages: [ { role: 'user', content: 'Hello, tell me about Output.' } ]
} );
// r1.text: "Output is an AI framework for..."
const r2 = await chatbot.generate( {
messages: [ { role: 'user', content: 'How does it handle retries?' } ]
} );
// r2 sees the full history from r1
MessageStore is:
interface MessageStore {
getMessages(): ModelMessage[] | Promise<ModelMessage[]>;
addMessages( messages: ModelMessage[] ): void | Promise<void>;
}
ModelMessage is an AI SDK type available through the aiSdk namespace or as an import from ai. There is no built-in store. Implement the interface in memory for a single process, or with your database for durable history.
Using Agent in Workflow Steps
In workflow steps, construct a new Agent per invocation. Variables come from the step input:
import { step, z } from '@outputai/core';
import { Agent, aiSdk } from '@outputai/core/llm';
const reviewSchema = z.object( {
summary: z.string().describe( 'Brief assessment' ),
issues: z.array( z.string() ).describe( 'Problems found' ),
suggestions: z.array( z.string() ).describe( 'Improvements' ),
score: z.number().describe( 'Quality score 0-100' )
} );
export const reviewContent = step( {
name: 'reviewContent',
description: 'Review technical content using Agent with structured output',
inputSchema: z.object( {
content: z.string().describe( 'The content to review' ),
content_type: z.string().describe( 'Type of content' ),
focus: z.string().describe( 'Review focus areas' )
} ),
outputSchema: reviewSchema,
fn: async input => {
const agent = new Agent( {
prompt: 'writing_assistant@v1',
variables: input,
output: aiSdk.Output.object( { schema: reviewSchema } )
} );
const { output } = await agent.generate();
return output;
}
} );
This is the standard pattern. Each step invocation is independent, and Agent construction is cheap.
Using Agent with Skills
List skill paths in the prompt frontmatter. See output-dev-skill-file for the full skills guide.
When to Use Agent vs generateText
generateText | Agent | |
|---|---|---|
| Best for | Single-shot LLM calls | Multi-step tool loops |
| Tools | Supported | Supported |
| Skills | Supported | Supported |
| Conversation history | Manual | Built-in with messageStore |
| Reusable instance | No (function call) | Yes (construct once, call many) |
| Structured output | aiSdk.Output.object() | aiSdk.Output.object() |
Start with generateText. Move to Agent when you need conversation state or a reusable instance with a fixed configuration.
generateText Example (for comparison)
import { generateText } from '@outputai/core/llm';
const { result } = await generateText( {
prompt: 'generate_summary@v1',
variables: {
company_name: input.name,
website_content: input.websiteContent
}
} );
Verification Checklist
- Import
Agentfrom@outputai/core/llm(not fromaidirectly) - Import
zfrom@outputai/core(never fromzod) - Prompt file exists in
prompts/folder - Variables match
{{ variable }}placeholders in the prompt - Prompt frontmatter sets
maxStepswhen skills or tools need a ceiling other than 10 -
aiSdk.Output.object({ schema })uses.describe()not.min()/.max()on numbers -
messageStoreis only used when multi-turn history is needed - Agent is constructed inside the step
fn(not at module level) for workflow steps - Prefer
generateWithStreaming()when callbacks are sufficient
Related Skills
output-dev-skill-file- Creating skill files for agentsoutput-dev-llm-streaming- Streaming progress and Temporal-safe error handlingoutput-dev-prompt-file- Creating .prompt files used by agentsoutput-dev-step-function- Using agents in step functionsoutput-dev-types-file- Defining Zod schemas for structured outputoutput-dev-workflow-function- Orchestrating agent-powered steps
Code Style Conventions
Overview
Generated code must match the style of the project it lives in. Many projects use their own linter or formatter (ESLint, Prettier, Biome, etc.) with rule sets that differ from Output's defaults. Always discover and follow the project's rules first. Only fall back to the Output SDK conventions below when the project has no linter or formatter configured.
When to Use This Skill
- Writing any TypeScript or JavaScript code in a workflow project
- Reviewing generated code before delivery
- Fixing lint errors after generation
Rules Discovery (do this first)
Before writing or reviewing code, determine the project's style rules:
- Check for a linter config. Look for
eslint.config.js,.eslintrc.*,biome.json,deno.json, or similar in the project root. - Check for a formatter config. Look for
.prettierrc*,.editorconfig, or formatter settings inpackage.json. - Check for lint/format scripts. Look in
package.jsonforlint,lint:fix,format, or similar scripts. - Read existing source files. If no config exists, infer conventions from the existing code (comma style, quote style, indentation, semicolons, spacing).
If the project has a linter or formatter
- Follow its rules. Do not apply Output SDK conventions that conflict.
- Run the project's lint/format command after generating code.
- If a rule is ambiguous, match the patterns in existing source files.
If the project has no linter or formatter
- Apply the Output SDK Default Conventions below.
- Match any conventions already present in existing source files -- consistency with the repo takes priority over the defaults below.
Output SDK Default Conventions
These rules reflect the Output SDK's own ESLint config. Apply them only when the project has no linter configured and no conflicting conventions are evident in existing code.
No Trailing Commas
Never use trailing commas in objects, arrays, function parameters, or type definitions.
// CORRECT
const config = {
name: 'workflow',
timeout: 30000
};
const items = [ 'a', 'b', 'c' ];
export const myStep = step( {
name: 'myStep',
inputSchema: MyInputSchema,
outputSchema: MyOutputSchema,
fn: async input => {
return { result: input.value };
}
} );
// WRONG - trailing commas
const config = {
name: 'workflow',
timeout: 30000, // <-- not allowed
}
const items = [ 'a', 'b', 'c', ] // <-- not allowed
No let Declarations
let is banned. Use const exclusively. When a value needs conditional assignment, use a ternary, an IIFE, or restructure the logic.
// CORRECT - ternary
const label = count > 1 ? 'items' : 'item';
// CORRECT - named helper for complex cases
const fetchWithFallback = async url => {
try {
return await fetchContent( url );
} catch {
return '[Content unavailable]';
}
};
const content = await fetchWithFallback( url );
// CORRECT - early return in a function
function resolve( input ) {
if ( input.mode === 'fast' ) {
return fastPath( input );
}
return standardPath( input );
}
// WRONG
let content; // <-- banned
try {
content = await fetchContent( url );
} catch {
content = '[Content unavailable]';
}
let label; // <-- banned
if ( count > 1 ) {
label = 'items';
} else {
label = 'item';
}
Arrow Parens Only When Needed
Single-parameter arrow functions must not have parentheses. Use parens only for zero, multiple, destructured parameters, or when a TypeScript return type annotation is present.
// CORRECT
items.map( item => item.id )
items.filter( s => s.url )
items.forEach( x => console.log( x ) )
fn: async input => { ... }
// Parens required for these cases:
items.reduce( ( acc, item ) => acc + item, 0 )
const run = ( { name, id } ) => `${name}-${id}`;
const noop = () => {};
fn: async ( input ): Promise<WorkflowOutput> => { ... } // return type annotation
// WRONG - unnecessary parens on single param
items.map( ( item ) => item.id )
items.filter( ( s ) => s.url )
fn: async ( input ) => { ... }
prefer-const
Always use const. If a binding is never reassigned, it must be const.
Operator Linebreak After
When an expression spans multiple lines, the operator stays on the first line.
// CORRECT
const result = longExpression +
anotherExpression;
const isValid = conditionA &&
conditionB &&
conditionC;
const value = condition ?
trueResult :
falseResult;
// WRONG - operator on next line
const result = longExpression
+ anotherExpression;
Spacing
- Space in parens:
fn( x )notfn(x), except empty parensfn() - Space in brackets:
[ 'a', 'b' ]not['a', 'b'] - Space in braces:
{ key: value }not{key: value} - Indent: 2 spaces
- Quotes: single quotes
- Semicolons: always
File and Folder Naming
- All file names:
snake_case(e.g.,fetch_data.ts,html_renderer.ts) - All folder names:
snake_case(e.g.,ai_hn_digest,shared_utils) - Exceptions: config files (
vitest.config.js,eslint.config.js)
Quick Reference (Output SDK defaults)
| Rule | Correct | Wrong |
|---|---|---|
| Trailing comma | { a: 1 } | { a: 1, } |
| Variable declaration | const x = 1 | let x = 1 |
| Single-param arrow | x => x.id | ( x ) => x.id |
| Operator linebreak | a +\n b | a\n + b |
| Parens spacing | fn( x ) | fn(x) |
Verification
- If the project has a lint command (
npm run lint,npx eslint, etc.), run it and fix any violations. - If the project has a format command (
npm run format,npx prettier --write, etc.), run it. - If neither exists, review the generated code against existing source files in the repo for consistency.
Cost Observability Hooks
Overview
This skill documents how to subscribe to Output's cost events so every priced LLM call and every HTTP call with attached cost (via addRequestCost, see output-dev-http-client-create) can be forwarded to your own observability system — a webhook, a structured log pipeline, a metrics backend, etc. Handler errors are caught and logged by the framework; they never affect the workflow or the request that triggered them.
When to Use This Skill
- Forwarding per-call spend to an external observability/webhook endpoint
- Logging cost events as structured fields for a log platform
- Building alerting or dashboards on top of workflow spend
- Auditing which workflows/activities are driving cost, in real time rather than after the fact
This skill is about project-wide hook registration for cost data already emitted by the framework. To make an HTTP client emit cost in the first place, see output-dev-http-client-create.
Setup
1. Create a hook file
// src/cost_hooks.ts
import { on } from '@outputai/core/hooks';
import type { HttpRequestCostEvent } from '@outputai/core/http';
import type { LLMGenerationMeteringEvent } from '@outputai/core/llm';
on<HttpRequestCostEvent>('cost:http:request', async event => {
// handle HTTP cost
});
on<LLMGenerationMeteringEvent>('llm:generation:metering', async event => {
// handle LLM cost
});
2. Register the file
Add the file to outputai.hookFiles in package.json, alongside any existing hook files. Paths are relative to the package root and point at the built .js output — npm run output:worker:build compiles src/ to dist/ (per tsconfig.json's rootDir/outDir), so src/cost_hooks.ts is registered as dist/cost_hooks.js, not the .ts source itself. The worker loads these files at startup.
{
"outputai": {
"hookFiles": [
"./dist/cost_hooks.js"
]
}
}
If you'd rather skip the build step, a hook file can also be plain, uncompiled JavaScript registered directly at its src/ path ("./src/cost_hooks.js") — see https://docs.output.ai/operations/error-hooks for that variant. The rest of this skill uses TypeScript + the built-output path, since that's what the framework's own examples (and every other file in a scaffolded project) use.
Events You Can Subscribe To
| Event | Type import | When it fires | Prefer for |
|---|---|---|---|
llm:generation:metering | LLMGenerationMeteringEvent from @outputai/core/llm | After every LLM generation (text, image, Agent, streaming) that reports usage — including failed calls that got at least partial usage | New LLM cost integrations |
cost:llm:request | LLMUsageEvent from @outputai/core/llm | Legacy/compatible LLM cost event, same completion path | Existing handlers only — do not use for new work |
cost:http:request | HttpRequestCostEvent from @outputai/core/http | Only when your code (or a client's afterResponse hook) calls addRequestCost(response, total) | Non-LLM paid API calls |
Every event carries the same envelope: eventId (UUID v4, stable idempotency key), eventDate (ms epoch), activityInfo and workflowDetails (present when emitted from within a step/evaluator), outputActivityKind, and payload (the event-specific data described above).
Pattern 1: Forward to an external endpoint
Use this when spend needs to reach an external observability system over HTTP. Forward the raw envelope plus payload; redact anything that might carry secrets (API keys or tokens embedded in query strings) before logging or sending the URL.
// src/cost_hooks.ts
import { on } from '@outputai/core/hooks';
import { credentials } from '@outputai/core/credentials';
import type { HttpRequestCostEvent } from '@outputai/core/http';
import type { LLMGenerationMeteringEvent } from '@outputai/core/llm';
// Use plain fetch here, not createKyClient/outputFetch. Handlers run inside
// the emitting step's async context, so a traced client would add its own
// HTTP trace event to that step's trace on every forwarded cost event.
//
// Read the credential lazily with `get`, not `require`, at module scope: a
// hook file is imported at worker startup by an unguarded `await import()`
// (there's no try/catch around it), so a `require()` that throws here takes
// the whole worker down. `get`/`require` also only see workflow-scoped
// credentials from inside an activity — at startup, outside any activity,
// they resolve the global credential set only, so keep this in a global
// (not per-workflow) credential file.
const getWebhookUrl = (): string | undefined => credentials.get('observability.webhook_url') as string | undefined;
const postEvent = async (json: Record<string, unknown>): Promise<void> => {
const webhookUrl = getWebhookUrl();
if (!webhookUrl) {
return;
}
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 5000);
try {
await fetch(`${webhookUrl}/events`, {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify(json),
signal: controller.signal
});
} finally {
clearTimeout(timeout);
}
};
// Strip query strings — some APIs put API keys or tokens there.
const redactUrl = (url: string): string => {
try {
const parsed = new URL(url);
return `${parsed.origin}${parsed.pathname}`;
} catch {
return '[unparseable-url]';
}
};
on<HttpRequestCostEvent>('cost:http:request', async event => {
if (!event.workflowDetails || !event.payload) {
return;
}
// A caught failure here logs with this handler's own context; an
// uncaught one is still caught and logged by the framework either way.
try {
await postEvent({
eventId: event.eventId,
eventDate: event.eventDate,
workflowId: event.workflowDetails.workflowId,
kind: 'http',
url: redactUrl(event.payload.url),
totalUsd: event.payload.total
});
} catch (error) {
console.warn('cost_hooks: failed to forward HTTP cost event', error);
}
});
on<LLMGenerationMeteringEvent>('llm:generation:metering', async event => {
if (!event.workflowDetails || !event.payload) {
return;
}
try {
await postEvent({
eventId: event.eventId,
eventDate: event.eventDate,
workflowId: event.workflowDetails.workflowId,
kind: 'llm',
providerId: event.payload.usage.providerId,
modelId: event.payload.usage.modelId,
promptName: event.payload.usage.promptName,
requestIds: event.payload.usage.requestDetails?.ids ?? [],
durationMs: event.payload.usage.requestDetails?.duration,
totalUsd: event.payload.cost?.total ?? null
});
} catch (error) {
console.warn('cost_hooks: failed to forward LLM cost event', error);
}
});
Pattern 2: Structured logging
Use this when spend just needs to land in your log platform as structured facets, without a separate network call. Use the framework Logger so fields are emitted consistently with the rest of the worker's logs.
// src/cost_hooks.ts
import { on } from '@outputai/core/hooks';
import { Logger } from '@outputai/core';
import type { HttpRequestCostEvent } from '@outputai/core/http';
import type { LLMGenerationMeteringEvent } from '@outputai/core/llm';
const log = Logger.createLogger('CostObservability');
const redactUrl = (url: string): string => {
try {
const parsed = new URL(url);
return `${parsed.origin}${parsed.pathname}`;
} catch {
return '[unparseable-url]';
}
};
on<HttpRequestCostEvent>('cost:http:request', async event => {
if (!event.workflowDetails || !event.payload) {
return;
}
log.info('http_request_cost', {
eventId: event.eventId,
workflowId: event.workflowDetails.workflowId,
url: redactUrl(event.payload.url),
totalUsd: event.payload.total
});
});
on<LLMGenerationMeteringEvent>('llm:generation:metering', async event => {
if (!event.workflowDetails || !event.payload) {
return;
}
log.info('llm_generation_cost', {
eventId: event.eventId,
workflowId: event.workflowDetails.workflowId,
providerId: event.payload.usage.providerId,
modelId: event.payload.usage.modelId,
promptName: event.payload.usage.promptName,
requestIds: event.payload.usage.requestDetails?.ids ?? [],
durationMs: event.payload.usage.requestDetails?.duration,
totalUsd: event.payload.cost?.total ?? null
});
});
Handler Safety Rules
- Handlers are wrapped in try/catch by the framework. A thrown or rejected handler is caught and logged (e.g.
<eventName> hook error) and never affects the workflow, the worker, or the request that triggered the event — the same guaranteeonErrorhandlers get. Adding your owntry/catchis still worthwhile so a forwarding failure logs with context you control, but it's not required for safety. - Skip incomplete events. Guard on
event.workflowDetailsandevent.payloadbefore using them — they're only populated when the event was emitted from within a step/evaluator context;eventIdandeventDate, by contrast, are always present. - Redact before logging or sending URLs. Some third-party APIs embed API keys or tokens in query strings. Strip the query string (and any path segments you know carry secrets) before it leaves the process.
- Use
eventIdas an idempotency key when forwarding to a system that might receive the same event more than once (e.g. retried delivery). - Expect one event per attempt. A step or evaluator that retries emits one cost event per attempt — that's correct, since each attempt is a real billed call.
- Group by prompt and correlate with the provider using
requestDetails.payload.usageandpayload.costboth carrypromptNameandrequestDetails(ids,count,durationin milliseconds).idsholds the provider request IDs in request order and is empty when the provider exposes none (for example Google Vertex), so do not assumeids.length === count. Both fields are optional in the types because traces recorded before they existed lack them; events always carry them. - Prefer
llm:generation:meteringovercost:llm:requestfor new integrations; the legacy event's shape is frozen and misses newer cost types (e.g. tool/grounding charges).
Verification Checklist
- Hook file registered in
outputai.hookFilesinpackage.json, pointing at the built.jspath (dist/...), not the.tssource - Hook file imports
onfrom@outputai/core/hooks - Handlers guard on missing
workflowDetails/payload - URLs are redacted (query string stripped) before logging or forwarding
-
llm:generation:meteringused instead of legacycost:llm:requestfor new work
Related Skills
output-dev-http-client-create- Attaching cost to a paid API client withaddRequestCostoutput-dev-workflow-cost- Post-hoc cost calculation for a single completed workflow run via the CLIoutput-dev-credentials- Storing the observability endpoint URL/token as a credential
Generate Workflow Skeleton with Output SDK CLI
Overview
This skill documents how to use the Output SDK CLI to generate a workflow skeleton. The skeleton provides a starting point with all required files and proper structure.
When to Use This Skill
- Starting a new workflow from scratch
- Understanding what files are needed for a workflow
- Scaffolding the basic structure before implementation
- Learning the Output SDK workflow patterns
CLI Command
npx output workflow generate --skeleton
This command creates the basic file structure for a new workflow.
Generated File Structure
After running the skeleton generator, you will have:
src/workflows/{workflow-name}/
├── workflow.ts # Main workflow definition
├── steps.ts # Step function definitions
├── types.ts # Zod schemas and types
├── prompts/ # Empty folder for prompt files
└── scenarios/ # Empty folder for test scenarios
Project Structure Overview
The skeleton is created within the standard Output SDK project structure:
src/
├── shared/ # Shared code (create if needed)
│ ├── clients/ # API clients
│ ├── utils/ # Utility functions
│ ├── services/ # Business logic services
│ ├── steps/ # Shared steps (optional)
│ └── evaluators/ # Shared evaluators (optional)
└── workflows/
└── {workflow-name}/ # Your new workflow
├── workflow.ts
├── steps.ts
├── types.ts
├── prompts/
└── scenarios/
Post-Generation Steps
Step 1: Review Generated Files
After generation, review each file to understand the template structure:
workflow.ts - Contains a basic workflow template:
import { workflow, z } from '@outputai/core';
import { exampleStep } from './steps.js';
import { WorkflowInputSchema } from './types.js';
export default workflow( {
name: 'workflowName',
description: 'Workflow description',
inputSchema: WorkflowInputSchema,
outputSchema: z.object( { result: z.string() } ),
fn: async input => {
const result = await exampleStep( input );
return { result };
}
} );
steps.ts - Contains example step template:
import { step, z } from '@outputai/core';
import { ExampleStepInputSchema } from './types.js';
export const exampleStep = step( {
name: 'exampleStep',
description: 'Example step description',
inputSchema: ExampleStepInputSchema,
outputSchema: z.object( { result: z.string() } ),
fn: async input => {
// Implement step logic here
return { result: 'example' };
}
} );
types.ts - Contains schema definitions:
import { z } from '@outputai/core';
export const WorkflowInputSchema = z.object( {
// Define input fields
} );
export type WorkflowInput = z.infer<typeof WorkflowInputSchema>;
Step 2: Customize the Workflow Name
- Update the folder name to match your workflow
- Update the
nameproperty inworkflow.ts - Follow naming conventions:
- Folder:
snake_case(e.g.,image_processor) - Workflow name:
camelCase(e.g.,imageProcessor)
- Folder:
Step 3: Define Your Schemas
In types.ts, define your actual input/output schemas:
import { z } from '@outputai/core';
export const WorkflowInputSchema = z.object( {
content: z.string().describe( 'Content to process' ),
options: z.object( {
format: z.enum( [ 'json', 'text' ] ).default( 'json' )
} ).optional()
} );
export type WorkflowInput = z.infer<typeof WorkflowInputSchema>;
export type WorkflowOutput = { processed: string };
Related Skill: output-dev-types-file
Step 4: Implement Your Steps
Replace the example step with your actual step implementations:
import { step, z, FatalError, ValidationError } from '@outputai/core';
import { ProcessContentInputSchema } from './types.js';
export const processContent = step( {
name: 'processContent',
description: 'Process the input content',
inputSchema: ProcessContentInputSchema,
outputSchema: z.object( { processed: z.string() } ),
fn: async ( { content } ) => {
// Implement your logic
return { processed: content.toUpperCase() };
}
} );
Related Skill: output-dev-step-function
Step 5: Update the Workflow
Wire up your steps in the workflow:
import { workflow, z } from '@outputai/core';
import { processContent } from './steps.js';
import { WorkflowInputSchema } from './types.js';
export default workflow( {
name: 'contentProcessor',
description: 'Process content with custom logic',
inputSchema: WorkflowInputSchema,
outputSchema: z.object( { processed: z.string() } ),
fn: async input => {
const result = await processContent( { content: input.content } );
return result;
}
} );
Related Skill: output-dev-workflow-function
Step 6: Add Prompts (If Needed)
If your workflow uses LLM operations, create prompt files:
prompts/
└── analyzeContent@v1.prompt
Related Skill: output-dev-prompt-file
Step 7: Create Test Scenarios
Add test input files to the scenarios folder:
scenarios/
├── basic_input.json
└── complex_input.json
Related Skill: output-dev-scenario-file
Step 8: Set Up Shared Resources (If Needed)
If your workflow needs shared clients, utilities, or services:
# Create shared directories if they don't exist
mkdir -p src/shared/clients
mkdir -p src/shared/utils
mkdir -p src/shared/services
Import shared resources in your steps:
import { GeminiService } from '../../shared/clients/gemini_client.js';
import { formatDate } from '../../shared/utils/date_helpers.js';
Related Skill: output-dev-http-client-create
Verification
After customization, verify your workflow:
1. List Available Workflows
npx output workflow list
Your workflow should appear in the list.
2. Run with Test Input
npx output workflow run {workflowName} --input path/to/scenarios/basic_input.json
3. Check for Errors
Common issues after skeleton generation:
- Import paths missing
.jsextension - Schema imported from
zodinstead of@outputai/core - Missing step exports
Customization Tips
Adding Multiple Steps
// steps.ts
export const stepOne = step( { ... } );
export const stepTwo = step( { ... } );
export const stepThree = step( { ... } );
// workflow.ts
const resultOne = await stepOne( input );
const resultTwo = await stepTwo( resultOne );
const resultThree = await stepThree( resultTwo );
Parallel Step Execution
// workflow.ts
const [ resultA, resultB ] = await Promise.all( [
stepA( input ),
stepB( input )
] );
Conditional Steps
// workflow.ts
if ( input.processImages ) {
await processImages( input );
}
Large Workflows - Folder-Based Organization
For workflows with many steps, use folder-based organization:
src/workflows/{workflow-name}/
├── workflow.ts
├── steps/ # Folder instead of single file
│ ├── fetch_data.ts
│ ├── process.ts
│ └── validate.ts
├── types.ts
└── ...
Verification Checklist
After generating and customizing the skeleton:
- Workflow folder follows
snake_casenaming -
workflow.tshas correct name in camelCase - All imports use
.jsextension -
zis imported from@outputai/core - Types are defined in
types.ts - Steps are defined in
steps.tsorsteps/folder - At least one test scenario exists
- Workflow appears in
npx output workflow list - Shared resources (if any) are in
src/shared/
Related Skills
output-dev-folder-structure- Understanding the complete folder layoutoutput-dev-workflow-function- Detailed workflow.ts documentationoutput-dev-step-function- Detailed steps.ts documentationoutput-dev-types-file- Creating Zod schemasoutput-dev-prompt-file- Adding LLM promptsoutput-dev-scenario-file- Creating test scenariosoutput-workflow-run- Running workflowsoutput-dev-code-style- Code style conventionsoutput-workflow-list- Listing available workflows
Encrypted Credentials Management
Overview
The @outputai/core/credentials package provides encrypted secrets management for Output SDK workflows. It replaces process.env patterns with a structured, encrypted YAML-based system that supports scoped credentials with deep merging.
When to Use This Skill
- Adding API keys or tokens to a workflow
- Migrating from
process.envto encrypted credentials - Setting up per-workflow or per-environment secrets
- Debugging missing credential errors (
MissingCredentialError,MissingKeyError) - Configuring custom credential providers (Vault, AWS Secrets Manager)
Library API
Import
import { credentials } from '@outputai/core/credentials';
credentials.get(path, defaultValue?)
Safe read with optional default. Never throws.
// Returns value or undefined
const region = credentials.get('aws.region');
// Returns value or default
const region = credentials.get('aws.region', 'us-east-1');
credentials.require(path)
Strict read. Throws MissingCredentialError if not found.
const apiKey = credentials.require('anthropic.api_key');
Error Types
| Error | Thrown When | Fix |
|---|---|---|
MissingCredentialError | credentials.require() path not found | Add the credential via output credentials edit |
MissingKeyError | No decryption key available | Set OUTPUT_CREDENTIALS_KEY env var or create .key file |
InvalidCredentialsKeyError | Key doesn't match the encrypted file, or the file is corrupted | Use the key the file was encrypted with |
MalformedCredentialsKeyError | Key isn't exactly 64 hex characters | Check the key for typos, whitespace, or truncation |
All four extend FatalError, so a step that throws one fails without retries. A key error while resolving credential: env vars stops the worker at startup.
CLI Commands
# Initialize credentials (generates key + encrypted YAML template)
output credentials init # Global
output credentials init -e production # Environment-specific
output credentials init -w payment_processing # Workflow-specific
# Edit credentials (decrypts, opens $EDITOR, re-encrypts on save)
output credentials edit # Global
output credentials edit -e production # Environment
output credentials edit -w payment_processing # Workflow
# Show decrypted credentials (debugging)
output credentials show # Global
output credentials show -e development # Environment
# Set a single credential value (no editor needed)
output credentials set anthropic.api_key sk-ant-... # Global
output credentials set stripe.key sk_live_... -w payment_processing # Workflow
# Get single credential value
output credentials get anthropic.api_key # Global
output credentials get stripe.key -w payment_processing # Workflow
Flags:
-e/--environment: Target environment (productionordevelopment, the only values the worker reads)-w/--workflow: Target a specific workflow-f/--force:initoverwrites existing credentials;editandsetre-encrypt with the current key when it can't decrypt the file (discards existing values)-y/--yes: Skip the confirmation whensetreplaces a value of a different shape (setonly)- Note:
-eand-ware mutually exclusive
Three-Tier Scope System
1. Global Credentials
config/credentials.yml.enc # Encrypted YAML
config/credentials.key # Decryption key (DO NOT COMMIT)
Key env var: OUTPUT_CREDENTIALS_KEY
2. Environment-Specific Credentials
config/credentials/production.yml.enc
config/credentials/production.key
Key env var: OUTPUT_CREDENTIALS_KEY_PRODUCTION
3. Per-Workflow Credentials
src/workflows/{name}/credentials.yml.enc
src/workflows/{name}/credentials.key
Key env var: OUTPUT_CREDENTIALS_KEY_{WORKFLOW_NAME} (uppercased)
Key Resolution Chain
For each scope, the key is resolved in order:
- Environment variable (
OUTPUT_CREDENTIALS_KEY,OUTPUT_CREDENTIALS_KEY_{ENV}, orOUTPUT_CREDENTIALS_KEY_{WORKFLOW}) - Key file on disk (e.g.,
config/credentials.key) - Throws
MissingKeyErrorif neither found
Workflow credentials fall back to the global key if no workflow-specific key exists.
Credential Merging
When a workflow has its own credentials, they deep-merge over global credentials. Workflow values win at the same path:
# Global (config/credentials.yml.enc)
anthropic:
api_key: sk-ant-global
aws:
region: us-east-1
# Workflow (src/workflows/my_workflow/credentials.yml.enc)
anthropic:
api_key: sk-ant-workflow-specific
stripe:
secret_key: sk_live_workflow
# Merged result at runtime:
# anthropic.api_key -> sk-ant-workflow-specific (overridden by workflow)
# aws.region -> us-east-1 (from global)
# stripe.secret_key -> sk_live_workflow (added by workflow)
Migration from process.env
Before (old pattern)
import { createKyClient } from '@outputai/core/http';
const API_KEY = process.env.SERVICE_API_KEY || '';
const client = createKyClient({
prefix: 'https://api.service.com',
headers: { Authorization: `Bearer ${API_KEY}` }
});
After (credentials pattern)
import { createKyClient } from '@outputai/core/http';
import { credentials } from '@outputai/core/credentials';
const apiKey = credentials.require('service.api_key');
const client = createKyClient({
prefix: 'https://api.service.com',
headers: { Authorization: `Bearer ${apiKey}` }
});
Migration Steps
- Run
output credentials initto create the encrypted file and key - Run
output credentials edit(oroutput credentials set <path> <value>) to add your secrets - Replace
process.env.Xreads withcredentials.require('x')orcredentials.get('x', default) - Remove environment variables from
.envfiles - Add
*.keyto.gitignore
Custom Providers
Replace the default encrypted YAML backend with Vault, AWS Secrets Manager, etc.:
import { setProvider } from '@outputai/core/credentials';
setProvider({
loadGlobal: ({ environment }) => {
return fetchFromVault(`credentials/${environment || 'default'}`);
},
loadForWorkflow: ({ workflowName, environment }) => {
return fetchFromVault(`workflows/${workflowName}`) ?? null;
}
});
Call setProvider() in a hook file listed under outputai.hookFiles, not in step code. Hook files load before the worker resolves credential: env vars, so those resolve through the custom provider too.
Provider Interface
interface CredentialsProvider {
loadGlobal(context: { environment: string | undefined }): Record<string, unknown>;
loadForWorkflow(context: {
workflowName: string;
workflowDir: string | undefined;
environment?: string | undefined;
}): Record<string, unknown> | null;
}
Security Considerations
- Never commit
.keyfiles - Add*.keyto.gitignore - Safe to commit
.yml.encfiles - Cannot be read without the key - Key file permissions - Created with mode
0o600(owner-only read/write) - Temp file cleanup - Plaintext overwritten with null bytes before deletion during
edit - Use env vars in CI/CD - Set
OUTPUT_CREDENTIALS_KEYin your pipeline - Encryption - AES-256-GCM with unique random nonce per encryption
Verification Checklist
-
credentialsimported from@outputai/core/credentials -
credentials.require()used for mandatory secrets (notprocess.env) -
credentials.get()used with default for optional values -
*.keylisted in.gitignore - Credentials initialized via
output credentials init - Secrets added via
output credentials edit
Related Skills
output-credentials-init- Initializing credentials files for the first timeoutput-credentials-edit- Viewing and editing credential valuesoutput-credentials-env-vars- Wiring credentials to env vars with thecredential:conventionoutput-dev-http-client-create- Creating HTTP clients that use credentialsoutput-dev-step-function- Using credentials in step functionsoutput-error-http-client- Troubleshooting HTTP client issues
Offline Evaluation Testing
Overview
The @outputai/core/evals module provides an offline evaluation framework for testing workflow quality using datasets and evaluators. This is complementary to the runtime evaluator() from @outputai/core:
| Aspect | Runtime Evaluators (@outputai/core) | Offline Eval Tests (@outputai/core/evals) |
|---|---|---|
| When | During workflow execution | After execution, at test time |
| Where | evaluators.ts in workflow folder | tests/evals/ in workflow folder |
| Purpose | Live quality scoring with confidence | Dataset-driven pass/fail verification |
| Triggered by | Workflow orchestration | output workflow test CLI command |
| Returns | EvaluationBooleanResult, etc. | Verdict helpers (pass/partial/fail) |
Use offline eval testing when you want to validate workflow behavior against known datasets, build regression test suites, or assess subjective quality with LLM judges.
When to Use This Skill
- Creating files in
tests/evals/ortests/datasets/ - Writing evaluators that use
verify()from@outputai/core/evals - Creating YAML dataset files for test cases
- Building eval workflows with
evalWorkflow() - Running
output workflow testcommands - Setting up ground truth data for evaluators
Directory Structure
Add a tests/ directory inside the workflow folder:
src/workflows/{workflow_name}/
├── workflow.ts
├── steps.ts
├── evaluators.ts # Runtime evaluators (optional)
├── types.ts
└── tests/
├── datasets/
│ ├── happy_path.yml
│ └── edge_case.yml
└── evals/
├── evaluators.ts # Offline eval test evaluators
├── workflow.ts # Eval workflow definition
└── judge_topic@v1.prompt # LLM judge prompts (optional)
Creating Evaluators with verify()
Import verify and Verdict from @outputai/core/evals (not the @outputai/core root):
// tests/evals/evaluators.ts
import { verify, Verdict } from '@outputai/core/evals';
import { z } from '@outputai/core';
verify() Signature
verify(options, checkFn)
Options:
name— unique evaluator identifier (snake_case)input— Zod schema for the workflow input (optional, defaults toz.any())output— Zod schema for the workflow output (optional, defaults toz.any())
Check function receives:
{
input, // typed workflow input
output, // typed workflow output
context: {
ground_truth: Record<string, unknown> // from dataset YAML
}
}
Returns: any Verdict helper result.
Basic Example
import { verify, Verdict } from '@outputai/core/evals';
import { z } from '@outputai/core';
export const evaluateSum = verify(
{
name: 'evaluate_sum',
input: z.object({ values: z.array(z.number()) }),
output: z.object({ result: z.number() })
},
({ input, output }) =>
Verdict.equals(output.result, input.values.reduce((a, b) => a + b, 0))
);
Using Ground Truth
Ground truth values come from the dataset YAML and are available via context.ground_truth:
export const lengthCheck = verify(
{ name: 'length_check', input: blogInput, output: blogOutput },
({ output, context }) =>
Verdict.gte(output.blog_post.length, Number(context.ground_truth.min_length ?? 100))
);
Verdict Helpers
All deterministic helpers return results with confidence 1.0.
Equality & Comparison
| Method | Description |
|---|---|
Verdict.equals(actual, expected) | Strict equality (===) |
Verdict.closeTo(actual, expected, tolerance) | Within numeric tolerance |
Verdict.gt(actual, threshold) | Greater than |
Verdict.gte(actual, threshold) | Greater than or equal |
Verdict.lt(actual, threshold) | Less than |
Verdict.lte(actual, threshold) | Less than or equal |
Verdict.inRange(actual, min, max) | Within inclusive range |
String & Array
| Method | Description |
|---|---|
Verdict.contains(haystack, needle) | String includes substring |
Verdict.matches(value, pattern) | Regex match |
Verdict.includesAll(actual, expected) | Array contains all expected values |
Verdict.includesAny(actual, expected) | Array contains at least one expected value |
Boolean
| Method | Description |
|---|---|
Verdict.isTrue(value) | Value is true |
Verdict.isFalse(value) | Value is false |
Manual Verdicts
| Method | Description |
|---|---|
Verdict.pass(reasoning?) | Explicit pass |
Verdict.partial(confidence, reasoning?, feedback?) | Partial pass with confidence |
Verdict.fail(reasoning, feedback?) | Explicit fail |
LLM Judge Evaluators
Before writing a judge prompt, identify the specific failure mode via error analysis (output-eval-error-analysis). Design the judge following output-eval-judge-prompt. After writing it, validate against human labels using output-eval-validate-judge.
For subjective quality assessments, use judge functions with .prompt files:
import { verify } from '@outputai/core/evals';
import { judgeVerdict, judgeScore, judgeLabel } from '@outputai/core/llm';
// Returns pass/partial/fail verdict from an LLM
export const evaluateTopic = verify(
{ name: 'evaluate_topic', input: blogInput, output: blogOutput },
async ({ input, output, context }) =>
judgeVerdict({
prompt: 'judge_topic@v1',
variables: {
blog_title: output.title,
blog_post: output.blog_post,
required_topic: String(context.ground_truth.required_topic ?? input.topic)
}
})
);
// Returns a numeric score from an LLM
export const evaluateQuality = verify(
{ name: 'evaluate_quality', input: blogInput, output: blogOutput },
async ({ input, output }) =>
judgeScore({
prompt: 'judge_quality@v1',
variables: { blog_title: output.title, blog_post: output.blog_post, topic: input.topic }
})
);
// Returns a string label from an LLM
export const evaluateTone = verify(
{ name: 'evaluate_tone', input: blogInput, output: blogOutput },
async ({ output }) =>
judgeLabel({
prompt: 'judge_tone@v1',
variables: { blog_title: output.title, blog_post: output.blog_post }
})
);
Judge .prompt File Format
Judge prompt files live alongside evaluators in tests/evals/:
# tests/evals/judge_topic@v1.prompt
---
provider: anthropic
# current as of 2026-05-04 — run output-dev-model-selection for the latest
model: claude-haiku-4-5-20251001
temperature: 0
maxOutputTokens: 1000
---
<system>
You are an evaluation judge. Assess whether a blog post is faithfully about the required topic.
Return a JSON object with:
- verdict: "pass" if the blog clearly focuses on the topic, "partial" if it mentions the topic but lacks depth, "fail" if it is not about the topic
- reasoning: a brief explanation of your judgment
</system>
<user>
Required topic: {{ required_topic }}
Blog title: {{ blog_title }}
Blog post:
{{ blog_post }}
Judge whether this blog post is faithfully about the required topic.
</user>
Creating Eval Workflows
The eval workflow wires evaluators together and defines how to interpret results.
// tests/evals/workflow.ts
import { evalWorkflow } from '@outputai/core/evals';
import { evaluateSum } from './evaluators.js';
export default evalWorkflow({
name: 'simple_eval',
evals: [
{
evaluator: evaluateSum,
criticality: 'required',
interpret: { type: 'boolean' }
}
]
});
Eval Definition Fields
Each entry in the evals array has:
evaluator— the function created byverify()criticality—'required'(affects pass/fail) or'informational'(reported but doesn't block)interpret— how to convert the evaluator's return value into a verdict
Interpret Types
| Type | Evaluator Returns | Mapping |
|---|---|---|
{ type: 'boolean' } | Verdict.equals(), Verdict.gte(), etc. | true = pass, false = fail |
{ type: 'verdict' } | judgeVerdict() or Verdict.pass/partial/fail() | Direct pass-through |
{ type: 'number', pass: 0.7, partial: 0.4 } | judgeScore() | >=pass = pass, >=partial = partial, else fail |
{ type: 'string', pass: ['a', 'b'], partial: ['c'] } | judgeLabel() | Label in pass list = pass, in partial list = partial, else fail |
Full Example with Mixed Evaluators
export default evalWorkflow({
name: 'blog_generator_eval',
evals: [
{
evaluator: lengthOfOutput,
criticality: 'required',
interpret: { type: 'boolean' }
},
{
evaluator: evaluateTopic,
criticality: 'required',
interpret: { type: 'verdict' }
},
{
evaluator: evaluateQuality,
criticality: 'required',
interpret: { type: 'number', pass: 0.7, partial: 0.4 }
},
{
evaluator: evaluateContent,
criticality: 'informational',
interpret: { type: 'boolean' }
},
{
evaluator: evaluateTone,
criticality: 'informational',
interpret: { type: 'string', pass: ['professional', 'informative'], partial: ['casual'] }
}
]
});
Naming Convention
The eval workflow name must end in _eval and match the pattern {workflow_name}_eval. The CLI resolves this automatically — output workflow test blog_generator looks for blog_generator_eval.
Dataset Files
For methodology on designing diverse datasets that cover failure-prone regions, see output-eval-dataset-design.
Datasets are YAML files in tests/datasets/. Each file represents one test case.
Basic Format
name: basic_input
input:
values:
- 1
- 2
- 3
- 4
- 5
last_output:
output:
result: 15
date: '2026-02-13T00:00:00.000Z'
With Ground Truth
Ground truth provides expected values for evaluators. You can set global values and per-evaluator overrides:
name: stripe_blog
input:
topic: "Stripe the payment processor"
requirements: "Include a link to https://stripe.com/en-gb/pricing"
last_output:
output:
title: "Stripe: The Modern Payment Processing Platform"
blog_post: |
Stripe has revolutionized online payment processing...
date: '2026-02-16T00:00:00.000Z'
ground_truth:
notes: "Known good case"
evals:
length_of_output:
min_length: 100
evaluate_topic:
required_topic: "Stripe the payment processor"
evaluate_content:
required_content: "https://stripe.com/en-gb/pricing"
The ground_truth.evals.<evaluator_name> values are merged with the top-level ground truth and passed to the evaluator via context.ground_truth.
CLI Commands
output workflow test <workflow_name>
Runs evaluations against all datasets for a workflow.
| Flag | Description |
|---|---|
--cached | Use cached output from dataset files (skip workflow execution) |
--save | Run workflow fresh and save output + eval results back to dataset files |
--dataset <names> | Comma-separated list of dataset names to run (default: all) |
--json | Output machine-readable JSON instead of the rendered report |
Execution flow:
- Loads all dataset YAML files from
tests/datasets/ - Without
--cached: executes the workflow for each dataset to get fresh output - Sends all datasets to the
{workflow_name}_evalworkflow - Reports per-dataset and per-evaluator verdicts
- Exits with code 1 if any required evaluator fails
output workflow dataset list <workflow_name>
Lists all datasets for a workflow with their cached status.
| Flag | Description |
|---|---|
--format <type> | Output format: table (default) or text |
--json | Output machine-readable JSON |
output workflow dataset generate <workflow_name> [scenario]
Generates a new dataset file by running the workflow, or from a completed run read from Temporal.
| Flag | Description |
|---|---|
--input <json> | Workflow input as a JSON string or file path |
--name <name> | Dataset filename (defaults to scenario name, or the workflow id with --workflow-id) |
--workflow-id <id> | Read a completed run of the workflow from Temporal instead of running it. Cannot be combined with a scenario or --input |
--run-id <runId> | With --workflow-id: the run to read (defaults to the latest run) |
--profile, --address, --namespace | With --workflow-id: which Temporal to read from |
Common Usage
# Generate dataset from inline JSON input
output workflow dataset generate my_workflow --input '{"key": "value"}' --name my_test
# Generate from a scenario file
output workflow dataset generate my_workflow basic
# Generate from a completed run (input and result read from Temporal)
output workflow dataset generate my_workflow --workflow-id <workflowId> --name from_run
# Run evals with cached output (fast, no re-execution)
output workflow test my_workflow --cached
# Run evals fresh and save results
output workflow test my_workflow --save
# Run specific datasets only
output workflow test my_workflow --dataset happy_path,edge_case
# List all datasets
output workflow dataset list my_workflow
Typical Workflow
# 1. Start the dev server
npm run output:dev
# 2. Generate datasets from real workflow runs
output workflow dataset generate blog_generator --input '{"topic": "AI"}' --name ai_post
# 3. Edit the dataset YAML to add ground_truth values for your evaluators
# 4. Run evals with --save to cache output and eval results
output workflow test blog_generator --save
# 5. Iterate on evaluators, re-run with cached output (fast)
output workflow test blog_generator --cached
# 6. List all datasets
output workflow dataset list blog_generator
Verification Checklist
- Evaluators import
verify,Verdictfrom@outputai/core/evals(not the@outputai/coreroot), and judges from@outputai/core/llm - Eval workflow imports
evalWorkflowfrom@outputai/core/evals - All imports use
.jsextension - Eval workflow name follows
{workflow_name}_evalpattern - Dataset YAML files are in
tests/datasets/ - Evaluator files are in
tests/evals/ - Each evaluator has a unique
namein snake_case -
criticalityis set to'required'or'informational'for each eval -
interprettype matches evaluator return type - Ground truth keys in dataset match evaluator names
- Judge
.promptfiles are intests/evals/alongside evaluators -
zis imported from@outputai/core(notzod)
Related Skills
output-dev-evaluator-function— Runtime evaluators usingevaluator()from@outputai/coreoutput-dev-scenario-file— Creating scenario JSON files for workflow executionoutput-dev-folder-structure— Understanding project directory layoutoutput-dev-prompt-file— Creating.promptfiles for LLM operationsoutput-eval-error-analysis— Identify failure modes before building evaluatorsoutput-eval-judge-prompt— Design effective LLM judge promptsoutput-eval-dataset-design— Generate diverse test datasetsoutput-eval-validate-judge— Validate LLM judges against human labelsoutput-eval-audit— Audit an existing eval suite for trustworthiness
Creating Evaluator Functions
Overview
This skill documents how to create evaluator functions in evaluators.ts for Output SDK workflows. Evaluators are used to assess quality, validate outputs, and provide confidence-scored judgments about workflow results.
When to Use This Skill
- Implementing quality assessment for workflow outputs
- Adding validation logic with confidence scores
- Creating LLM-powered content evaluation
- Building reusable evaluation components
File Organization
Option 1: Flat File (Default)
For smaller workflows, use a single evaluators.ts file:
src/workflows/{workflow-name}/
├── workflow.ts
├── steps.ts
├── evaluators.ts # All evaluators in one file
├── types.ts
└── ...
Option 2: Folder-Based (Large workflows)
For larger workflows with many evaluators, use an evaluators/ folder:
src/workflows/{workflow-name}/
├── workflow.ts
├── steps.ts
├── evaluators/ # Evaluators split into individual files
│ ├── quality.ts
│ ├── accuracy.ts
│ └── completeness.ts
├── types.ts
└── ...
Component Location Rules
Important: evaluator() calls MUST be in files containing 'evaluators' in the path:
src/workflows/my_workflow/evaluators.ts✓src/workflows/my_workflow/evaluators/quality.ts✓src/shared/evaluators/common_evaluators.ts✓src/workflows/my_workflow/helpers.ts✗ (cannot contain evaluator() calls)
Activity Isolation Constraints
Evaluators are Temporal activities with strict import rules to ensure deterministic replay.
Evaluators CAN import from:
- Local workflow files:
./utils.js,./types.js,./helpers.js - Local subdirectories:
./lib/helpers.js - Shared utilities:
../../shared/utils/*.js - Shared clients:
../../shared/clients/*.js - Shared services:
../../shared/services/*.js
Evaluators CANNOT import:
- Other evaluator files (activity isolation)
- Step files
- Workflow files
Example of WRONG imports:
// WRONG - evaluators cannot import other evaluators
import { otherEvaluator } from '../../shared/evaluators/other.js'; // ✗
import { anotherEvaluator } from './other_evaluators.js'; // ✗
Critical Import Patterns
Core Imports
// CORRECT - Import from @outputai/core
import {
evaluator,
z,
EvaluationBooleanResult,
EvaluationNumberResult,
EvaluationStringResult,
EvaluationFeedback
} from '@outputai/core';
// WRONG - Never import z from zod
import { z } from 'zod';
LLM Client Import (for LLM-powered evaluators)
// CORRECT - Use @outputai/core/llm wrapper
import { generateText, aiSdk } from '@outputai/core/llm';
// WRONG - Never call LLM providers directly
import OpenAI from 'openai';
ES Module Imports
All imports MUST use .js extension:
// CORRECT
import { BlogContent } from './types.js';
// WRONG - Missing .js extension
import { BlogContent } from './types';
Basic Structure
import { evaluator, z, EvaluationBooleanResult } from '@outputai/core';
export const myEvaluator = evaluator( {
name: 'my_evaluator',
description: 'Description of what this evaluator assesses',
inputSchema: z.object( { /* input schema */ } ),
fn: async input => {
// Evaluation logic
return new EvaluationBooleanResult( {
value: true,
confidence: 0.95
} );
}
} );
Required Properties
name (string)
Unique identifier for the evaluator. Use snake_case.
name: 'evaluate_content_quality'
description (string)
Human-readable description of what the evaluator assesses.
description: 'Evaluate the quality and completeness of generated content'
inputSchema (Zod schema)
Schema for validating evaluator input.
inputSchema: z.object( {
content: z.string(),
expectedLength: z.number()
} )
fn (async function)
The evaluator execution function. Returns an evaluation result with value and confidence.
fn: async input => {
const isValid = input.content.length >= input.expectedLength;
return new EvaluationBooleanResult( {
value: isValid,
confidence: 0.95
} );
}
Result Types
EvaluationBooleanResult
Use for pass/fail or true/false evaluations:
import { EvaluationBooleanResult } from '@outputai/core';
return new EvaluationBooleanResult( {
value: true, // boolean result
confidence: 0.95, // 0.0 to 1.0
reasoning: 'Optional explanation of the evaluation'
} );
EvaluationNumberResult
Use for numeric scores or ratings:
import { EvaluationNumberResult } from '@outputai/core';
return new EvaluationNumberResult( {
value: 85, // numeric result (e.g., 0-100 score)
confidence: 0.85, // 0.0 to 1.0
reasoning: 'Optional explanation of the score'
} );
EvaluationStringResult
Use for categorical or text-based evaluations:
import { EvaluationStringResult } from '@outputai/core';
return new EvaluationStringResult( {
value: 'positive', // string result (e.g., category, sentiment, label)
confidence: 0.9, // 0.0 to 1.0
reasoning: 'Optional explanation of the classification'
} );
Result Properties
| Property | Type | Required | Description |
|---|---|---|---|
value | boolean, number, or string | Yes | The evaluation result |
confidence | number (0.0-1.0) | Yes | Confidence in the evaluation |
reasoning | string | No | Explanation of the evaluation |
name | string | No | Name for this specific result (useful in dimensions) |
feedback | EvaluationFeedback[] | No | Array of feedback objects with issues and suggestions |
dimensions | EvaluationResult[] | No | Nested results for multi-dimensional evaluation |
Simple Evaluator Examples
Boolean Evaluator - Content Validation
import { evaluator, z, EvaluationBooleanResult } from '@outputai/core';
export const evaluateCompleteness = evaluator( {
name: 'evaluate_completeness',
description: 'Check if content meets minimum length requirements',
inputSchema: z.object( {
content: z.string(),
minLength: z.number().default( 100 )
} ),
fn: async ( { content, minLength } ) => {
const isComplete = content.length >= minLength;
return new EvaluationBooleanResult( {
value: isComplete,
confidence: 1.0,
reasoning: isComplete ?
`Content has ${content.length} characters, meets minimum of ${minLength}` :
`Content has ${content.length} characters, below minimum of ${minLength}`
} );
}
} );
Boolean Evaluator - Pattern Detection
import { evaluator, z, EvaluationBooleanResult } from '@outputai/core';
export const evaluateGibberish = evaluator( {
name: 'evaluate_gibberish',
description: 'Check if a given string is gibberish',
inputSchema: z.string(),
fn: async content => {
const gibberishPatterns = [ 'foo', 'bar', 'lorem', 'ipsum' ];
const isGibberish = gibberishPatterns.some( p => content.toLowerCase().includes( p ) );
return new EvaluationBooleanResult( {
value: !isGibberish,
confidence: 0.95
} );
}
} );
Number Evaluator - Quality Score
import { evaluator, z, EvaluationNumberResult } from '@outputai/core';
export const evaluateReadability = evaluator( {
name: 'evaluate_readability',
description: 'Calculate readability score based on sentence structure',
inputSchema: z.object( {
content: z.string()
} ),
fn: async ( { content } ) => {
const sentences = content.split( /[.!?]+/ ).filter( s => s.trim() );
const words = content.split( /\s+/ ).filter( w => w.trim() );
const avgWordsPerSentence = words.length / Math.max( sentences.length, 1 );
// Simple readability score (lower avg words = more readable)
const score = Math.max( 0, Math.min( 100, 100 - ( avgWordsPerSentence - 15 ) * 5 ) );
return new EvaluationNumberResult( {
value: Math.round( score ),
confidence: 0.8,
reasoning: `Average ${avgWordsPerSentence.toFixed( 1 )} words per sentence`
} );
}
} );
String Evaluator - Sentiment Classification
import { evaluator, z, EvaluationStringResult } from '@outputai/core';
export const evaluateSentiment = evaluator( {
name: 'evaluate_sentiment',
description: 'Classify the sentiment of content',
inputSchema: z.object( {
content: z.string()
} ),
fn: async ( { content } ) => {
const positiveWords = [ 'great', 'excellent', 'amazing', 'good', 'love' ];
const negativeWords = [ 'bad', 'terrible', 'awful', 'hate', 'poor' ];
const lowerContent = content.toLowerCase();
const positiveCount = positiveWords.filter( w => lowerContent.includes( w ) ).length;
const negativeCount = negativeWords.filter( w => lowerContent.includes( w ) ).length;
const { sentiment, confidence } = positiveCount > negativeCount ?
{ sentiment: 'positive', confidence: Math.min( 0.95, 0.6 + positiveCount * 0.1 ) } :
negativeCount > positiveCount ?
{ sentiment: 'negative', confidence: Math.min( 0.95, 0.6 + negativeCount * 0.1 ) } :
{ sentiment: 'neutral', confidence: 0.7 };
return new EvaluationStringResult( {
value: sentiment,
confidence,
reasoning: `Found ${positiveCount} positive and ${negativeCount} negative indicators`
} );
}
} );
LLM-Powered Evaluator Examples
Note: Evaluators are self-contained components that don't share schemas across steps, so defining aiSdk.Output.object() schemas inline is acceptable here. For workflow steps that share schemas, define them in types.ts instead.
generateText arguments: prompt, promptDir, variables, tools, output, toolChoice, stopWhen, abortSignal.
Using generateText with aiSdk.Output.object() for Evaluation
import { evaluator, z, EvaluationNumberResult } from '@outputai/core';
import { generateText, aiSdk } from '@outputai/core/llm';
export const evaluateSignalToNoise = evaluator( {
name: 'evaluate_signal_to_noise',
description: 'Evaluate the signal-to-noise ratio of content',
inputSchema: z.object( {
title: z.string(),
content: z.string()
} ),
fn: async ( { title, content } ) => {
const { output } = await generateText( {
prompt: 'signal_noise@v1', // References prompts/signal_noise@v1.prompt
variables: {
title,
content
},
output: aiSdk.Output.object( {
schema: z.object( {
score: z.number().describe( 'Signal-to-noise score 0-100' )
} )
} )
} );
return new EvaluationNumberResult( {
value: output.score,
confidence: 0.85
} );
}
} );
LLM Boolean Evaluation
import { evaluator, z, EvaluationBooleanResult } from '@outputai/core';
import { generateText, aiSdk } from '@outputai/core/llm';
export const evaluateFactualAccuracy = evaluator( {
name: 'evaluate_factual_accuracy',
description: 'Check if content contains factual claims that can be verified',
inputSchema: z.object( {
content: z.string(),
topic: z.string()
} ),
fn: async ( { content, topic } ) => {
const { output } = await generateText( {
prompt: 'factual_check@v1',
variables: { content, topic },
output: aiSdk.Output.object( {
schema: z.object( {
isFactual: z.boolean().describe( 'Whether content appears factually accurate' ),
confidence: z.number().describe( 'Confidence in assessment 0-1' ),
issues: z.array( z.string() ).optional().describe( 'Any factual issues found' )
} )
} )
} );
return new EvaluationBooleanResult( {
value: output.isFactual,
confidence: output.confidence,
reasoning: output.issues?.length ?
`Issues found: ${output.issues.join( ', ' )}` :
'No factual issues detected'
} );
}
} );
LLM String Evaluation - Content Classification
import { evaluator, z, EvaluationStringResult } from '@outputai/core';
import { generateText, aiSdk } from '@outputai/core/llm';
export const evaluateContentCategory = evaluator( {
name: 'evaluate_content_category',
description: 'Classify content into a category',
inputSchema: z.object( {
content: z.string(),
categories: z.array( z.string() )
} ),
fn: async ( { content, categories } ) => {
const { output } = await generateText( {
prompt: 'categorize_content@v1',
variables: {
content,
categories: categories.join( ', ' )
},
output: aiSdk.Output.object( {
schema: z.object( {
category: z.string().describe( 'The best matching category' ),
confidence: z.number().describe( 'Confidence in classification 0-1' ),
explanation: z.string().describe( 'Why this category was chosen' )
} )
} )
} );
return new EvaluationStringResult( {
value: output.category,
confidence: output.confidence,
reasoning: output.explanation
} );
}
} );
EvaluationResult with Feedback
Use the feedback field to provide actionable improvement suggestions alongside your evaluation result. Import EvaluationFeedback from @outputai/core to create feedback objects.
import { evaluator, z, EvaluationStringResult, EvaluationFeedback } from '@outputai/core';
export const evaluateWithFeedback = evaluator( {
name: 'evaluate_with_feedback',
description: 'Evaluate content quality and provide actionable feedback',
inputSchema: z.string(),
fn: async response => {
const feedback = [];
if ( response.length < 50 ) {
feedback.push( new EvaluationFeedback( {
issue: 'Response is too short',
suggestion: 'Expand the response with more detail',
priority: 'medium'
} ) );
}
return new EvaluationStringResult( {
value: feedback.length === 0 ? 'good' : 'needs_improvement',
confidence: 0.85,
feedback: feedback
} );
}
} );
EvaluationFeedback Properties
| Property | Type | Description |
|---|---|---|
issue | string | The problem identified |
suggestion | string | Recommended fix |
priority | string | Priority level (e.g., 'low', 'medium', 'high') |
Multi-Dimensional Evaluation
Use the dimensions field to nest EvaluationResult instances for sub-scores. Each dimension should use the name field to identify it.
import { evaluator, z, EvaluationStringResult, EvaluationNumberResult } from '@outputai/core';
export const evaluateMultiDimensional = evaluator( {
name: 'evaluate_multi_dimensional',
description: 'Evaluate content across multiple quality dimensions',
inputSchema: z.string(),
fn: async response => {
const coherenceScore = calculateCoherence( response );
const relevanceScore = calculateRelevance( response );
const overallScore = ( coherenceScore + relevanceScore ) / 2;
return new EvaluationStringResult( {
value: overallScore > 0.7 ? 'high_quality' : 'low_quality',
confidence: 0.9,
dimensions: [
new EvaluationNumberResult( {
value: coherenceScore,
confidence: 0.85,
name: 'coherence'
} ),
new EvaluationNumberResult( {
value: relevanceScore,
confidence: 0.88,
name: 'relevance'
} )
]
} );
}
} );
Complete Example
Based on a real workflow evaluator file:
import { evaluator, z, EvaluationBooleanResult, EvaluationNumberResult } from '@outputai/core';
import { generateText, aiSdk } from '@outputai/core/llm';
import { blogContentSchema } from './types.js';
import type { BlogContent, QualityMetrics } from './types.js';
// Simple boolean evaluator
export const evaluateMinimumLength = evaluator( {
name: 'evaluate_minimum_length',
description: 'Check if blog content meets minimum length requirements',
inputSchema: blogContentSchema,
fn: async ( input: BlogContent ) => {
const MIN_TOKENS = 500;
const meetsRequirement = input.tokenCount >= MIN_TOKENS;
return new EvaluationBooleanResult( {
value: meetsRequirement,
confidence: 1.0,
reasoning: `Content has ${input.tokenCount} tokens (minimum: ${MIN_TOKENS})`
} );
}
} );
// LLM-powered number evaluator
export const evaluateSignalToNoise = evaluator( {
name: 'evaluate_signal_to_noise',
description: 'Evaluate the signal-to-noise ratio of blog content',
inputSchema: blogContentSchema,
fn: async ( input: BlogContent ) => {
const { output } = await generateText( {
prompt: 'signal_noise@v1',
variables: {
title: input.title,
content: input.content
},
output: aiSdk.Output.object( {
schema: z.object( {
score: z.number().describe( 'Signal-to-noise score 0-100' )
} )
} )
} );
return new EvaluationNumberResult( {
value: output.score,
confidence: 0.85
} );
}
} );
// LLM-powered boolean evaluator
export const evaluateRelevance = evaluator( {
name: 'evaluate_relevance',
description: 'Check if content is relevant to the stated topic',
inputSchema: z.object( {
content: z.string(),
topic: z.string(),
keywords: z.array( z.string() )
} ),
fn: async ( { content, topic, keywords } ) => {
const { output } = await generateText( {
prompt: 'relevance_check@v1',
variables: { content, topic, keywords: keywords.join( ', ' ) },
output: aiSdk.Output.object( {
schema: z.object( {
isRelevant: z.boolean(),
relevanceScore: z.number().describe( 'Relevance score 0-1' ),
explanation: z.string()
} )
} )
} );
return new EvaluationBooleanResult( {
value: output.isRelevant,
confidence: output.relevanceScore,
reasoning: output.explanation
} );
}
} );
Best Practices
1. Use Appropriate Result Types
// Boolean for pass/fail decisions
return new EvaluationBooleanResult( { value: true, confidence: 0.9 } );
// Number for scores and ratings
return new EvaluationNumberResult( { value: 85, confidence: 0.85 } );
// String for categories, labels, or classifications
return new EvaluationStringResult( { value: 'positive', confidence: 0.9 } );
2. Provide Meaningful Confidence Scores
// High confidence for deterministic checks
confidence: 1.0 // e.g., length checks, pattern matching
// Medium confidence for heuristic-based evaluations
confidence: 0.85 // e.g., LLM-based assessments
// Lower confidence for uncertain evaluations
confidence: 0.7 // e.g., subjective quality judgments
3. Include Reasoning for Transparency
return new EvaluationBooleanResult( {
value: false,
confidence: 0.95,
reasoning: `Content contains ${errorCount} grammatical errors, exceeding threshold of ${maxErrors}`
} );
4. Keep Evaluators Focused
// Good - single responsibility
export const evaluateGrammar = evaluator( { ... } );
export const evaluateReadability = evaluator( { ... } );
export const evaluateTone = evaluator( { ... } );
// Avoid - doing too much in one evaluator
export const evaluateEverything = evaluator( { ... } );
5. Use Descriptive Names
// Good - clear what is being evaluated
name: 'evaluate_content_originality'
name: 'evaluate_factual_accuracy'
name: 'evaluate_sentiment_alignment'
// Avoid - vague names
name: 'check'
name: 'validate'
name: 'evaluate_stuff'
6. Use Feedback for Actionable Improvements
feedback: [
new EvaluationFeedback( {
issue: 'Missing conclusion paragraph',
suggestion: 'Add a summary paragraph at the end',
priority: 'high'
} )
]
7. Use Dimensions for Multi-Criteria Evaluation
dimensions: [
new EvaluationNumberResult( { value: 8, confidence: 0.9, name: 'coherence' } ),
new EvaluationNumberResult( { value: 6, confidence: 0.85, name: 'relevance' } )
]
Verification Checklist
-
evaluator,z, result types imported from@outputai/core -
generateTextandaiSdkimported from@outputai/core/llmif using LLM (not direct provider) - LLM output schemas use
.describe()instead of.min()/.max()onz.number() - All imports use
.jsextension - Named exports used for each evaluator
- Each evaluator has
name,description,inputSchema,fn - Evaluator name uses
snake_case - Returns appropriate result type (
EvaluationBooleanResult,EvaluationNumberResult, orEvaluationStringResult) - Confidence score between 0.0 and 1.0
- Evaluators only import allowed dependencies (local files, shared code)
- No imports of other evaluators, steps, or workflows
-
EvaluationFeedbackimported from@outputai/corewhen using feedback - Feedback objects include
issue,suggestion, andpriority - Dimensions use the
namefield to identify sub-evaluations
Related Skills
output-dev-workflow-function- Orchestrating evaluators in workflow.tsoutput-dev-step-function- Creating step functionsoutput-dev-types-file- Defining evaluator input schemasoutput-dev-prompt-file- Creating prompt files for LLM-powered evaluatorsoutput-dev-folder-structure- Understanding project layoutoutput-eval-error-analysis— Identify what to evaluate before writing evaluators
Workflow Folder Structure Conventions
Overview
This skill documents the standard folder structure for Output SDK workflows. Following these conventions ensures consistency across the codebase and enables proper tooling support.
When to Use This Skill
- Creating a new workflow from scratch
- Reorganizing an existing workflow
- Understanding where to place different file types
- Reviewing workflow structure for compliance
Standard Project Structure
src/
├── shared/ # Shared code across workflows
│ ├── clients/ # API clients (using @outputai/core/http)
│ ├── utils/ # Utility functions & helpers
│ ├── services/ # Business logic services
│ ├── steps/ # Shared steps (optional)
│ └── evaluators/ # Shared evaluators (optional)
└── workflows/
└── {workflow-name}/ # Individual workflow directory
├── workflow.ts # Workflow definition (REQUIRED)
├── steps.ts # OR steps/ folder
├── evaluators.ts # OR evaluators/ folder (optional)
├── types.ts # Zod schemas and TypeScript types
├── utils.ts # Workflow-specific utilities (optional)
├── prompts/ # LLM prompt templates (optional)
│ └── {promptName}@v1.prompt
└── scenarios/ # Test input scenarios (optional)
└── {scenario_name}.json
File Purposes
workflow.ts (Required)
- Contains the main
workflow()function definition - Default exports the workflow
- Must be deterministic - no direct I/O operations
- Orchestrates step calls
Related Skill: output-dev-workflow-function
steps.ts or steps/ folder (Required)
- Contains all
step()function definitions - Handles all I/O operations (HTTP, LLM, file system, etc.)
- Named exports for each step function
- Includes error handling with FatalError and ValidationError
Related Skill: output-dev-step-function
evaluators.ts or evaluators/ folder (Optional)
- Contains
evaluator()function definitions - Used for workflow quality assessment and validation
- Named exports for each evaluator function
types.ts (Required)
- Contains Zod schemas for input/output validation
- Exports TypeScript types derived from schemas
- Imports
zfrom@outputai/core(never fromzod)
Related Skill: output-dev-types-file
utils.ts (Optional)
- Contains pure helper functions
- No I/O operations - those belong in steps
- Shared utility logic for the workflow
prompts/ folder (Optional)
- Contains
.promptfiles for LLM operations - File naming:
{promptName}@v1.prompt - Uses YAML frontmatter and Liquid.js templating
Related Skill: output-dev-prompt-file
scenarios/ folder (Optional)
- Contains JSON test input files
- File naming:
{scenario_name}.json - Matches workflow inputSchema structure
Related Skill: output-dev-scenario-file
Organization Options
Option 1: Flat Files (Recommended for smaller workflows)
src/workflows/{workflow-name}/
├── workflow.ts
├── steps.ts # All steps in one file
├── evaluators.ts # All evaluators in one file (optional)
├── types.ts
└── ...
Option 2: Folder-Based (For larger workflows)
src/workflows/{workflow-name}/
├── workflow.ts
├── steps/ # Steps split into individual files
│ ├── fetch_data.ts
│ ├── process.ts
│ └── validate.ts
├── evaluators/ # Evaluators split into individual files
│ ├── quality.ts
│ └── accuracy.ts
├── types.ts
└── ...
Component Location Rules (Strict)
The Output SDK enforces strict rules about where components can be defined:
| Component | Must be in |
|---|---|
step() calls | Files containing 'steps' in path |
evaluator() calls | Files containing 'evaluators' in path |
workflow() calls | workflow.ts file |
Examples:
src/workflows/my_workflow/steps.ts✓src/workflows/my_workflow/steps/fetch_data.ts✓src/shared/steps/common_steps.ts✓src/workflows/my_workflow/helpers.ts✗ (cannot contain step() calls)
Import Rules (Activity Isolation)
Steps and evaluators are Temporal activities with isolation constraints to ensure deterministic replay.
Steps CAN import from:
- Local workflow files:
./utils.js,./types.js,./helpers.js - Local subdirectories:
./clients/pokeapi.js,./lib/helpers.js - Shared utilities:
../../shared/utils/*.js - Shared clients:
../../shared/clients/*.js - Shared services:
../../shared/services/*.js
Steps CANNOT import:
- Other steps (activity isolation)
- Evaluators
- Workflow files
Evaluators follow the same rules:
- CAN import local files and shared code
- CANNOT import other evaluators, steps, or workflows
Import Pattern Examples:
// From workflow steps.ts - importing shared client
import { GeminiImageService } from '../../shared/clients/gemini_client.js';
// From workflow steps.ts - importing local utility
import { formatResponse } from './utils.js';
// From workflow steps.ts - importing types
import { InputSchema, OutputSchema } from './types.js';
// WRONG - steps cannot import other steps
import { otherStep } from '../../shared/steps/other.js'; // ✗
Shared Resources
src/shared/clients/
HTTP clients shared across workflows:
src/shared/clients/
├── gemini_client.ts # Google Gemini API client
├── jina_client.ts # Jina AI client
└── perplexity_client.ts # Perplexity API client
Import pattern in workflow steps:
import { GeminiImageService } from '../../shared/clients/gemini_client.js';
Related Skill: output-dev-http-client-create
src/shared/utils/
Utility functions shared across workflows:
src/shared/utils/
├── string_helpers.ts
├── date_formatters.ts
└── validators.ts
src/shared/services/
Business logic services shared across workflows:
src/shared/services/
├── image_service.ts
└── content_service.ts
src/shared/steps/ (Optional)
Shared steps that can be imported by workflows:
src/shared/steps/
└── common_steps.ts
Note: Workflows import shared steps, but steps cannot import other steps directly.
Naming Conventions
Folder Names
- Use
snake_casefor workflow folder names - Example:
image_infographic_nano,resume_parser
File Names
- Use
camelCasefor.tsfiles (exceptworkflow.ts,steps.ts,types.ts,evaluators.ts) - Use
camelCase@v{n}for.promptfiles - Use
snake_casefor.jsonscenario files
Workflow Names
- The
nameproperty inworkflow()should be camelCase - Example:
imageInfographicNano
Example: Complete Workflow Structure
src/workflows/image_infographic_nano/
├── workflow.ts # workflow({ name: 'imageInfographicNano', ... })
├── steps.ts # generateImageIdeas, generateImages, validateReferenceImages
├── types.ts # WorkflowInputSchema, WorkflowOutput, step schemas
├── utils.ts # normalizeReferenceImageUrls, buildS3Url, etc.
├── prompts/
│ └── generateImageIdeas@v1.prompt
└── scenarios/
├── test_input_complex.json
└── test_input_solar_panels.json
Verification Checklist
When reviewing workflow structure, verify:
-
workflow.tsexists with default export -
steps.tsorsteps/folder exists with all step definitions -
types.tsexists with Zod schemas - All
.tsimports use.jsextension -
prompts/folder exists if LLM operations are used -
scenarios/folder exists with at least one test input - Folder naming follows
snake_caseconvention - Workflow name in code follows
camelCaseconvention - Steps only import allowed dependencies (local files, shared code)
- No cross-component imports (steps don't import other steps)
Related Skills
output-dev-workflow-function- Writing workflow.ts filesoutput-dev-step-function- Writing step functionsoutput-dev-evaluator-function- Writing evaluators.ts filesoutput-dev-types-file- Creating Zod schemasoutput-dev-prompt-file- Creating prompt filesoutput-dev-scenario-file- Creating test scenariosoutput-dev-http-client-create- Creating shared HTTP clients
Creating HTTP Clients
Overview
This skill documents how to create shared HTTP clients for Output SDK workflows. Clients are stored in src/shared/clients/ and shared across all workflows to ensure consistent error handling, retry logic, and API integration patterns.
When to Use This Skill
- Integrating a new external API service
- Creating a reusable HTTP wrapper for a service
- Standardizing error handling for API calls
- Moving inline HTTP logic to a shared client
Location Convention
HTTP clients are stored in the shared clients folder:
src/shared/clients/
├── gemini_client.ts # Google Gemini API client
├── jina_client.ts # Jina AI client
├── perplexity_client.ts # Perplexity API client
└── {service}_client.ts # Your new client
Important: Clients are shared across ALL workflows. Do NOT create per-workflow HTTP clients.
Other Shared Code Locations
src/shared/
├── clients/ # API clients (this skill)
├── utils/ # Utility functions & helpers
├── services/ # Business logic services
├── steps/ # Shared step definitions (optional)
└── evaluators/ # Shared evaluators (optional)
Import Pattern in Workflows
Use relative imports from workflow files to shared clients:
// CORRECT - Relative path from workflow steps.ts
import { GeminiImageService } from '../../shared/clients/gemini_client.js';
import { parseResumeWithJina } from '../../shared/clients/jina_client.js';
// From shared steps (if used)
import { JinaClient } from '../clients/jina_client.js';
Critical Import Rules
HTTP Client Import
// CORRECT - Use @outputai/core/http wrapper
import { createKyClient } from '@outputai/core/http';
// WRONG - Never use axios directly
import axios from 'axios';
Error Types Import
// CORRECT - Import error types from @outputai/core
import { FatalError, ValidationError } from '@outputai/core';
// WRONG - Custom error classes
class MyCustomError extends Error { ... }
Credentials Import
// CORRECT - Use @outputai/core/credentials for secrets
import { credentials } from '@outputai/core/credentials';
const apiKey = credentials.require('service.api_key');
// WRONG - Never use process.env for secrets
const apiKey = process.env.SERVICE_API_KEY;
Cost Tracking Import
// CORRECT - Attach spend to paid API calls with @outputai/core/http
import { addRequestCost, createKyClient } from '@outputai/core/http';
// WRONG - Tracking cost yourself in a side table, or skipping it for paid APIs
Basic Client Structure
Simple Function-Based Client
import { FatalError, ValidationError } from '@outputai/core';
import { createKyClient } from '@outputai/core/http';
import { credentials } from '@outputai/core/credentials';
const API_KEY = credentials.require('service.api_key');
const BASE_URL = 'https://api.service.com';
const client = createKyClient({
prefix: BASE_URL,
headers: {
Authorization: `Bearer ${API_KEY}`,
Accept: 'application/json'
},
timeout: 30000,
retry: {
limit: 3,
statusCodes: [408, 429, 500, 502, 503, 504]
}
});
/**
* Fetch data from the service
*
* @param query - Search query string
* @returns Processed response data
* @throws {FatalError} If authentication fails or resource not found
* @throws {ValidationError} If temporary error occurs
*/
export async function fetchServiceData(query: string): Promise<ServiceResponse> {
const response = await client.get('endpoint', {
searchParams: { q: query }
});
const data = await response.json();
if (!data.results) {
throw new FatalError('No results returned from service');
}
return data;
}
Class-Based Client
import { FatalError, ValidationError } from '@outputai/core';
import { createKyClient } from '@outputai/core/http';
import { credentials } from '@outputai/core/credentials';
export interface ServiceOptions {
model?: string;
timeout?: number;
}
export class ServiceClient {
private readonly client: ReturnType<typeof createKyClient>;
private readonly model: string;
constructor(apiKey?: string) {
const key = apiKey ?? credentials.require('service.api_key');
this.client = createKyClient({
prefix: 'https://api.service.com',
headers: {
Authorization: `Bearer ${key}`,
'Content-Type': 'application/json'
},
timeout: 30000,
retry: {
limit: 3,
statusCodes: [408, 429, 500, 502, 503, 504]
}
});
this.model = 'default-model';
}
async process(input: ProcessInput): Promise<ProcessOutput> {
try {
const response = await this.client.post('process', {
json: {
model: this.model,
input
}
});
return await response.json();
} catch (error: unknown) {
const err = error as { status?: number; message?: string };
if (err.status === 429) {
throw new ValidationError(`Rate limit exceeded: ${err.message}`);
}
if (err.status === 401 || err.status === 403) {
throw new FatalError(`Authentication failed: ${err.message}`);
}
throw new ValidationError(`Service call failed: ${err.message}`);
}
}
}
Real-World Examples
Example 1: Jina Client (Function-Based)
import { FatalError } from '@outputai/core';
import { createKyClient } from '@outputai/core/http';
import { credentials } from '@outputai/core/credentials';
const JINA_API_KEY = credentials.require('jina.api_key');
const JINA_BASE_URL = 'https://r.jina.ai';
const client = createKyClient({
prefix: JINA_BASE_URL,
headers: {
Authorization: `Bearer ${JINA_API_KEY}`,
Accept: 'application/json'
},
timeout: 30000,
retry: {
limit: 3,
statusCodes: [408, 413, 429, 500, 502, 503, 504]
}
});
/**
* Parse PDF resume using Jina Reader API
*/
export async function parseResumeWithJina(base64Pdf: string): Promise<string> {
const response = await client.post('', {
json: { pdf: base64Pdf },
headers: {
'Content-Type': 'application/json'
}
});
const data: {
data: {
content: string;
title?: string;
};
} = await response.json();
if (!data.data?.content) {
throw new FatalError('No content returned from Jina PDF parser');
}
return data.data.content;
}
/**
* Scrape text content from URL using Jina Reader
*/
export async function scrapeTextWithJina(url: string): Promise<string> {
const response = await client.get(url, {
headers: {
'X-Return-Format': 'text',
'X-No-Cache': 'true',
'X-Timeout': '30'
}
});
const data: {
data: {
text?: string;
content?: string;
};
} = await response.json();
const textContent = data.data?.text || data.data?.content;
if (!textContent) {
throw new FatalError(`No text content returned from URL: ${url}`);
}
return textContent;
}
Example 2: Gemini Client (Class-Based)
import { GoogleGenerativeAI } from '@google/generative-ai';
import { FatalError, ValidationError } from '@outputai/core';
import { credentials } from '@outputai/core/credentials';
export interface GeminiImageGenerationOptions {
prompt: string;
referenceImages?: Array<{
inlineData: {
mimeType: string;
data: string;
};
}>;
aspectRatio?: string;
resolution?: string;
numberOfImages?: number;
}
export class GeminiImageService {
private readonly client: GoogleGenerativeAI;
// current as of 2026-05-04 — run output-dev-model-selection for the latest
private readonly model: string = 'gemini-3-pro-image';
constructor(apiKey = credentials.require('google.api_key')) {
if (!apiKey) {
throw new FatalError(
'GeminiImageService: No API Key provided (google.api_key credential).'
);
}
this.client = new GoogleGenerativeAI(apiKey);
}
async generateImage(options: GeminiImageGenerationOptions): Promise<string[]> {
const { prompt, referenceImages = [], aspectRatio = '1:1', resolution = '1K', numberOfImages = 1 } = options;
try {
const model = this.client.getGenerativeModel({ model: this.model });
const parts: Array<{ text: string } | { inlineData: { mimeType: string; data: string } }> = [];
if (referenceImages.length > 0) {
referenceImages.forEach(img => parts.push(img));
}
const finalPrompt = `${prompt}\n\nGenerate this as a ${aspectRatio} aspect ratio image at ${resolution} resolution.`;
parts.push({ text: finalPrompt });
const result = await model.generateContent({
contents: [{ role: 'user', parts }],
generationConfig: {
temperature: 1.0,
topP: 0.95,
candidateCount: numberOfImages,
maxOutputTokens: 8192
}
});
const images: string[] = [];
const candidates = result.response.candidates || [];
for (const candidate of candidates) {
if (candidate.content?.parts) {
for (const part of candidate.content.parts) {
if (part.inlineData?.data) {
images.push(part.inlineData.data);
}
}
}
}
if (images.length === 0) {
throw new ValidationError('No images were generated by Gemini');
}
return images;
} catch (error: unknown) {
const err = error as { status?: number; message?: string };
if (err.status === 429) {
throw new ValidationError(`Gemini rate limit exceeded: ${err.message}`);
}
if (err.status === 401 || err.status === 403) {
throw new FatalError(`Gemini authentication failed: ${err.message}`);
}
throw new ValidationError(`Gemini image generation failed: ${err.message}`);
}
}
}
Error Handling Patterns
HTTP Status Code Handling
const RETRY_STATUS_CODES = [408, 429, 500, 502, 503, 504];
const FATAL_STATUS_CODES = [401, 403, 404];
const client = createKyClient({
retry: {
limit: 3,
statusCodes: RETRY_STATUS_CODES
},
hooks: {
beforeError: [
( { error } ) => {
const status = error.response?.status;
const message = error.message;
if (status && FATAL_STATUS_CODES.includes(status)) {
throw new FatalError(`HTTP ${status} error: ${message}`);
}
throw new ValidationError(`HTTP request failed: ${message}`);
}
]
}
});
Error Type Guidelines
| Status Code | Error Type | Reason |
|---|---|---|
| 401, 403 | FatalError | Auth failures won't succeed on retry |
| 404 | FatalError | Resource doesn't exist |
| 408 | ValidationError | Timeout, may succeed on retry |
| 429 | ValidationError | Rate limit, will succeed after wait |
| 500+ | ValidationError | Server errors may be temporary |
Attaching Request Cost (Paid APIs)
If the service a client wraps charges money per request, attach that cost to the request's trace so it shows up in npx output workflow cost (see output-dev-workflow-cost) and in any cost:http:request hook handlers (see output-dev-cost-hooks).
Do this on the client itself, in an afterResponse hook — not at each call site. That way every function or method that goes through the client is costed automatically, and nobody integrating with the client later has to remember to do it.
addRequestCost(response, totalUsd) only works on a response that came from outputFetch or createKyClient — including a response.clone() of one, so it's safe to call from inside an afterResponse hook (ky passes hooks a clone). If the response didn't originate from @outputai/core/http, addRequestCost no-ops with a console warning.
There are two common pricing shapes for paid third-party APIs:
Metered APIs (cost is in the response)
Some APIs report what a specific call cost — in a response header or a JSON field (credits used, units billed, etc). Read that field and pass it straight to addRequestCost. Guard against free or cached responses that legitimately report zero cost.
import { addRequestCost, createKyClient } from '@outputai/core/http';
import { credentials } from '@outputai/core/credentials';
const API_KEY = credentials.require('service.api_key');
const client = createKyClient({
prefix: 'https://api.service.com',
headers: { Authorization: `Bearer ${API_KEY}` },
timeout: 30000,
retry: { limit: 3, statusCodes: [408, 429, 500, 502, 503, 504] },
hooks: {
afterResponse: [
async (_request, _options, response) => {
// Errored responses (4xx/5xx) and retried attempts aren't billed the
// same as a successful call — skip them rather than over-counting.
if (!response.ok) {
return;
}
try {
const body = await response.clone().json() as { creditsUsed?: number; cached?: boolean };
// Skip cached or free responses that legitimately cost nothing.
if (body.cached || !body.creditsUsed) {
return;
}
const usdPerCredit = 0.01;
addRequestCost(response, body.creditsUsed * usdPerCredit);
} catch {
// Never let cost tracking break the request.
}
}
]
}
});
Flat-rate / subscription APIs (assign a notional cost)
When a service is billed as a flat monthly fee or subscription tier, there's no true marginal cost per call — but recording some per-call figure keeps spend comparable across services in workflow cost and any cost dashboards. Assign a notional per-call USD rate as a constant, overridable by a non-secret environment variable so it can be tuned without a code change.
import { addRequestCost, createKyClient } from '@outputai/core/http';
import { credentials } from '@outputai/core/credentials';
const API_KEY = credentials.require('service.api_key');
// Notional per-call cost for a flat-rate subscription. Not a secret, so a
// plain env var is fine here — unlike API keys, which must go through
// @outputai/core/credentials.
const NOTIONAL_COST_USD = Number(process.env.SERVICE_NOTIONAL_COST_USD ?? 0.005);
const client = createKyClient({
prefix: 'https://api.service.com',
headers: { Authorization: `Bearer ${API_KEY}` },
timeout: 30000,
retry: { limit: 3, statusCodes: [408, 429, 500, 502, 503, 504] },
hooks: {
afterResponse: [
(_request, _options, response) => {
// Errored responses (4xx/5xx) and retried attempts aren't billed the
// same as a successful call — skip them rather than over-counting.
if (!response.ok) {
return;
}
addRequestCost(response, NOTIONAL_COST_USD);
}
]
}
});
The constant above is a placeholder — config/costs.yml (see output-dev-workflow-cost) can override the actual dollar figure per service afterward, without another code change. Add a services.<name> entry with type: request, a matching url_pattern, and an endpoints.<name> block giving pattern + price for the call. (default_price only applies inside a models/model_path entry, for picking a fallback price when the request's model isn't in models — it does nothing paired with a bare url_pattern.) Prefer that file over inventing more env vars once you have more than one or two notional rates to tune.
Rules:
- Never
throwfrom anafterResponsecost hook — a throw there fails the request itself. Wrap body parsing intry/catchand silently skip on parse failure. - The hook fires once per attempt, including retries — skip
!response.okattempts unless you know the API bills failed requests too (the framework itself does count a recorded cost event "regardless of HTTP status" once one exists — the event is proof of a charge). When in doubt, check the provider's billing docs before adding the guard. - Only do this for paid third-party APIs. Free or internal services don't need it.
- Skip this pattern for LLM providers — those costs are computed automatically from token usage via
llm:generation:metering.addRequestCostis for non-LLM HTTP calls only. - To consume these costs elsewhere (forward to your own observability system, log them, alert on them), see
output-dev-cost-hooks.
Best Practices
1. Use Credentials for API Keys
Prefer @outputai/core/credentials over process.env for secrets management. See output-dev-credentials skill for details.
import { credentials } from '@outputai/core/credentials';
// credentials.require() throws MissingCredentialError if not found
const apiKey = credentials.require('service.api_key');
// credentials.get() returns undefined or default if not found
const region = credentials.get('aws.region', 'us-east-1');
2. Document Functions with JSDoc
/**
* Fetch user profile from external service
*
* @param userId - Unique user identifier
* @returns User profile data
* @throws {FatalError} If user not found or auth fails
* @throws {ValidationError} If temporary error occurs
*
* @example
* const profile = await fetchUserProfile('user-123');
*/
export async function fetchUserProfile(userId: string): Promise<UserProfile> {
// ...
}
3. Use Consistent Timeouts
// Standard timeout: 30 seconds
timeout: 30000
// Long-running operations: 60 seconds
timeout: 60000
4. Consume or Cancel Response Bodies
createKyClient follows Fetch response-body semantics: callers own returned response bodies. Prefer body readers like .json() or .text()
when the payload is needed. If a request only reads metadata such as response.url, response.status, or headers, cancel the
unused body in a finally block.
const response = await client.get( url );
try {
return response.url;
} finally {
await response.body?.cancel();
}
HEAD requests do not have response bodies, so this is only needed for methods that can return one.
5. Export TypeScript Interfaces
// Export interfaces for consumers
export interface ServiceResponse {
data: {
id: string;
content: string;
};
metadata: {
processedAt: string;
};
}
Verification Checklist
- Client file located in
src/shared/clients/directory - File named
{service}_client.ts -
createKyClientimported from@outputai/core/http(not axios) -
FatalErrorandValidationErrorimported from@outputai/core - API key validation in constructor/initialization
- Retry configuration for appropriate status codes
- FatalError used for 401, 403, 404 responses
- ValidationError used for 429, 5xx responses
- Non-HEAD responses are consumed or cancelled when only metadata is used
- JSDoc documentation for exported functions
- TypeScript interfaces exported for response types
- Paid third-party API calls attach cost via
addRequestCostin anafterResponsehook
Related Skills
output-dev-step-function- Using clients in step functionsoutput-dev-evaluator-function- Using clients in evaluatorsoutput-dev-folder-structure- Understanding project layoutoutput-dev-credentials- Encrypted secrets managementoutput-dev-cost-hooks- Subscribing to cost events (cost:http:request,llm:generation:metering) for observabilityoutput-error-http-client- Troubleshooting HTTP issuesoutput-error-try-catch- Proper error handling patterns
LLM Text Streaming
When to Use This Skill
- Adding token or chunk progress to an LLM-powered step
- Choosing between completed generation and direct stream access
- Using
onChunk, oronEnd/onErroronstreamText()/Agent.stream() - Making stream failures trigger Temporal activity retries
- Streaming Agent responses or persisting streamed conversations
Choose the API
| Need | Use |
|---|---|
| Complete single-shot result | generateText() |
Complete result plus onChunk progress | generateTextWithStreaming() |
Direct access to textStream or stream | streamText() |
Complete Agent result plus onChunk progress | Agent.generateWithStreaming() |
| Direct access to the Agent stream | Agent.stream() |
In workflow steps, prefer generateTextWithStreaming() or Agent.generateWithStreaming() when onChunk progress is sufficient. They consume the stream internally, return complete results like generateText() or Agent.generate(), and reject on provider, transport, or abort errors. Rejection allows Temporal to record the failed activity attempt and apply the step retry policy.
streamText() and Agent.stream() remain supported for code that needs direct control over stream consumption.
generateTextWithStreaming()
import { generateTextWithStreaming } from '@outputai/core/llm';
const result = await generateTextWithStreaming( {
prompt: 'draft@v1',
variables: { topic },
onChunk( { chunk } ) {
if ( chunk.type === 'text-delta' ) {
process.stdout.write( chunk.text );
}
}
} );
return result.result;
The result has the same complete response fields as generateText(), including result, text, output, usage, finishReason, and cost. Structured output passed with aiSdk.Output.* is available through result.output.
Agent.generateWithStreaming()
const result = await agent.generateWithStreaming( {
onChunk( { chunk } ) {
if ( chunk.type === 'text-delta' ) {
process.stdout.write( chunk.text );
}
}
} );
generateWithStreaming() returns a complete Agent response and automatically stores messages when the Agent has a messageStore.
Direct stream error handling
AI SDK streaming delivers provider and transport failures through onError. Iterating textStream does not reliably throw the original error. When using streamText() in a workflow step, capture the error and throw it after consumption:
import { streamText } from '@outputai/core/llm';
const captured: { error: unknown } = { error: null };
const result = streamText( {
prompt: 'draft@v1',
variables: { topic },
onError( { error } ) {
captured.error = error;
}
} );
const chunks: string[] = [];
for await ( const chunk of result.textStream ) {
chunks.push( chunk );
}
if ( captured.error ) {
throw captured.error;
}
return chunks.join( '' );
Registering onError without throwing the captured error can let the step return an empty successful result, preventing Temporal from retrying it. Awaiting a completion property may also produce a generic no-output error instead of the original provider error.
Agent.stream() stores conversation messages in its wrapped onEnd when finishReason is not 'error'. Use Agent.generateWithStreaming() when a complete stored response meets the requirement.
Streaming call arguments: prompt, promptDir, variables, tools, output, toolChoice, stopWhen, abortSignal, plus onChunk (generateTextWithStreaming) or onChunk / onEnd / onError (streamText). Agent methods: messages, abortSignal, toolChoice, plus those same stream callbacks.
Rules
- Prefer the completed streaming APIs in Temporal steps unless direct stream access is required.
- Do not rely on
onErroralone to fail a step usingstreamText(). - Throw the captured error only after stream consumption finishes.
- Keep
onChunkside effects bounded. A Temporal signal per token creates a history event per signal, so batch high-frequency updates. - Do not describe
streamText()orAgent.stream()as deprecated.
Related Skills
output-dev-step-function- Put LLM calls inside Temporal activity stepsoutput-dev-agent-class- Construct and use reusable Agentsoutput-dev-prompt-file- Create prompt files for generationoutput-error-try-catch- Handle step and workflow failures
Picking a Model for an Output SDK Prompt
This skill is the single source of truth for model selection across Output SDK skills and agents. Other skills link here instead of pinning specific model IDs, because model rosters drift faster than docs.
Live model snapshot
We run this at skill-load time to fetch the 10 most recently released models per provider from the Vercel AI Gateway:
output=$(curl -fsS https://ai-gateway.vercel.sh/v1/models 2>/dev/null | jq '
.data as $models
| {
anthropic: ([ $models[] | select(.id | startswith("anthropic/")) ] | sort_by(.released) | reverse | .[0:10]),
openai: ([ $models[] | select(.id | startswith("openai/")) ] | sort_by(.released) | reverse | .[0:10]),
google: ([ $models[] | select(.id | startswith("google/")) ] | sort_by(.released) | reverse | .[0:10])
}
' 2>/dev/null)
if [ -n "$output" ]; then printf '%s\n' "$output"; else echo "(snapshot unavailable)"; fi
Snapshot Data
output=$(curl -fsS https://ai-gateway.vercel.sh/v1/models 2>/dev/null | jq '
.data as $models
| {
anthropic: ([ $models[] | select(.id | startswith("anthropic/")) ] | sort_by(.released) | reverse | .[0:10]),
openai: ([ $models[] | select(.id | startswith("openai/")) ] | sort_by(.released) | reverse | .[0:10]),
google: ([ $models[] | select(.id | startswith("google/")) ] | sort_by(.released) | reverse | .[0:10])
}
' 2>/dev/null)
if [ -n "$output" ]; then printf '%s\n' "$output"; else echo "(snapshot unavailable)"; fi
Snapshot Fallback
If the block above is empty, the script didn't execute automatically — likely because part of it (jq, curl, or network access) is missing. Query and filter the snapshot yourself before continuing.
Snapshot shape
{
"anthropic": [ <model>, ..., <up to 10> ],
"openai": [ <model>, ..., <up to 10> ],
"google": [ <model>, ..., <up to 10> ]
}
Each <model> is the unmodified gateway payload. Useful fields per model:
| Field | What to use it for |
|---|---|
id | The provider-prefixed ID (eg anthropic/claude-sonnet-4.6) — translate to prompt-file form (Step 5) |
released | Unix timestamp of release. Snapshot is already sorted newest-first per provider. |
name | Human-readable name |
description | One-paragraph capability summary — read this when comparing similarly-named tiers |
context_window | Max input tokens. Matters when prompts include large context (codebases, long docs) |
max_tokens | Max single-response output tokens |
tags | Capability flags. reasoning, tool-use, vision, file-input, web-search, image-generation, explicit-caching, implicit-caching |
pricing.input / pricing.output | Per-token cost (USD). Multiply by 1,000,000 for "per 1M tokens" |
pricing.input_cache_read | Cached-input price — usually 10× cheaper than input |
type | language for chat models; image models surface as image-generation and aren't valid for .prompt files |
Decision flow
Step 1 — Determine task priority
Pick the first row that fits. If unclear, default to reasoning.
| Priority | Use when |
|---|---|
| reasoning (default) | Complex multi-step logic, structured output extraction, judges with edge cases, anything where wrong > slow |
| balance | Most generative work — summarization, classification, content drafting, conversation |
| speed | Short interactive responses, low-latency UI loops, simple transforms |
| cost | Bulk batch processing where token spend dominates and quality floor is forgiving |
Step 2 — Determine provider
Scan existing *.prompt files in the workflow (and its siblings under src/workflows/) and tally what provider: they declare.
- If the workflow (or sibling workflows) already use one provider, match it. Mixing providers means the runtime needs API keys for each — operational footgun.
- If no existing prompts, default to
anthropic. - Only switch provider when the user explicitly asks, or when a feature you need (eg Gemini's
useSearchGrounding, OpenAI'smaxToolCalls) is provider-specific. - When the chosen provider's
@ai-sdk/*package (eg@ai-sdk/google-vertex) is not inpackage.json, install it. New projects ship with@ai-sdk/anthropicand@ai-sdk/openai.
Step 3 — Map provider name to snapshot key
Output SDK provider: values don't always line up with the snapshot keys, since Vercel groups Gemini under google/:
| Output SDK provider | Snapshot key |
|---|---|
anthropic | anthropic |
openai | openai |
google-vertex (Gemini models) | google |
google-vertex (Claude models) | anthropic (then re-add the @vertex suffix manually) |
amazon-bedrock | anthropic (then translate to bedrock namespace manually) |
Step 4 — Pick a model from the provider's list
The list is already sorted newest-first. Walk it top-down and pick the first model whose id matches the tier for your priority.
Skip these by default:
type != "language"(eggpt-image-2,gemini-embedding-2) — not valid for.promptfiles.- IDs containing
preview,alpha, orbeta. Use stable / GA models only, even if a newer preview/alpha/beta exists. Only pick a non-stable model when the user explicitly asks for it ("use the preview", "I want the new beta", etc.).
| Priority | Anthropic — match id containing | OpenAI — match id | Google — match id |
|---|---|---|---|
| reasoning | claude-opus- (and tags includes reasoning) | ends with -pro | contains -pro |
| balance | claude-sonnet- | base gpt-N.M (no -mini/-nano/-pro suffix) | contains -pro |
| speed | claude-haiku- | ends with -mini | ends with -flash (not -flash-lite) |
| cost | claude-haiku- | ends with -nano | contains -flash-lite |
Tie-breakers when multiple stable models match:
- Prefer the unversioned alias (
claude-sonnet-4.6) over a dated snapshot (claude-sonnet-4-20250514) unless reproducibility is required (eg eval judges). - If two truly equivalent rows exist, take the one with the larger
context_window, then lowerpricing.input.
If every match in the snapshot is a preview/alpha/beta — meaning the entire tier is in pre-release — surface that to the user and ask before picking one. Don't silently use a preview because it was the only thing available.
Step 5 — Translate the gateway ID into a prompt-file model string
Gateway IDs carry a provider prefix and use dots; prompt-file IDs strip the prefix and use hyphens. Apply two transformations: drop everything up to and including the first /, then replace . with -.
Gateway id | Prompt-file model: |
|---|---|
anthropic/claude-sonnet-4.6 | claude-sonnet-4-6 |
openai/gpt-5.5 | gpt-5-5 |
google/gemini-3-flash | gemini-3-flash |
Drop the translated string into your .prompt frontmatter:
---
provider: anthropic
model: claude-sonnet-4-6
temperature: 0.7
maxOutputTokens: 4096
---
See also
output-dev-prompt-file— overall.promptfile structureoutput-dev-upgrade-prompt-models— bulk-upgrade existing prompts to the latest version of their current familyoutput-eval-judge-prompt— judge-specific selection guidance (start small, escalate on TPR/TNR failures)
Creating .prompt Files
Overview
This skill documents how to create .prompt files for LLM operations in Output SDK workflows. Prompt files use YAML frontmatter for configuration and Liquid.js templating for dynamic content.
When to Use This Skill
- Creating prompts for LLM-powered workflow steps
- Configuring LLM provider settings (model, temperature, etc.)
- Using template variables in prompts
- Troubleshooting prompt formatting issues
Location Convention
Prompt files are stored INSIDE the workflow folder:
src/workflows/{workflow-name}/
├── workflow.ts
├── steps.ts
├── types.ts
└── prompts/
├── analyzeContent@v1.prompt
├── generateSummary@v1.prompt
└── extractData@v2.prompt
Important: Prompts are workflow-specific and live inside the workflow folder, NOT in a shared location.
File Naming Convention
{promptName}@v{version}.prompt
Examples:
generateImageIdeas@v1.promptanalyzeContent@v1.promptsummarizeText@v2.prompt
The version suffix (@v1, @v2) allows for prompt versioning without breaking existing code.
Basic Structure
Picking a model? See
output-dev-model-selectionfor the current decision tree and AI Gateway lookup script. Examples below show concrete IDs as of 2026-05-04 — refresh them with that skill.
---
provider: anthropic
# current as of 2026-05-04 — run output-dev-model-selection for the latest
model: claude-sonnet-4-6
temperature: 0.7
maxOutputTokens: 4096
---
<system>
System instructions go here.
</system>
<user>
User message with {{ variable }} placeholders.
</user>
The body uses exactly one mode:
- Message mode starts with a role tag and produces
messages. Use it withgenerateText,generateTextWithStreaming,streamText, andAgent. - Instruction mode starts with plain text and produces
instructions. Use it withgenerateImageor when consumingloadPrompt()results directly:
---
provider: openai
model: gpt-image-1
---
Create a cinematic image of {{ subject }}.
Leading whitespace and HTML comments do not affect mode selection. Once plain text selects instruction mode, later tag-shaped text remains part of the instructions.
YAML Frontmatter Options
Required Fields
---
provider: anthropic # LLM provider: anthropic, openai, google-vertex, amazon-bedrock, azure, perplexity
# current as of 2026-05-04 — run output-dev-model-selection for the latest
model: claude-sonnet-4-6
---
Provider Consistency
All prompt files in a workflow should use the same provider unless the user explicitly requests otherwise. Mixing providers (e.g., some prompts using anthropic and others using openai) requires the user to have API keys for all providers, which causes runtime failures if they don't.
Each provider also needs its AI SDK package in the project's package.json: @ai-sdk/anthropic, @ai-sdk/openai, @ai-sdk/google-vertex, @ai-sdk/amazon-bedrock, @ai-sdk/azure, or @ai-sdk/perplexity. New projects ship with @ai-sdk/anthropic and @ai-sdk/openai. When a prompt uses a provider whose package is missing, install it; otherwise the call fails with Provider "<name>" requires the "<package>" package, which is not installed. Under output dev, the worker only picks up new packages after a restart: run output dev down, then output dev.
When no existing prompts dictate a provider, default to anthropic. For the model itself, see output-dev-model-selection — it walks priority (reasoning/balance/speed/cost), provider lookup, and produces a current model ID.
Optional Fields
---
provider: anthropic
# current as of 2026-05-04 - run output-dev-model-selection for the latest
model: claude-sonnet-4-6
temperature: 0.7 # Supported range and default vary by provider
maxOutputTokens: 4096 # Maximum output tokens
maxSteps: 5 # Tool-loop ceiling when tools or skills are present (default 10)
skills: # Skill file or directory paths, relative to this prompt
- ./skills
providerOptions: # Provider-specific options
thinking:
type: enabled
budgetTokens: 2000
---
Frontmatter is a strict camelCase allowlist. Unknown top-level keys throw Invalid prompt file. A snake_case alias of a known field fails with a suggestion (max_output_tokens -> use maxOutputTokens). Put provider-specific keys (effort, reasoningEffort) under providerOptions, which stays open. Nested thinking stays open too (budgetTokens is the documented key; extra nested keys are not rejected as unknown top-level config).
Allowed top-level keys: provider, model, temperature, maxOutputTokens, deprecated maxTokens, topP, topK, presencePenalty, frequencyPenalty, stopSequences, seed, maxSteps, skills, tools, providerOptions, messageOptions, n, maxImagesPerCall, size, aspectRatio.
Use maxOutputTokens for new prompts. Deprecated maxTokens remains on the loaded config and populates maxOutputTokens when the canonical key is absent; when both are set, maxOutputTokens takes precedence.
Call arguments: prompt, promptDir, variables, tools, output, toolChoice, stopWhen, abortSignal on generateText (plus onChunk on generateTextWithStreaming; plus onChunk / onEnd / onError on streamText). generateImage: prompt, promptDir, variables, images, mask, abortSignal.
Common Provider Configurations
Each example below pins a model that was current as of 2026-05-04. Run
output-dev-model-selectionwhen picking or refreshing.
Anthropic (Claude)
---
provider: anthropic
model: claude-sonnet-4-6
temperature: 0.7
maxOutputTokens: 8192
---
Anthropic with Extended Thinking
---
provider: anthropic
model: claude-sonnet-4-6
temperature: 0.7
maxOutputTokens: 32000
providerOptions:
thinking:
type: enabled
budgetTokens: 2000
---
OpenAI
---
provider: openai
# current as of 2026-05-04 — run output-dev-model-selection for the latest
model: gpt-5-5
temperature: 0.7
maxOutputTokens: 4096
---
Google Vertex (Gemini)
---
provider: google-vertex
# current as of 2026-05-04 — run output-dev-model-selection for the latest
model: gemini-3-pro
temperature: 0.7
maxOutputTokens: 8192
---
Message Blocks
Message mode uses a small XML-like syntax, not a general HTML or XML parser. The only valid top-level role tags are <system>, <user>, and <assistant>.
Do not author <tool> blocks. AI SDK tool results are structured message parts tied to a preceding tool call; AI SDK creates them during execution, and Agent callers may supply them through messages or messageStore.
Follow these parser rules:
- Put only whitespace or HTML comments between top-level role blocks. Root text and self-closing blocks are invalid.
- Close every top-level role block with the matching tag.
- Different-name tags inside a message, such as
<context>inside<user>, remain message content. - Do not nest a non-self-closing tag with the same name as its containing message. Escape literal examples as
<user>example</user>. - Code and HTML-like text inside a message are preserved, including forms such as
Array<string>. - On role tags, only the
optionsattribute is supported. It must have a value naming one or more frontmattermessageOptionssets, for exampleoptions="cached fast". Bareoptionsand unknown attributes throw when the prompt loads.
System Message
<system>
You are an expert at analyzing technical content.
Your responses should be clear and structured.
</system>
User Message
<user>
Please analyze the following content:
{{ content }}
</user>
Assistant Message (for few-shot examples)
<assistant>
I'll analyze this content step by step...
</assistant>
Liquid.js Templating
Variable Substitution
<user>
Analyze this content about {{ topic }}:
{{ content }}
Generate {{ numberOfIdeas }} ideas.
</user>
Conditional Content
<system>
You are an expert content analyzer.
{% if colorPalette %}
**Color Palette Constraints:** {{ colorPalette }}
{% endif %}
{% if artDirection %}
**Art Direction Constraints:** {{ artDirection }}
{% endif %}
</system>
Loops
<user>
Analyze each of these items:
{% for item in items %}
- {{ item.name }}: {{ item.description }}
{% endfor %}
</user>
Default Values
<user>
Generate {{ numberOfIdeas | default: 3 }} ideas for {{ topic }}.
</user>
Complete Example
Based on a real prompt file (generateImageIdeas@v1.prompt):
---
provider: anthropic
# current as of 2026-05-04 — run output-dev-model-selection for the latest
model: claude-sonnet-4-6
temperature: 0.7
maxOutputTokens: 32000
providerOptions:
thinking:
type: enabled
budgetTokens: 2000
---
<system>
You are an expert at creating structured, precise infographic prompts optimized for Gemini's image generation model.
Your task is to generate prompts for informational infographics that illustrate key concepts from the provided content.
CRITICAL RULES you MUST follow:
- Use Markdown dashed lists to specify constraints
- Use ALL CAPS for "MUST" requirements to ensure strict adherence
- Include specific compositional constraints (e.g., rule of thirds, lighting)
- Always include negative constraints to prevent unwanted elements
- Keep each infographic focused on ONE clear concept
{% if colorPalette %}
**Color Palette Constraints:** {{ colorPalette }}
{% endif %}
{% if artDirection %}
**Art Direction Constraints:** {{ artDirection }}
{% endif %}
</system>
<user>
Generate {{ numberOfIdeas }} structured infographic prompts based on key topics from this content.
<content>
{{ content }}
</content>
Each prompt MUST follow this structure:
Create an infographic about [specific topic]. The infographic MUST follow ALL of these constraints:
- The infographic MUST use the reference images as a visual style guide
- The composition MUST follow the rule of thirds for visual balance
- The infographic MUST use clean, minimal design with simple lines and shapes
{% if colorPalette %}- The color palette MUST strictly follow: {{ colorPalette }}{% endif %}
{% if artDirection %}- The art direction MUST strictly follow: {{ artDirection }}{% endif %}
- NEVER include any watermarks, logos, or decorative overlays
- NEVER use generic AI art buzzwords like "hyperrealistic"
Focus on the most important concepts that would benefit from visual explanation.
</user>
Structured Variables
The variables field in generateText and Agent accepts scalars, nested objects, and arrays. Pass structured data directly when the prompt benefits from Liquid loops, conditions, or dot notation:
const { output } = await generateText( {
prompt: 'rank@v1',
variables: {
stories: storyArray,
interests: interestArray
}
} );
{% for story in stories %}
- {{ story.title }} (score: {{ story.score }}, by: {{ story.author }})
{% endfor %}
Interests: {{ interests | join: ", " }}
Pre-format data in the step only when the exact rendered text is application logic rather than prompt presentation.
Using Prompts in Steps
With generateText and aiSdk.Output.object()
import { generateText, aiSdk } from '@outputai/core/llm';
import { z } from '@outputai/core';
const { output } = await generateText( {
prompt: 'generateImageIdeas@v1', // References prompts/generateImageIdeas@v1.prompt
variables: {
content: 'Solar panel technology explained...',
numberOfIdeas: 3,
colorPalette: 'blue and green tones',
artDirection: 'minimalist style'
},
output: aiSdk.Output.object( {
schema: z.object( {
ideas: z.array( z.string() )
} )
} )
} );
// output contains { ideas: [...] }
With generateText
import { generateText } from '@outputai/core/llm';
const { result } = await generateText( {
prompt: 'summarize@v1',
variables: {
content: 'Long article text...',
maxLength: 200
}
} );
// result contains the generated text string
Using Skills with Prompts
Prompts can load skill files that provide lazy-loaded instructions to the LLM. Skills keep the initial context small while giving the LLM access to deep expertise on demand. See output-dev-skill-file for the full guide on creating skill files.
Place .md files next to the prompt (commonly in prompts/skills/) and list the path in frontmatter. A sibling skills/ folder is not loaded unless you list it:
src/workflows/{workflow-name}/
└── prompts/
├── writing_assistant@v1.prompt
└── skills/
├── clarity_guidelines.md
└── structure_guide.md
---
provider: anthropic
model: claude-sonnet-4-6
skills:
- ./skills
---
Mention load_skill in the system message so the LLM knows to use it:
<system>
You are an expert technical writing assistant.
Use load_skill to get the full instructions for any skill before applying it.
</system>
List skills: paths in this prompt's frontmatter. See output-dev-skill-file for the file format and path rules.
Using Prompts with Agent
Prompts work with both generateText and the Agent class. Use Agent for multi-step tool loops and stateful conversations. See output-dev-agent-class for the full guide.
import { Agent, aiSdk } from '@outputai/core/llm';
const agent = new Agent( {
prompt: 'writing_assistant@v1',
variables: {
content_type: 'documentation',
focus: 'clarity',
content: input.content
},
output: aiSdk.Output.object( { schema: reviewSchema } )
} );
const { output } = await agent.generate();
CRITICAL: Prompts and Structured Output Schemas
Do Not Duplicate the Schema in the Prompt
When a step uses aiSdk.Output.object() with generateText, the Zod schema is automatically sent to the LLM provider as a tool definition. The LLM already knows the exact JSON shape it must return. Do not also specify the schema in the prompt.
This is a best practice documented by multiple LLM providers:
- Anthropic: The schema is sent as a tool definition;
.describe()on fields is how you guide the model's output. The SDK automatically transforms unsupported constraints into field descriptions. - Google Vertex AI: "Only specify the schema in the schema object. Don't also specify the schema in the prompt. Doing both can reduce performance." If you must discuss the schema in the prompt, match the exact field order from the schema.
Why this matters:
- Performance: Redundant schema instructions can confuse the model and reduce output quality
- Maintenance: When the schema changes, you must update both the schema AND the prompt, or they drift apart
- Correctness: The prompt's JSON examples can contradict the actual schema (wrong field names, missing fields, wrong types)
What NOT to Include in Prompts
When aiSdk.Output.object() is used, do not include any of these in the prompt:
## Output Formatsections describing the JSON shape- JSON examples showing the expected response structure
- Field-by-field descriptions that mirror the schema
- Instructions like "Return a JSON object with exactly these fields"
- Instructions like "Return only the JSON object with no surrounding explanation"
<!-- WRONG - prompt duplicates what aiSdk.Output.object() already sends -->
<system>
## Output Format
Return a JSON object with this shape:
{
"title": "string",
"summary": "string",
"tags": ["string"]
}
</system>
What TO Include in Prompts
Use the prompt for quality expectations, domain knowledge, and content guidance -- things the schema cannot express:
<!-- CORRECT - prompt focuses on content quality, not structure -->
<system>
Write a concise, specific title (under 80 characters).
The summary should capture the main argument, not just the topic.
Choose tags from the reader's domain -- avoid generic terms like "technology".
</system>
Use .describe() on Schema Fields Instead
The right place to communicate field-level expectations is on the schema itself, using .describe(). LLM providers use these descriptions when generating output:
// In types.ts -- .describe() guides the LLM on each field
const ArticleSummarySchema = z.object( {
title: z.string().describe( 'Concise title under 80 characters' ),
summary: z.string().describe( 'One-sentence summary capturing the main argument' ),
tags: z.array( z.string() ).describe( '3-5 domain-specific tags, avoid generic terms' )
} );
The schema handles structure AND field-level guidance; the prompt handles task framing, methodology, and quality standards.
When the Step Does NOT Use aiSdk.Output.object()
If generateText is called without aiSdk.Output.object() (plain text output), then including output format instructions in the prompt is appropriate since no schema is sent to the provider.
Best Practices
1. Be Explicit About Requirements
<system>
CRITICAL RULES you MUST follow:
- Rule 1
- Rule 2
- NEVER do X
- ALWAYS do Y
</system>
2. Use XML Tags for Structure in User Messages
<user>
Analyze the following:
<content>
{{ content }}
</content>
<requirements>
{{ requirements }}
</requirements>
</user>
3. Provide Examples (Few-Shot)
<system>
You analyze sentiment. Return: positive, negative, or neutral.
</system>
<user>
"I love this product!"
</user>
<assistant>
positive
</assistant>
<user>
"{{ text }}"
</user>
4. Version Your Prompts
When making significant changes, create a new version:
analyzeContent@v1.prompt- OriginalanalyzeContent@v2.prompt- Improved with better examples
Update the step to use the new version:
prompt: 'analyzeContent@v2' // Changed from v1
5. Handle Optional Variables
{% if optionalField %}
Additional context: {{ optionalField }}
{% endif %}
Common Patterns
The model lines in the patterns below were current as of 2026-05-04. Refresh via
output-dev-model-selectionwhen copying into a new prompt.
Classification Prompt
---
provider: anthropic
model: claude-sonnet-4-6
temperature: 0.3
---
<system>
You are a content classifier. Categorize content into exactly one category.
Available categories: {{ categories | join: ", " }}
</system>
<user>
Classify this content:
{{ content }}
</user>
Extraction Prompt
---
provider: anthropic
model: claude-sonnet-4-6
temperature: 0.2
---
<system>
You extract structured data from text. Be precise and only include information explicitly stated.
</system>
<user>
Extract the following fields from this text:
{% for field in fields %}
- {{ field }}
{% endfor %}
Text:
{{ text }}
</user>
Generation Prompt
---
provider: anthropic
# current as of 2026-05-04 — run output-dev-model-selection for the latest
model: claude-sonnet-4-6
temperature: 0.8
---
<system>
You are a creative writer. Generate engaging content based on the given parameters.
</system>
<user>
Generate {{ count }} {{ type }} about {{ topic }}.
Requirements:
{{ requirements }}
</user>
Verification Checklist
- File located in
prompts/folder inside workflow directory - File named
{promptName}@v{version}.prompt - YAML frontmatter includes
providerandmodel - Frontmatter uses camelCase only; no unknown top-level keys (
effort,reasoningEffort) or snake_case aliases (max_output_tokens) - The body uses message mode for text generation, or instruction mode for
generateImage/ directloadPrompt()consumption - Message blocks use supported role tags (
<system>,<user>,<assistant>); no authored<tool>blocks - Message mode has no root text between blocks, no self-closing role blocks, and no unclosed blocks
- Literal same-name role tags inside a message are escaped (
<user>...</user>) - Role-tag attributes use only
options="<messageOptions names>";optionsis never bare - Variables use
{{ variableName }}syntax - Conditionals use
{% if %}...{% endif %}syntax - All required variables are documented or have defaults
- Step code references correct prompt name
- No JSON output format instructions when step uses
aiSdk.Output.object()(schema handles structure) - If the prompt uses skills, frontmatter lists
skills:paths (a siblingskills/folder is not auto-loaded) - If the prompt uses skills or tools, set
maxStepswhen the default of 10 is wrong
Related Skills
output-dev-skill-file- Creating skill files for promptsoutput-dev-agent-class- Using the Agent class with promptsoutput-dev-step-function- Using prompts in step functionsoutput-dev-evaluator-function- Using prompts in evaluatorsoutput-dev-folder-structure- Understanding prompts folder locationoutput-dev-workflow-function- Orchestrating LLM-powered stepsoutput-eval-judge-prompt— Methodology for designing effective LLM judge prompts
Creating Scenario Files
Overview
This skill documents how to create test scenario JSON files for Output SDK workflows. Scenarios provide predefined inputs for testing workflows during development and validation.
When to Use This Skill
- Creating test inputs for a new workflow
- Documenting different use cases
- Setting up regression tests
- Debugging workflow behavior with specific inputs
Location Convention
Scenario files are stored INSIDE the workflow folder:
src/workflows/{workflow-name}/
├── workflow.ts
├── steps.ts
├── types.ts
└── scenarios/
├── basic_input.json
├── complex_input.json
└── edge_case_empty.json
Important: Scenarios are workflow-specific and live inside the workflow folder.
File Naming Convention
Use snake_case for scenario file names:
{description}_input.json
Examples:
basic_input.jsontest_input_solar_panels.jsonedge_case_empty_content.jsoncomplex_with_references.json
Naming patterns:
basic_*- Minimal valid inputcomplex_*- Full-featured input with all optionsedge_case_*- Boundary conditions and edge caseserror_*- Inputs expected to produce errors
Basic Structure
A scenario file is a JSON file that matches the workflow's inputSchema:
{
"fieldName": "value",
"optionalField": "optional value",
"numericField": 42,
"arrayField": ["item1", "item2"]
}
Matching inputSchema
The scenario JSON must match the Zod schema defined in types.ts:
Example Schema (types.ts)
export const WorkflowInputSchema = z.object({
content: z.string().describe('Text content to process'),
numberOfIdeas: z.number().min(1).max(10).default(1),
colorPalette: z.string().optional(),
aspectRatio: z.enum(['1:1', '16:9', '9:16']).default('1:1'),
referenceUrls: z.array(z.string()).optional()
});
Corresponding Scenarios
basic_input.json (minimal required fields)
{
"content": "This is sample content for testing the workflow."
}
complete_input.json (all fields specified)
{
"content": "This is sample content for testing the workflow.",
"numberOfIdeas": 3,
"colorPalette": "blue and green tones",
"aspectRatio": "16:9",
"referenceUrls": [
"https://example.com/image1.jpg",
"https://example.com/image2.jpg"
]
}
Real-World Example
Based on image_infographic_nano workflow:
test_input_solar_panels.json
{
"content": "Solar panels work by converting sunlight into electricity through the photovoltaic effect. The process begins when photons from sunlight strike the silicon cells in the panel, knocking electrons loose from their atoms. These free electrons flow through the semiconductor material, creating an electric current. The panels contain multiple layers: a protective glass covering, anti-reflective coating to maximize light absorption, silicon cells (both n-type and p-type layers forming a junction), and a backing material. The DC electricity generated by the panels flows through an inverter, which converts it to AC electricity suitable for home use or feeding back into the power grid. Modern solar panels achieve 15-20% efficiency, meaning they convert that percentage of sunlight into usable electricity. The entire system includes mounting hardware, wiring, inverters, and often battery storage for excess energy.",
"numberOfIdeas": 3,
"aspectRatio": "16:9",
"resolution": "2K",
"numberOfGenerations": 1
}
test_input_complex.json
{
"content": "Detailed explanation of the topic...",
"numberOfIdeas": 5,
"colorPalette": "warm earth tones with orange accents",
"artDirection": "minimalist corporate style",
"aspectRatio": "1:1",
"resolution": "4K",
"numberOfGenerations": 2,
"referenceImageUrls": [
"https://storage.example.com/style-guide.png"
],
"storageNamespace": "test/infographics"
}
Running Scenarios
Using CLI
# Run with scenario file
npx output workflow run workflowName --input path/to/scenarios/basic_input.json
# Run with inline JSON
npx output workflow run workflowName --input '{"content": "test"}'
Example Commands
# Basic scenario
npx output workflow run contentUtilsImageInfographicNano --input src/workflows/content_utils/image_infographic_nano/scenarios/test_input_solar_panels.json
# Complex scenario
npx output workflow run contentUtilsImageInfographicNano --input src/workflows/content_utils/image_infographic_nano/scenarios/test_input_complex.json
Related Skill: output-workflow-run for detailed CLI usage
Scenario Categories
1. Basic/Happy Path
Minimal valid input to verify the workflow works:
{
"content": "Simple test content",
"numberOfIdeas": 1
}
2. Complete/Full-Featured
All optional fields populated:
{
"content": "Detailed content...",
"numberOfIdeas": 5,
"colorPalette": "custom palette",
"artDirection": "specific style",
"aspectRatio": "16:9",
"resolution": "4K",
"numberOfGenerations": 3,
"referenceImageUrls": ["https://example.com/ref.jpg"],
"storageNamespace": "test/folder"
}
3. Edge Cases
Test boundary conditions:
edge_case_min_values.json
{
"content": "x",
"numberOfIdeas": 1
}
edge_case_max_values.json
{
"content": "Very long content string...",
"numberOfIdeas": 10
}
4. Error Cases (for validation testing)
error_missing_required.json
{
"numberOfIdeas": 3
}
Note: Error scenarios won't pass validation but are useful for testing error handling.
Best Practices
1. Document the Purpose
Add a comment field (if supported) or create a companion README:
{
"_comment": "Tests workflow with multiple reference images",
"content": "...",
"referenceImageUrls": ["url1", "url2", "url3"]
}
2. Use Realistic Data
{
"content": "Actual representative content that matches real use cases..."
}
Not:
{
"content": "test"
}
3. Cover All Enum Values
If schema has enums, create scenarios for each:
// scenario_aspect_1x1.json
{ "aspectRatio": "1:1", ... }
// scenario_aspect_16x9.json
{ "aspectRatio": "16:9", ... }
// scenario_aspect_9x16.json
{ "aspectRatio": "9:16", ... }
4. Include Optional Field Variations
// without_optional_fields.json
{ "content": "..." }
// with_all_optional_fields.json
{ "content": "...", "colorPalette": "...", "artDirection": "..." }
// with_some_optional_fields.json
{ "content": "...", "colorPalette": "..." }
5. Create Regression Test Scenarios
Save inputs from bug reports:
// regression_issue_123.json
{
"_issue": "https://github.com/org/repo/issues/123",
"content": "Input that caused the bug..."
}
Scenario Organization
For workflows with many scenarios, organize into subfolders:
scenarios/
├── basic/
│ └── minimal_input.json
├── complete/
│ └── all_options.json
├── edge_cases/
│ ├── empty_array.json
│ └── max_length.json
└── regression/
└── issue_123.json
Verification Checklist
- Scenario file located in
scenarios/folder inside workflow directory - File uses
.jsonextension - File name uses
snake_case - JSON is valid and parseable
- All required fields from inputSchema are present
- Field types match schema (strings, numbers, arrays, etc.)
- Enum values are valid options from schema
- Numbers are within min/max constraints
- At least one basic scenario exists
- Workflow runs successfully with the scenario
Testing Scenarios
Validate JSON Syntax
# Check JSON is valid
cat scenarios/basic_input.json | jq .
Run and Verify
# Run workflow with scenario
npx output workflow run workflowName --input scenarios/basic_input.json
# Check status if async
npx output workflow status <workflowId>
# Get result
npx output workflow result <workflowId>
Related Skills
output-dev-types-file- Defining inputSchema that scenarios must matchoutput-dev-folder-structure- Understanding scenarios folder locationoutput-workflow-run- Running workflows with scenario filesoutput-workflow-list- Finding available workflows
Creating Skill Files
Overview
This skill documents how to create .md skill files for the Output framework's skills system. Skills are lazy-loaded instruction packages that keep prompts lightweight. The LLM sees a list of skill names and descriptions in the system message, then calls a load_skill tool to retrieve full instructions on demand.
Important: These are framework skills (.md files loaded by LLMs at runtime), not Claude Code plugin skills. The naming is similar but the systems are separate.
When to Use This Skill
- Adding reusable instruction sets to LLM prompts
- Listing skill paths in prompt frontmatter
- Debugging skill resolution or
load_skilltool issues - Organizing shared expertise across multiple prompts
Location Convention
Skill files live in a skills/ folder next to the prompt file. List that folder (or individual files) in the prompt frontmatter. A sibling skills/ directory is not loaded unless the prompt names it:
src/workflows/{workflow-name}/
├── workflow.ts
├── steps.ts
├── types.ts
└── prompts/
├── writing_assistant@v1.prompt
└── skills/
├── clarity_guidelines.md
├── response_format.md
└── structure_guide.md
The skills/ folder is relative to the prompt file location, not the workflow root.
Skill File Format
Skill files are markdown documents with an optional YAML frontmatter block:
---
name: clarity_guidelines
description: Rules for writing clear, readable technical content
---
# Clarity Guidelines
When reviewing or writing technical content for clarity:
1. **Sentence length**: Keep sentences under 25 words when possible.
Break complex ideas into multiple sentences.
2. **Active voice**: Prefer active voice ("The function returns X")
over passive ("X is returned by the function").
3. **Jargon**: Define technical terms on first use.
Avoid unnecessary acronyms without explanation.
4. **Concrete examples**: Every abstract concept should have
a concrete example.
When applying this skill, flag any violations you find
and suggest improvements.
Frontmatter Fields
| Field | Required | Default | Description |
|---|---|---|---|
name | No | Filename without .md | Identifier the LLM uses with load_skill |
description | No | Same as name | Shown in system message, helps LLM decide when to load |
| Body | Yes | - | Full instructions returned when LLM calls load_skill |
If you omit the frontmatter entirely, the filename (without .md) is used as both the name and description. A file named clarity_guidelines.md with no frontmatter gets name: "clarity_guidelines" and description: "clarity_guidelines".
Write good descriptions. They appear in the system message and are what the LLM uses to decide whether to load a skill. "Rules for writing clear, readable technical content" is better than "clarity_guidelines".
How Skills Are Loaded
List skill paths in the prompt YAML frontmatter. Paths resolve relative to the prompt file and can be individual .md files or directories of .md files. (Model lines below are current as of 2026-05-04 - refresh via output-dev-model-selection.)
---
provider: anthropic
model: claude-sonnet-4-6
maxOutputTokens: 2048
skills:
- ./skills
- ../shared_skills/tone_guide.md
---
<system>
You are an expert technical writing assistant.
Use load_skill to get full instructions for any skill before applying it.
</system>
<user>
Review the following {{ content_type }} content focusing on {{ focus }}.
Content:
{{ content }}
</user>
At runtime, Output loads the listed paths and:
- Adds a summary of available skills to the system message
- Injects a
load_skilltool the LLM can call
List skill paths in the prompt frontmatter.
Omit skills (or set skills: []) when a prompt should load none. A sibling skills/ folder used by other prompts in the same directory is not inherited.
Complete Example
Skill File
---
name: response_format
description: Standard format requirements for all review responses
---
# Response Format
Every response MUST end with the exact string "OUTPUT_COMPLETE" on its own line.
Structure your review as follows:
1. **Summary**: 2-3 sentence overview of the content quality
2. **Issues**: Numbered list of specific problems found
3. **Suggestions**: Actionable improvements for each issue
4. **Score**: Overall quality score from 0-100
OUTPUT_COMPLETE
Prompt File Using Skills
---
provider: anthropic
# current as of 2026-05-04 - run output-dev-model-selection for the latest
model: claude-sonnet-4-6
maxOutputTokens: 2048
skills:
- ./skills
---
<system>
You are an expert technical writing assistant.
Use load_skill to get the full instructions for any skill before applying it.
After reviewing, provide structured feedback with specific issues and suggestions.
</system>
<user>
Review the following {{ content_type }} content focusing on {{ focus }}.
Content:
{{ content }}
</user>
Step Using the Prompt
import { step, z } from '@outputai/core';
import { Agent, aiSdk } from '@outputai/core/llm';
export const reviewContent = step( {
name: 'reviewContent',
description: 'Review content using skills for specialized expertise',
inputSchema: z.object( {
content: z.string(),
content_type: z.string(),
focus: z.string()
} ),
outputSchema: z.object( {
summary: z.string(),
issues: z.array( z.string() ),
suggestions: z.array( z.string() ),
score: z.number()
} ),
fn: async input => {
const agent = new Agent( {
prompt: 'writing_assistant@v1',
variables: input,
output: aiSdk.Output.object( {
schema: z.object( {
summary: z.string().describe( '2-3 sentence overview' ),
issues: z.array( z.string() ).describe( 'Specific problems found' ),
suggestions: z.array( z.string() ).describe( 'Actionable improvements' ),
score: z.number().describe( 'Quality score 0-100' )
} )
} )
} );
const { output } = await agent.generate();
return output;
}
} );
Best Practices
1. Write Focused Skills
Each skill should cover one area of expertise. Prefer multiple focused skills over one large skill:
skills/
├── clarity_guidelines.md # Writing clarity
├── structure_guide.md # Document structure
└── response_format.md # Output formatting
2. Write Descriptive Descriptions
The description appears in the system message. Make it clear when the LLM should load this skill:
---
name: clarity_guidelines
description: Rules for writing clear, readable technical content
---
Not:
---
name: clarity_guidelines
description: clarity_guidelines
---
3. Structure Instructions with Headers
Use markdown headers and lists for scannable instructions:
# Clarity Guidelines
## Rules
1. Keep sentences under 25 words
2. Prefer active voice
3. Define jargon on first use
## When to Flag
- Sentences over 30 words
- Passive voice in instructions
- Undefined acronyms
4. Include Actionable Guidance
Tell the LLM what to do with the skill, not just what the skill is about:
When applying this skill, flag any violations you find and suggest improvements.
Verification Checklist
- Prompt frontmatter lists skill paths (
skills: ./skillsor explicit files) - Each skill has a clear, descriptive
descriptionin frontmatter - Skill body contains actionable instructions
- Prompt file mentions
load_skillin the system message - Listed paths resolve relative to the prompt file
- Prompts that should load no skills omit
skills(or setskills: []) - Skills are focused (one area of expertise per file)
- Prompt frontmatter sets
maxStepswhen the tool-loop ceiling should not be 10
Related Skills
output-dev-prompt-file- Creating .prompt files that use skillsoutput-dev-agent-class- Using the Agent class with skillsoutput-dev-step-function- Using skills in step functionsoutput-dev-folder-structure- Understanding skill file locations
Creating Step Functions
Overview
This skill documents how to create step functions in steps.ts for Output SDK workflows. Steps are where all I/O operations happen - HTTP requests, LLM calls, database operations, file system access, etc.
When to Use This Skill
- Implementing I/O operations for a workflow
- Adding HTTP client integrations
- Implementing LLM-powered steps
- Handling errors with FatalError and ValidationError
- Creating reusable step components
File Organization
Option 1: Flat File (Default)
For smaller workflows, use a single steps.ts file:
src/workflows/{workflow-name}/
├── workflow.ts
├── steps.ts # All steps in one file
├── types.ts
└── ...
Option 2: Folder-Based (Large workflows)
For larger workflows with many steps, use a steps/ folder:
src/workflows/{workflow-name}/
├── workflow.ts
├── steps/ # Steps split into individual files
│ ├── fetch_data.ts
│ ├── process.ts
│ └── validate.ts
├── types.ts
└── ...
Component Location Rules
Important: step() calls MUST be in files containing 'steps' in the path:
src/workflows/my_workflow/steps.ts✓src/workflows/my_workflow/steps/fetch_data.ts✓src/shared/steps/common_steps.ts✓src/workflows/my_workflow/helpers.ts✗ (cannot contain step() calls)
Activity Isolation Constraints
Steps are Temporal activities with strict import rules to ensure deterministic replay.
Steps CAN import from:
- Local workflow files:
./utils.js,./types.js,./helpers.js - Local subdirectories:
./clients/pokeapi.js,./lib/helpers.js - Shared utilities:
../../shared/utils/*.js - Shared clients:
../../shared/clients/*.js - Shared services:
../../shared/services/*.js
Steps CANNOT import:
- Other step files (even shared steps - workflows import those)
- Evaluator files
- Workflow files
Example of WRONG imports:
// WRONG - steps cannot import other steps
import { otherStep } from '../../shared/steps/other.js'; // ✗
import { anotherStep } from './other_steps.js'; // ✗
Critical Import Patterns
Core Imports
// CORRECT - Import from @outputai/core
import { step, z, FatalError, ValidationError } from '@outputai/core';
// WRONG - Never import z from zod
import { z } from 'zod';
HTTP Client Import
// CORRECT - Use @outputai/core/http wrapper
import { createKyClient } from '@outputai/core/http';
// WRONG - Never use axios directly
import axios from 'axios';
Related Skill: output-error-http-client
LLM Client Import
// CORRECT - Use @outputai/core/llm wrapper
import { generateText, aiSdk } from '@outputai/core/llm';
// WRONG - Never call LLM providers directly
import OpenAI from 'openai';
ES Module Imports
All imports MUST use .js extension:
// CORRECT
import { InputSchema, OutputSchema } from './types.js';
import { GeminiService } from '../../shared/clients/gemini_client.js';
// WRONG - Missing .js extension
import { InputSchema, OutputSchema } from './types';
Basic Structure
import { step, z, FatalError, ValidationError } from '@outputai/core';
import { createKyClient } from '@outputai/core/http';
import { generateText, aiSdk } from '@outputai/core/llm';
import { StepInputSchema, StepOutputSchema } from './types.js';
export const myStep = step( {
name: 'myStep',
description: 'Description of what this step does',
inputSchema: StepInputSchema,
outputSchema: StepOutputSchema,
fn: async input => {
// Implementation with I/O operations
return { /* output matching outputSchema */ };
}
} );
Required Properties
name (string)
Unique identifier for the step. Use camelCase.
name: 'generateImageIdeas'
description (string)
Human-readable description of the step's purpose.
description: 'Generate creative infographic prompt ideas using Claude'
inputSchema (Zod schema)
Schema for validating step input. Define in types.ts and import.
inputSchema: z.object( {
content: z.string(),
numberOfIdeas: z.number()
} )
outputSchema (Zod schema)
Schema for validating step output. Define in types.ts and import.
outputSchema: z.array( z.string() )
fn (async function)
The step execution function. This is where I/O operations happen.
fn: async input => {
const result = await someExternalService( input );
return result;
}
HTTP Client Usage
Creating an HTTP Client
import { createKyClient } from '@outputai/core/http';
import { FatalError, ValidationError } from '@outputai/core';
const RETRY_STATUS_CODES = [ 408, 429, 500, 502, 503, 504 ];
const FATAL_STATUS_CODES = [ 401, 403, 404 ];
const client = createKyClient( {
timeout: 30000,
retry: {
limit: 3,
statusCodes: RETRY_STATUS_CODES
},
hooks: {
beforeError: [
( { error } ) => {
const status = error.response?.status;
const message = error.message;
if ( status && FATAL_STATUS_CODES.includes( status ) ) {
throw new FatalError(
`HTTP ${status} error: ${message}. This is a permanent error.`
);
}
throw new ValidationError(
`HTTP request failed: ${message}`
);
}
]
}
} );
Making HTTP Requests
// GET request
const response = await client.get( 'https://api.example.com/data' );
const data = await response.json();
// POST request with JSON body
const response = await client.post( 'https://api.example.com/submit', {
json: { field: 'value' }
} );
// HEAD request (check URL accessibility)
const response = await client.head( url );
const contentType = response.headers.get( 'content-type' );
When a non-HEAD request only uses response metadata, such as response.url, response.status, or headers, cancel the
unused body in a finally block. Responses read with .json(), .text(), etc. are already consumed.
const response = await client.get( url );
try {
return response.url;
} finally {
await response.body?.cancel();
}
Related Skill: output-dev-http-client-create for creating shared clients
LLM Operations
Important: Define LLM Schemas in types.ts
Schemas used in aiSdk.Output.object() must be defined in types.ts and imported -- never defined inline in step functions. Inline schemas lead to duplication, drift between the step's outputSchema and the LLM schema, and make it harder to maintain types.
// WRONG - inline schema in aiSdk.Output.object()
output: aiSdk.Output.object( {
schema: z.object( {
analysis: z.string()
} )
} )
// CORRECT - import from types.ts
import { AnalysisLlmSchema } from './types.js';
// ...
output: aiSdk.Output.object( {
schema: AnalysisLlmSchema
} )
Using generateText with aiSdk.Output.object()
The variables field accepts scalars, nested objects, and arrays. Use Liquid loops and dot notation in the prompt when it owns presentation; pre-format in the step when the exact rendered text is application logic.
generateText arguments: prompt, promptDir, variables, tools, output, toolChoice, stopWhen, abortSignal.
import { generateText, aiSdk } from '@outputai/core/llm';
import {
AnalyzeContentInputSchema,
AnalyzeContentOutputSchema,
AnalysisLlmSchema
} from './types.js';
export const analyzeContent = step( {
name: 'analyzeContent',
description: 'Analyze content using Claude',
inputSchema: AnalyzeContentInputSchema,
outputSchema: AnalyzeContentOutputSchema,
fn: async ( { content } ) => {
const { output } = await generateText( {
prompt: 'analyzeContent@v1',
variables: {
content
},
output: aiSdk.Output.object( {
schema: AnalysisLlmSchema
} )
} );
return { analysis: output.analysis };
}
} );
Using generateText
import { generateText } from '@outputai/core/llm';
import { SummarizeInputSchema, SummarizeOutputSchema } from './types.js';
export const generateSummary = step( {
name: 'generateSummary',
description: 'Generate a text summary',
inputSchema: SummarizeInputSchema,
outputSchema: SummarizeOutputSchema,
fn: async ( { content } ) => {
const { result } = await generateText( {
prompt: 'summarize@v1',
variables: { content }
} );
return { summary: result };
}
} );
Related Skill: output-dev-prompt-file for creating prompt files
Streaming LLM Progress
Prefer generateTextWithStreaming() in steps when the caller needs progress callbacks and a complete result:
import { generateTextWithStreaming } from '@outputai/core/llm';
const reportProgress = ( chunk: string ) => process.stdout.write( chunk );
const result = await generateTextWithStreaming( {
prompt: 'summarize@v1',
variables: { content },
onChunk( { chunk } ) {
if ( chunk.type === 'text-delta' ) {
reportProgress( chunk.text );
}
}
} );
return result.result;
The returned promise rejects on stream failures, allowing Temporal to retry the activity. Use streamText() only when direct stream access is required; capture onError and throw the captured error after consumption. See output-dev-llm-streaming for complete patterns.
Error Handling
FatalError (Non-Retryable)
Use FatalError for permanent failures that should not be retried:
import { FatalError } from '@outputai/core';
import { credentials } from '@outputai/core/credentials';
// Authentication failures
if ( response.status === 401 ) {
throw new FatalError( 'Invalid API key' );
}
// Invalid input that cannot be fixed by retry
if ( !input.requiredField ) {
throw new FatalError( 'Missing required field: requiredField' );
}
// Resource not found
if ( response.status === 404 ) {
throw new FatalError( `Resource not found: ${resourceId}` );
}
// Configuration errors
if ( !credentials.get( 'service.api_key' ) ) {
throw new FatalError( 'service.api_key credential not set' );
}
ValidationError (Retryable)
Use ValidationError for temporary failures that may succeed on retry:
import { ValidationError } from '@outputai/core';
// Rate limiting
if ( response.status === 429 ) {
throw new ValidationError( 'Rate limit exceeded, will retry' );
}
// Temporary service unavailability
if ( response.status === 503 ) {
throw new ValidationError( 'Service temporarily unavailable' );
}
// Network errors
try {
const response = await client.get( url );
} catch ( error ) {
throw new ValidationError( `Network error: ${error.message}` );
}
// Empty response that might be temporary
if ( results.length === 0 ) {
throw new ValidationError( 'No results returned, will retry' );
}
Related Skill: output-error-try-catch for proper error handling patterns
Complete Example
Based on a real workflow step:
import { step, z, FatalError, ValidationError } from '@outputai/core';
import { createKyClient } from '@outputai/core/http';
import { generateText, aiSdk } from '@outputai/core/llm';
import { GeminiImageService } from '../../shared/clients/gemini_client.js';
import {
GenerateImageIdeasInputSchema,
GenerateImagesInputSchema,
ImageIdeasSchema
} from './types.js';
const RETRY_STATUS_CODES = [ 408, 429, 500, 502, 503, 504 ];
const FATAL_STATUS_CODES = [ 401, 403, 404 ];
const client = createKyClient( {
timeout: 30000,
retry: {
limit: 3,
statusCodes: RETRY_STATUS_CODES
},
hooks: {
beforeError: [
( { error } ) => {
const status = error.response?.status;
const message = error.message;
if ( status && FATAL_STATUS_CODES.includes( status ) ) {
throw new FatalError( `HTTP ${status} error: ${message}` );
}
throw new ValidationError( `HTTP request failed: ${message}` );
}
]
}
} );
// Step 1: Generate Ideas using LLM
export const generateImageIdeas = step( {
name: 'generateImageIdeas',
description: 'Generate creative infographic prompt ideas using Claude',
inputSchema: GenerateImageIdeasInputSchema,
outputSchema: z.array( z.string() ),
fn: async ( { content, numberOfIdeas, colorPalette, artDirection } ) => {
const { output } = await generateText( {
prompt: 'generateImageIdeas@v1',
variables: {
content,
numberOfIdeas,
colorPalette: colorPalette || '',
artDirection: artDirection || ''
},
output: aiSdk.Output.object( {
schema: ImageIdeasSchema
} )
} );
return output.ideas;
}
} );
// Step 2: Generate Images using external API
export const generateImages = step( {
name: 'generateImages',
description: 'Generate images using Gemini API',
inputSchema: GenerateImagesInputSchema,
outputSchema: z.array( z.string() ),
fn: async ( { input, prompt } ) => {
const geminiImageService = new GeminiImageService();
const generatedImages = await geminiImageService.generateImage( {
prompt,
aspectRatio: input.aspectRatio,
resolution: input.resolution,
numberOfImages: input.numberOfGenerations
} );
if ( generatedImages.length === 0 ) {
throw new ValidationError( 'No images were generated by Gemini' );
}
return generatedImages;
}
} );
// Step 3: Validate URLs using HTTP client
export const validateReferenceImages = step( {
name: 'validateReferenceImages',
description: 'Validates that all provided reference image URLs are accessible',
inputSchema: z.object( {
referenceImageUrls: z.array( z.string() ).optional()
} ),
outputSchema: z.boolean(),
fn: async ( { referenceImageUrls } ) => {
if ( !referenceImageUrls || referenceImageUrls.length === 0 ) {
return true;
}
for ( const [ index, url ] of referenceImageUrls.entries() ) {
const response = await client.head( url );
const contentType = response.headers.get( 'content-type' );
if ( contentType && !contentType.startsWith( 'image/' ) ) {
throw new FatalError(
`Reference URL ${index + 1} (${url}) is not an image file`
);
}
}
return true;
}
} );
Best Practices
1. One Responsibility Per Step
// Good - focused step
export const fetchUserData = step( {
name: 'fetchUserData',
description: 'Fetch user data from the API'
// ...
} );
// Avoid - step doing too much
export const fetchAndProcessAndSaveUserData = step( {
name: 'fetchAndProcessAndSaveUserData'
// ...
} );
2. Clear Error Messages
// Good - specific error message
throw new FatalError( `Invalid API key for service: ${serviceName}` );
// Avoid - generic error message
throw new FatalError( 'Error occurred' );
3. Validate Input Early
fn: async input => {
if ( !input.url.startsWith( 'https://' ) ) {
throw new FatalError( 'URL must use HTTPS protocol' );
}
const response = await client.get( input.url );
// ...
}
Verification Checklist
-
step,z,FatalError,ValidationErrorimported from@outputai/core -
createKyClientimported from@outputai/core/http(not axios) -
generateTextandaiSdkimported from@outputai/core/llm(not direct provider) - Structured output uses
aiSdk.Output.object()with.describe()(not.min()/.max()/.length()) on number and array schemas - Schemas for
aiSdk.Output.object()are defined intypes.tsand imported, not inline - All imports use
.jsextension - Named exports used for each step
- Each step has
name,description,inputSchema,outputSchema,fn - FatalError used for non-retryable failures
- ValidationError used for retryable failures
- Non-HEAD HTTP responses are consumed or cancelled when only metadata is used
- No bare try-catch blocks that swallow errors
- Steps only import allowed dependencies (local files, shared code)
- No imports of other steps, evaluators, or workflows
- Code follows style conventions (see
output-dev-code-style)
Related Skills
output-dev-workflow-function- Orchestrating steps in workflow.tsoutput-dev-evaluator-function- Using steps in evaluator functionsoutput-dev-types-file- Defining step input/output schemasoutput-dev-code-style- Code formatting and style conventionsoutput-dev-http-client-create- Creating shared HTTP clientsoutput-dev-llm-streaming- Streaming LLM progress with Temporal-safe failuresoutput-dev-prompt-file- Creating prompt files for LLM operationsoutput-error-try-catch- Proper error handling patternsoutput-error-direct-io- Avoiding direct I/O in workflows
Creating types.ts Files with Zod Schemas
Overview
This skill documents how to create types.ts files for Output SDK workflows. These files contain Zod schemas for input/output validation and their corresponding TypeScript types.
When to Use This Skill
- Creating a new workflow's type definitions
- Adding new schemas for steps
- Fixing schema validation errors
- Refactoring existing type definitions
Critical Import Rule
ALWAYS import z from @outputai/core, NEVER from zod directly:
// CORRECT
import { z } from '@outputai/core';
// WRONG - will cause runtime errors
import { z } from 'zod';
Related Skill: output-error-zod-import for troubleshooting import issues
Basic Structure
import { z } from '@outputai/core';
// 1. Workflow Input Schema
export const WorkflowInputSchema = z.object( {
// Define input fields
} );
// 2. Workflow Output Type
export type WorkflowInput = z.infer<typeof WorkflowInputSchema>;
export type WorkflowOutput = /* output type */;
// 3. Step Schemas (for each step)
export const StepNameInputSchema = z.object( {
// Step input fields
} );
export const StepNameOutputSchema = z.object( {
// Step output fields
} );
// 4. Type Exports
export type StepNameInput = z.infer<typeof StepNameInputSchema>;
export type StepNameOutput = z.infer<typeof StepNameOutputSchema>;
CRITICAL: Schema Constraints for LLM Output
Schemas passed to aiSdk.Output.object() are sent to LLM providers as tool definitions. Anthropic rejects several JSON Schema constraints that Zod methods produce. Getting this wrong causes runtime errors.
What Is NOT Allowed in LLM Output Schemas
- Numbers:
.min(),.max()onz.number()produceminimum/maximum-- rejected by Anthropic. - Arrays:
.min(),.max(),.length()onz.array()produceminItems/maxItems-- Anthropic only supportsminItemsof0or1. Values like.length( 3 )or.min( 2 )will be rejected.
Use .describe() Instead
.describe() is the primary mechanism for guiding LLM output quality. LLM providers use field names and descriptions from the schema to understand what each field should contain. Write clear, specific descriptions that communicate your intent.
Important: .describe() replaces both unsupported constraints AND prompt-based format instructions. Do not also describe the schema in the prompt -- the schema is sent to the provider automatically, and duplicating it reduces performance and creates drift risk. See output-dev-prompt-file for details.
// LLM output schema (sent to provider via aiSdk.Output.object()) -- .describe() ONLY
const llmOutputSchema = z.object( {
score: z.number().describe( 'Quality score 0-100' ),
confidence: z.number().describe( 'Confidence 0-1' ),
predictions: z.array( predictionSchema ).describe( 'Exactly 3 predictions' )
} );
// Workflow/step validation schema (Zod-only, NOT sent to LLM) -- .min()/.max()/.length() OK
const workflowOutputSchema = z.object( {
score: z.number().min( 0 ).max( 100 ).describe( 'Quality score 0-100' ),
confidence: z.number().min( 0 ).max( 1 ).describe( 'Confidence 0-1' ),
predictions: z.array( predictionSchema ).length( 3 ).describe( 'Exactly 3 predictions' )
} );
When to Use Which
| Context | .min()/.max()/.length() | .describe() |
|---|---|---|
Schema passed to aiSdk.Output.object() | No (numbers or arrays) | Yes |
inputSchema / outputSchema on steps | OK | Optional |
inputSchema / outputSchema on workflows | OK | Optional |
outputSchema on evaluators | OK | Optional |
LLM Schemas Must Live in types.ts
Define all schemas used in aiSdk.Output.object() in types.ts and import them in step functions. Never define them inline -- this causes duplication and makes it harder to verify they follow the constraints above.
Common Schema Patterns
Basic Types
import { z } from '@outputai/core';
// Strings
const stringField = z.string();
const optionalString = z.string().optional();
const stringWithDefault = z.string().default( 'default value' );
const describedString = z.string().describe( 'Field description' );
// Numbers
const numberField = z.number();
const integerField = z.number().int();
const rangedNumber = z.number().min( 1 ).max( 100 ); // runtime only — NOT safe for aiSdk.Output.object() schemas
// Booleans
const booleanField = z.boolean();
const defaultBoolean = z.boolean().default( false );
// Enums
const enumField = z.enum( [ 'option1', 'option2', 'option3' ] );
const enumWithDefault = z.enum( [ 'small', 'medium', 'large' ] ).default( 'medium' );
Complex Types
import { z } from '@outputai/core';
// Arrays
const stringArray = z.array( z.string() );
const objectArray = z.array( z.object( { id: z.string(), name: z.string() } ) );
// Objects
const nestedObject = z.object( {
user: z.object( {
id: z.string(),
email: z.string().email()
} ),
settings: z.object( {
notifications: z.boolean()
} )
} );
// Union Types
const flexibleInput = z.union( [
z.string(),
z.array( z.string() )
] );
// Records
const keyValueMap = z.record( z.string(), z.number() );
Validation Patterns
import { z } from '@outputai/core';
// String Validations
const emailField = z.string().email();
const urlField = z.string().url();
const uuidField = z.string().uuid();
const minLengthString = z.string().min( 1 );
const maxLengthString = z.string().max( 1000 );
// Number Validations
const positiveNumber = z.number().positive();
const nonNegativeNumber = z.number().nonnegative();
const percentageNumber = z.number().min( 0 ).max( 100 );
// Array Validations (runtime only — NOT safe for aiSdk.Output.object() schemas)
const nonEmptyArray = z.array( z.string() ).min( 1 );
const limitedArray = z.array( z.string() ).max( 10 );
const fixedLengthArray = z.array( z.string() ).length( 3 );
Complete Example
Based on a real workflow (image_infographic_nano):
import { z } from '@outputai/core';
// ============================================
// Workflow Schemas
// ============================================
export const WorkflowInputSchema = z.object( {
content: z.string().describe( 'Text content to generate image ideas from' ),
mode: z.enum( [ 'infographic' ] ).default( 'infographic' ).describe( 'Type of image to generate' ),
colorPalette: z.string().optional().describe( 'Color palette preference for the images' ),
artDirection: z.string().optional().describe( 'Art direction or style preference' ),
numberOfIdeas: z.number().min( 1 ).max( 10 ).default( 1 ).describe( 'Number of image concepts to generate' ),
referenceImageUrls: z.union( [
z.string(),
z.array( z.string() )
] ).optional().describe( 'Reference image URLs for style guidance (max 14)' ),
aspectRatio: z.enum( [ '1:1', '16:9', '9:16', '4:3', '3:4' ] ).default( '1:1' ).describe( 'Aspect ratio for generated images' ),
resolution: z.enum( [ '1K', '2K', '4K' ] ).default( '1K' ).describe( 'Resolution for generated images' ),
numberOfGenerations: z.number().min( 1 ).max( 10 ).default( 1 ).describe( 'Number of images to generate per concept' ),
storageNamespace: z.string().optional().describe( 'S3 folder path for storing images' )
} );
export type WorkflowInput = z.infer<typeof WorkflowInputSchema>;
export type WorkflowOutput = string[];
// ============================================
// Step Schemas
// ============================================
export const ValidateReferenceImagesInputSchema = z.object( {
referenceImageUrls: z.array( z.string() ).optional()
} );
export const GenerateImageIdeasInputSchema = z.object( {
content: z.string(),
numberOfIdeas: z.number(),
colorPalette: z.string().optional(),
artDirection: z.string().optional()
} );
export const GenerateImagesInputSchema = z.object( {
input: z.object( {
referenceImageUrls: z.union( [ z.string(), z.array( z.string() ) ] ).optional(),
aspectRatio: z.enum( [ '1:1', '16:9', '9:16', '4:3', '3:4' ] ),
resolution: z.enum( [ '1K', '2K', '4K' ] ),
numberOfGenerations: z.number(),
storageNamespace: z.string().optional()
} ),
prompt: z.string()
} );
// Schema for LLM response validation
export const ImageIdeasSchema = z.object( {
ideas: z.array( z.string() ).describe( 'Array of detailed image prompts for Gemini' )
} );
// ============================================
// Type Exports
// ============================================
export type ValidateReferenceImagesInput = z.infer<typeof ValidateReferenceImagesInputSchema>;
export type GenerateImageIdeasInput = z.infer<typeof GenerateImageIdeasInputSchema>;
export type GenerateImagesInput = z.infer<typeof GenerateImagesInputSchema>;
export type ImageIdeas = z.infer<typeof ImageIdeasSchema>;
Best Practices
1. Use Descriptive Field Descriptions
// Good - helps with documentation and error messages
z.string().describe( 'User email address for notifications' )
// Avoid - no context for errors
z.string()
2. Provide Sensible Defaults
// Good - workflow works without optional fields
numberOfIdeas: z.number().min( 1 ).max( 10 ).default( 1 )
// Avoid - forces users to provide every field
numberOfIdeas: z.number().min( 1 ).max( 10 )
3. Separate Workflow and Step Schemas
// Workflow input schema (what the user provides)
export const WorkflowInputSchema = z.object( { ... } );
// Step schemas (internal data shapes)
export const StepNameInputSchema = z.object( { ... } );
4. Export Both Schemas and Types
// Export schema for runtime validation
export const UserSchema = z.object( { ... } );
// Export type for TypeScript type checking
export type User = z.infer<typeof UserSchema>;
Verification Checklist
-
zis imported from@outputai/core - WorkflowInputSchema is defined and exported
- WorkflowInput type is exported
- WorkflowOutput type is defined
- Each step has corresponding input/output schemas
- All schemas have
.describe()for important fields - Optional fields use
.optional()or.default() - Numeric fields have appropriate constraints (
.min()/.max()for runtime schemas,.describe()foraiSdk.Output.object()schemas) - Code follows style conventions (see output-dev-code-style)
Related Skills
output-dev-workflow-function- Using schemas in workflow definitionsoutput-dev-step-function- Using schemas in step definitionsoutput-dev-evaluator-function- Using schemas in evaluator definitionsoutput-dev-folder-structure- Where types.ts belongs in the projectoutput-error-zod-import- Troubleshooting schema import issuesoutput-dev-code-style- Code style conventions
Upgrade Prompt Models In-Place
Walks every .prompt file in a project (or scoped subtree), classifies each model into its provider+family bucket, looks up the latest stable model in that bucket via the output-dev-model-selection snapshot, and rewrites the model: line. Provider and family tier are preserved — a Haiku stays a Haiku, an Anthropic stays an Anthropic.
This skill explicitly does not swap providers or escalate tiers (eg Haiku → Sonnet). Those are deliberate human decisions handled separately.
When to invoke
- A periodic refresh: "upgrade all my prompt models to the latest"
- After a long break between framework updates, where dated snapshot IDs (eg
claude-sonnet-4-20250514) have aged out - Right after creating a new project from the CLI scaffolder, to pull every templated default forward to the current best
Workflow
Step 1 — Discover
Find every .prompt file under the target scope. Default scope is the project's src/ tree; the user may scope to a single workflow.
Step 2 — Parse current model
For each file, read the YAML frontmatter (between the first pair of --- lines) and pull out provider: and model:.
Step 3 — Classify family
Match the existing model into a family bucket. Family is preserved across the upgrade.
| Pattern | Family |
|---|---|
claude-opus-* | anthropic-opus |
claude-sonnet-* | anthropic-sonnet |
claude-haiku-* | anthropic-haiku |
gpt-*-pro | openai-pro |
gpt-*-mini | openai-mini |
gpt-*-nano | openai-nano |
gpt-N.M (no suffix) | openai-default |
gemini-*-flash-lite* | google-flash-lite |
gemini-*-flash* | google-flash |
gemini-*-pro* | google-pro |
If a model doesn't match any pattern, skip the file and log a warning. Do not guess.
Step 4 — Look up latest in family
For each prompt, find the latest stable model in the same family by following output-dev-model-selection — fetch its snapshot, apply its filter rules (skip preview/alpha/beta, prefer unversioned aliases), and translate the chosen id to prompt-file form.
Use this family → snapshot-key + id regex map to pin the lookup to the existing tier:
| Family | Snapshot key | id regex |
|---|---|---|
anthropic-opus | anthropic | claude-opus- |
anthropic-sonnet | anthropic | claude-sonnet- |
anthropic-haiku | anthropic | claude-haiku- |
openai-pro | openai | -pro$ |
openai-default | openai | ^openai/gpt-[0-9.]+$ |
openai-mini | openai | -mini$ |
openai-nano | openai | -nano$ |
google-pro | google | -pro (excluding -flash) |
google-flash | google | -flash$|-flash-[0-9] |
google-flash-lite | google | -flash-lite |
If no stable match exists for a family (only pre-release entries available), surface that to the user and skip the file rather than guessing or downgrading to a different family.
Bail loudly on a failed snapshot. If the snapshot fetch itself returned nothing — network down, gateway shape changed, curl or jq missing — abort the run before Step 5. Do not continue with an empty snapshot and report "no upgrades needed", because that lies: the prompts weren't actually checked. Tell the user the snapshot fetch failed, point them at the manual-fallback steps in output-dev-model-selection, and exit.
Step 5 — Diff & confirm
Build a per-file report comparing the current model to the resolved latest:
src/workflows/foo/prompts/bar@v1.prompt claude-sonnet-4-20250514 → claude-sonnet-4-6
src/workflows/foo/prompts/baz@v1.prompt claude-haiku-4-5 ✓ already latest
Print the full report. Wait for explicit user confirmation before writing. In CI / non-interactive contexts, default to dry-run.
Step 6 — Edit
For each confirmed file, edit only the YAML frontmatter:
- Replace the
model:line with the resolved latest ID. - If a
# current as of YYYY-MM-DD …comment is present (the convention used inoutput-dev-prompt-fileexamples and CLI scaffolds), update its date to today's (date +%Y-%m-%d). Match the comment by the literalcurrent as ofprefix and only rewrite the date — leave the trailing text intact. - Leave
provider:,temperature:,maxOutputTokens:(or deprecatedmaxTokens:),providerOptions:, and the message body untouched.
Refreshing the dated comment in the same edit keeps the "as of" convention coherent — without it, an upgraded prompt would have a fresh model paired with a stale date.
Step 7 — Verify
After the batch:
- Spot-check a handful of files to confirm the YAML still has a frontmatter delimiter and message body.
- Rebuild the worker so the new model strings take effect (
npm run output:worker:build).
The Output SDK doesn't validate prompt model IDs at build time — invalid IDs only surface at first run. If smoke-tests are available, run at least one workflow per upgraded family.
Caveats
- Within-family only. This skill never upgrades Sonnet → Opus, never swaps Anthropic for OpenAI. To change tier or provider, edit prompts manually or use
output-dev-prompt-file. - Dated snapshots get bumped. A pin like
claude-sonnet-4-20250514becomes the unversioned aliasclaude-sonnet-4-6. If the pin was load-bearing for reproducibility, surface that and skip the file. @vertexandbedrocknamespace suffixes. Models likeclaude-sonnet-4-20250514@vertexoranthropic.claude-sonnet-4-20250514-v1:0need manual upgrade. The AI Gateway listing covers direct provider IDs only.- Models.dev pricing lag. Even after the upgrade,
response.costmay be incomplete withtotal: nullfor the brand-new model until models.dev catches up. The runtime call still works.
See also
output-dev-model-selection— canonical snapshot + selection rules this skill consumesoutput-dev-prompt-file—.promptfile structure
Workflow Run Cost
Overview
This skill helps you calculate the cost of a completed workflow execution using the Output CLI. It breaks down costs by LLM model (token usage) and external API service calls.
When to Use This Skill
- Checking how much a workflow run cost in LLM tokens
- Analyzing API service spend per workflow execution
- Comparing costs between workflow runs
- Presenting cost data during a demo or review
Step 1: Get a Workflow Run ID
The workflow cost command requires a workflow run ID — not the workflow name.
A run ID looks like: process_transcripts_2026-03-23T19:35:17.000Z_c2biRk_F9rYktF-wagBf5
If you already have a run ID, skip to Step 2.
If you don't have a run ID:
Option A — List recent runs (use skill: output-workflow-runs-list):
npx output workflow runs list <workflowName> --limit 5
Copy the run ID from the most recent completed run.
Option B — Run the workflow now and capture the ID:
If you know the input, run it synchronously and the run ID will be in the output:
npx output workflow run <workflowName> --input '{"key": "value"}'
Or asynchronously (use skill: output-workflow-start):
npx output workflow start <workflowName> --input '{"key": "value"}'
# Returns: Workflow ID: <runId>
If you don't know the input, check for a scenario file (use skill: output-dev-scenario-file) or the workflow's inputSchema in its types.ts.
Note: The workflow name (e.g.
process_transcripts) identifies the workflow type. The workflow run ID (e.g.process_transcripts_2026-03-23T19:35:17.000Z_abc123) identifies a specific execution.workflow costrequires the run ID.
Step 2: Calculate the Cost
Basic usage (text output):
npx output workflow cost <runId>
With a local trace file:
npx output workflow cost <runId> path/to/trace.json
JSON output (for programmatic use or display):
npx output workflow cost <runId> --json
Verbose — show per-call breakdown:
npx output workflow cost <runId> --verbose
All flags:
| Flag | Description | Default |
|---|---|---|
--json | Output machine-readable JSON instead of the text report | false |
--verbose | Show detailed per-LLM-call breakdown | false |
Understanding the Output
Text format example:
Pricing values are illustrative — the model and per-token rates shown below were current as of 2026-05-04. Live pricing comes from models.dev. For current model IDs, see
output-dev-model-selection.
Costs come in two figures per row: Original is the as-charged cost recorded
in the trace attributes (llm:generation:cost / http:request:cost, with
llm:generation:usage and legacy llm:usage fallbacks), and Adjusted is
the cost after applying any costs.yml override (equal to Original when no
override applies). The bottom line and JSON totalCost are the Adjusted total.
Workflow: process_transcripts
Duration: 12.3s
LLM Costs:
Model | Calls | Original | Adjusted
claude-sonnet-4-6 | 3 | $0.0123 | $0.0123
Subtotal | 3 | $0.0123 | $0.0123
API Costs:
Host | Calls | Original | Adjusted
r.jina.ai | 2 | $0.0040 | $0.0040
Subtotal | 2 | $0.0040 | $0.0040
TOTAL ESTIMATED COST (adjusted) $0.0163
As-charged (from trace) $0.0163
API Costs only include calls where the client attached a dollar amount with addRequestCost (see output-dev-http-client-create, "Attaching Request Cost (Paid APIs)"). An HTTP call that never calls addRequestCost is still traced but contributes nothing here — that's expected for free or internal services, and a gap to fix for paid ones.
JSON format fields:
{
"workflowName": "process_transcripts",
"durationMs": 12300,
"llmOriginalCost": 0.0123,
"llmAdjustedCost": 0.0123,
"totalInputTokens": 1234,
"totalOutputTokens": 567,
"httpCosts": [{ "host": "r.jina.ai", "originalTotalCost": 0.004, "adjustedTotalCost": 0.004 }],
"httpOriginalCost": 0.004,
"httpAdjustedCost": 0.004,
"originalTotalCost": 0.0163,
"totalCost": 0.0163
}
Per-call entries in llmCalls and httpCosts[].calls carry originalCost and
adjustedCost. (Older releases exposed llmTotalCost, services[],
serviceTotalCost, and unknownModels — replaced by the fields above.)
Examples
Get cost of the last run:
# First get the run ID
npx output workflow runs list process_transcripts --limit 1 --json
# Then get the cost
npx output workflow cost process_transcripts_2026-03-23T19:35:17.000Z_abc123
Get cost as JSON and extract total:
npx output workflow cost <runId> --json | jq '.totalCost'
Show full per-call breakdown:
npx output workflow cost <runId> --verbose
Related Skills
output-workflow-runs-list— Find run IDs from execution historyoutput-workflow-start— Start a workflow and capture its run IDoutput-workflow-run— Run a workflow synchronouslyoutput-dev-scenario-file— Create or find scenario files for workflow input
Creating workflow.ts Files
Overview
This skill documents how to create workflow.ts files for Output SDK workflows. The workflow file contains the main orchestration logic that coordinates step execution.
When to Use This Skill
- Creating a new workflow's main definition
- Understanding workflow structure requirements
- Debugging workflow orchestration issues
- Refactoring existing workflow logic
Critical Rules
1. Import Pattern
// CORRECT - Import from @outputai/core
import { workflow, z } from '@outputai/core';
// WRONG - Never import z from zod
import { z } from 'zod';
2. ES Module Imports
All imports MUST use .js extension:
// CORRECT
import { stepName } from './steps.js';
import { WorkflowInputSchema } from './types.js';
// WRONG - Missing .js extension
import { stepName } from './steps';
import { WorkflowInputSchema } from './types';
3. Determinism Requirement
CRITICAL: The workflow fn must be deterministic. No direct I/O operations are allowed in the workflow function.
// WRONG - Direct I/O in workflow
export default workflow( {
// ...
fn: async input => {
const response = await fetch( 'https://api.example.com' ); // NEVER do this!
return response.json();
}
} );
// CORRECT - Delegate I/O to steps
export default workflow( {
// ...
fn: async input => {
const result = await fetchDataStep( input ); // Steps handle I/O
return result;
}
} );
Related Skill: output-error-nondeterminism
Basic Structure
import { workflow, z } from '@outputai/core';
import { stepOne, stepTwo } from './steps.js';
import { WorkflowInputSchema, WorkflowOutput } from './types.js';
export default workflow( {
name: 'workflowName',
description: 'Brief description of what the workflow does',
inputSchema: WorkflowInputSchema,
outputSchema: z.object( { /* output shape */ } ),
fn: async ( input ): Promise<WorkflowOutput> => {
// Orchestrate step calls
const result = await stepOne( input );
const final = await stepTwo( result );
return final;
}
} );
Required Properties
name (string)
Unique identifier for the workflow. Use camelCase.
name: 'contentUtilsImageInfographicNano'
description (string)
Human-readable description of the workflow's purpose.
description: 'Generate high-quality infographic images using AI-powered ideation'
inputSchema (Zod schema)
Schema for validating workflow input. Import from types.ts.
inputSchema: WorkflowInputSchema
Related Skill: output-dev-types-file
outputSchema (Zod schema)
Schema for validating workflow output.
outputSchema: z.object( {
results: z.array( z.string() ),
metadata: z.object( {
processedAt: z.string()
} )
} )
fn (async function)
The workflow execution function. Must be deterministic.
fn: async ( input ): Promise<WorkflowOutput> => {
// Step orchestration only - no direct I/O
const result = await processStep( input );
return result;
}
Complete Example
Based on a real workflow (image_infographic_nano):
import { workflow, z } from '@outputai/core';
import {
generateImageIdeas,
generateImages,
validateReferenceImages
} from './steps.js';
import {
WorkflowInput,
WorkflowInputSchema,
WorkflowOutput
} from './types.js';
import { normalizeReferenceImageUrls } from './utils.js';
export default workflow( {
name: 'contentUtilsImageInfographicNano',
description: 'Generate high-quality infographic images using Google Gemini 3 Pro Image model with AI-powered ideation',
inputSchema: WorkflowInputSchema,
outputSchema: z.array( z.string() ),
fn: async ( rawInput: WorkflowInput ): Promise<WorkflowOutput> => {
// Pre-process input (pure function - OK in workflow)
const input = {
...rawInput,
referenceImageUrls: normalizeReferenceImageUrls( rawInput.referenceImageUrls )
};
// Conditional step execution
if ( input.referenceImageUrls && input.referenceImageUrls.length > 0 ) {
await validateReferenceImages( {
referenceImageUrls: input.referenceImageUrls as string[]
} );
}
// Sequential step execution
const ideas = await generateImageIdeas( {
content: input.content,
numberOfIdeas: input.numberOfIdeas,
colorPalette: input.colorPalette,
artDirection: input.artDirection
} );
// Parallel step execution
const generations = await Promise.all(
ideas.map( idea =>
generateImages( {
input: {
referenceImageUrls: input.referenceImageUrls,
aspectRatio: input.aspectRatio,
resolution: input.resolution,
numberOfGenerations: input.numberOfGenerations,
storageNamespace: input.storageNamespace
},
prompt: idea
} )
)
);
return generations.flat();
}
} );
Orchestration Patterns
Sequential Execution
Execute steps one after another:
fn: async input => {
const step1Result = await stepOne( input );
const step2Result = await stepTwo( step1Result );
const step3Result = await stepThree( step2Result );
return step3Result;
}
Parallel Execution
Execute independent steps concurrently:
fn: async input => {
const [ resultA, resultB, resultC ] = await Promise.all( [
stepA( input ),
stepB( input ),
stepC( input )
] );
return { resultA, resultB, resultC };
}
Conditional Execution
Execute steps based on conditions:
fn: async input => {
if ( input.includeImages ) {
await processImages( input );
}
const result = input.mode === 'fast' ?
await quickProcess( input ) :
await detailedProcess( input );
return result;
}
Fan-Out Pattern
Process multiple items in parallel:
fn: async input => {
const results = await Promise.all(
input.items.map( item => processItem( { item } ) )
);
return { processedItems: results };
}
Pipeline Pattern
Chain multiple transformations:
fn: async input => {
const extracted = await extractData( input );
const transformed = await transformData( extracted );
const validated = await validateData( transformed );
const enriched = await enrichData( validated );
return enriched;
}
Child Workflows
Invoke another workflow by calling it like a function. It runs as a child workflow and inherits the parent's trace, activity options and search attributes:
import enrichCompanyWorkflow from '../enrich_company/workflow.js';
fn: async input => {
const company = await enrichCompanyWorkflow( { domain: input.domain } );
return company;
}
Pass activity options as the second argument: await enrichCompanyWorkflow( input, { activityOptions: { retry: { maximumAttempts: 1 } } } ). The options object also accepts detached and context; see https://docs.output.ai/workflows/child-workflows.
What is Allowed in Workflow fn
Allowed (Deterministic Operations)
- Calling step functions
- Calling other workflows as functions (child workflows)
- Pure data transformations
- Conditional logic based on input
- Array operations (map, filter, reduce)
- Object destructuring and construction
- Promise.all for parallel steps
- Control flow (if/else, loops)
NOT Allowed (Non-Deterministic Operations)
- HTTP requests (use steps)
- Database queries (use steps)
- File system operations (use steps)
- Random number generation
- Current date/time access
- Environment variable access
- Any external service calls
Verification Checklist
-
workflowandzimported from@outputai/core - All imports use
.jsextension - Default export used for the workflow
-
nameis camelCase and unique -
descriptionclearly explains the workflow -
inputSchemaimported fromtypes.ts -
outputSchemamatches actual return type -
fnis deterministic (no direct I/O) - All I/O delegated to step functions
- Code follows style conventions (see output-dev-code-style)
Related Skills
output-dev-step-function- Creating step functions that handle I/Ooutput-dev-evaluator-function- Using steps in evaluator functionsoutput-dev-types-file- Defining input/output schemasoutput-dev-folder-structure- Where workflow.ts belongsoutput-error-nondeterminism- Fixing determinism violationsoutput-error-zod-import- Fixing schema import issuesoutput-dev-code-style
Fix Direct I/O in Workflow Functions
Overview
This skill helps diagnose and fix a critical error pattern where I/O operations (HTTP calls, database queries, file operations) are performed directly in workflow functions instead of in steps. This violates Temporal's determinism requirements.
When to Use This Skill
You're seeing:
- Workflow hangs indefinitely
- Undefined or empty responses
- "workflow must be deterministic" errors
- Network operations failing silently
- Timeouts without clear cause
Root Cause
Workflow functions must be deterministic - they should only orchestrate steps, not perform I/O directly. When you make HTTP calls, database queries, or any external operations directly in a workflow function:
- Hangs: The workflow may hang because I/O isn't properly handled
- Determinism violations: Temporal replays workflows, and I/O results differ
- No retry logic: Direct calls bypass Output SDK's retry mechanisms
- No tracing: Operations aren't recorded in the workflow trace
Symptoms
Direct fetch/axios in Workflow
// WRONG: I/O directly in workflow
export default workflow( {
fn: async input => {
const response = await fetch( 'https://api.example.com/data' ); // BAD!
const data = await response.json();
return { data };
}
} );
Direct Database Calls
// WRONG: Database I/O in workflow
export default workflow( {
fn: async input => {
const user = await db.users.findById( input.userId ); // BAD!
return { user };
}
} );
File System Operations
// WRONG: File I/O in workflow
import fs from 'fs/promises';
export default workflow( {
fn: async input => {
const data = await fs.readFile( input.path, 'utf-8' ); // BAD!
return { data };
}
} );
Solution
Move ALL I/O operations to step functions. Steps are designed to handle non-deterministic operations.
Before (Wrong)
export default workflow( {
fn: async input => {
const response = await fetch( 'https://api.example.com/data' );
const data = await response.json();
return { data };
}
} );
After (Correct)
import { z, step, workflow } from '@outputai/core';
import { createKyClient } from '@outputai/core/http';
// Create a step for the I/O operation
export const fetchData = step( {
name: 'fetchData',
inputSchema: z.object( {
endpoint: z.string()
} ),
outputSchema: z.object( {
data: z.unknown()
} ),
fn: async input => {
const client = createKyClient( { prefix: 'https://api.example.com' } );
const data = await client.get( input.endpoint ).json();
return { data };
}
} );
// Workflow only orchestrates steps
export default workflow( {
inputSchema: z.object( {} ),
outputSchema: z.object( { data: z.unknown() } ),
fn: async input => {
const result = await fetchData( { endpoint: 'data' } );
return result;
}
} );
Complete Example: Database Operation
Before (Wrong)
export default workflow( {
fn: async input => {
const user = await prisma.user.findUnique( {
where: { id: input.userId }
} );
const orders = await prisma.order.findMany( {
where: { userId: input.userId }
} );
return { user, orders };
}
} );
After (Correct)
import { z, step, workflow } from '@outputai/core';
import { prisma } from '../lib/db';
export const fetchUser = step( {
name: 'fetchUser',
inputSchema: z.object( { userId: z.string() } ),
outputSchema: z.object( {
user: z.object( {
id: z.string(),
name: z.string(),
email: z.string()
} ).nullable()
} ),
fn: async input => {
const user = await prisma.user.findUnique( {
where: { id: input.userId }
} );
return { user };
}
} );
export const fetchOrders = step( {
name: 'fetchOrders',
inputSchema: z.object( { userId: z.string() } ),
outputSchema: z.object( {
orders: z.array( z.object( {
id: z.string(),
total: z.number()
} ) )
} ),
fn: async input => {
const orders = await prisma.order.findMany( {
where: { userId: input.userId }
} );
return { orders };
}
} );
export default workflow( {
inputSchema: z.object( { userId: z.string() } ),
outputSchema: z.object( {
user: z.unknown(),
orders: z.array( z.unknown() )
} ),
fn: async input => {
const { user } = await fetchUser( { userId: input.userId } );
const { orders } = await fetchOrders( { userId: input.userId } );
return { user, orders };
}
} );
Finding Direct I/O in Workflows
Search for common I/O patterns in workflow files:
# Find fetch calls
grep -rn "await fetch" src/workflows/
# Find axios calls
grep -rn "axios\." src/workflows/
# Find database operations
grep -rn "prisma\.\|db\.\|mongoose\." src/workflows/
# Find file system operations
grep -rn "fs\.\|readFile\|writeFile" src/workflows/
Then review each match to see if it's in a workflow function vs a step function.
What CAN Be in Workflow Functions
Workflow functions should contain:
- Step calls:
await myStep( input ) - Orchestration logic: conditionals, loops (over step calls)
- Data transformation: Pure functions on step results
- Constants: Static values and configuration
Workflow functions should NOT contain:
- HTTP/API calls
- Database operations
- File system operations
- External service calls
- Anything that talks to the network or filesystem
Verification
After moving I/O to steps:
- Run the workflow:
npx output workflow run <name> --input '<input>' - Check the run history:
npx output workflow debug <id> - Verify steps appear: Look for your I/O steps in the trace
- Confirm no errors: No determinism warnings or hangs
Benefits of Steps for I/O
- Retry logic: Steps can be retried on failure
- Tracing: I/O operations appear in workflow traces
- Timeouts: Steps can have individual timeouts
- Determinism: Replays use recorded results
- Debugging: Clear visibility into what happened
Related Issues
- For HTTP client best practices, see
output-error-http-client - For non-determinism from other causes, see
output-error-nondeterminism
Fix HTTP Client Misuse
Overview
This skill helps diagnose and fix issues caused by using axios, fetch, or other HTTP clients directly instead of Output SDK's createKyClient from @outputai/core/http. The Output SDK client provides tracing, automatic retries, and better error handling.
When to Use This Skill
You're seeing:
- Untraced HTTP requests (not appearing in workflow traces)
- Missing error details for failed requests
- axios-related errors or import issues
- Retries not working for HTTP failures
- Inconsistent timeout behavior
Root Cause
Using axios, fetch, or other HTTP clients directly bypasses Output SDK's:
- Request/response tracing: Calls aren't logged in workflow traces
- Automatic retries: Failed requests aren't retried
- Error standardization: Error formats may be inconsistent
- Timeout handling: Timeouts may not integrate with step timeouts
Symptoms
Using axios Directly
// WRONG: Using axios
import axios from 'axios';
export const fetchData = step( {
name: 'fetchData',
fn: async input => {
const response = await axios.get( 'https://api.example.com/data' );
return response.data;
}
} );
Using fetch Directly
// WRONG: Using fetch
export const fetchData = step( {
name: 'fetchData',
fn: async input => {
const response = await fetch( 'https://api.example.com/data' );
return response.json();
}
} );
Solution
Use createKyClient from @outputai/core/http:
Basic Usage
import { z, step } from '@outputai/core';
import { createKyClient } from '@outputai/core/http';
export const fetchData = step( {
name: 'fetchData',
inputSchema: z.object( {
endpoint: z.string()
} ),
outputSchema: z.object( {
data: z.unknown()
} ),
fn: async input => {
const client = createKyClient( {
prefix: 'https://api.example.com'
} );
const data = await client.get( input.endpoint ).json();
return { data };
}
} );
With Full Configuration
import { createKyClient } from '@outputai/core/http';
const client = createKyClient( {
prefix: 'https://api.example.com',
timeout: 30000, // 30 second timeout
retry: {
limit: 3, // Retry up to 3 times
methods: [ 'GET', 'POST' ], // Which methods to retry
statusCodes: [ 408, 500, 502, 503, 504 ] // Which status codes trigger retry
},
headers: {
'Authorization': `Bearer ${apiKey}`,
'Content-Type': 'application/json'
}
} );
HTTP Methods
GET Request
const data = await client.get( 'users/123' ).json();
POST Request
const result = await client.post( 'users', {
json: {
name: 'John',
email: 'john@example.com'
}
} ).json();
PUT Request
const updated = await client.put( 'users/123', {
json: {
name: 'John Updated'
}
} ).json();
DELETE Request
await client.delete( 'users/123' );
With Query Parameters
const data = await client.get( 'search', {
searchParams: {
q: 'query',
limit: 10
}
} ).json();
Metadata-Only Responses
When code only reads metadata from a non-HEAD response, such as response.url, response.status, or headers, cancel the
unused body. Reading a body with .json(), .text(), etc. already consumes it.
const response = await client.get( url );
try {
return response.url;
} finally {
await response.body?.cancel();
}
Complete Migration Example
Before (Wrong - using axios)
import axios from 'axios';
import { step } from '@outputai/core';
export const createUser = step( {
name: 'createUser',
fn: async input => {
try {
const response = await axios.post(
'https://api.example.com/users',
{ name: input.name, email: input.email },
{
headers: { 'Authorization': `Bearer ${process.env.API_KEY}` },
timeout: 30000
}
);
return response.data;
} catch ( error ) {
if ( axios.isAxiosError( error ) ) {
throw new Error( `API Error: ${error.response?.data?.message}` );
}
throw error;
}
}
} );
After (Correct - using createKyClient)
import { z, step } from '@outputai/core';
import { createKyClient } from '@outputai/core/http';
import { credentials } from '@outputai/core/credentials';
export const createUser = step( {
name: 'createUser',
inputSchema: z.object( {
name: z.string(),
email: z.string().email()
} ),
outputSchema: z.object( {
id: z.string(),
name: z.string(),
email: z.string()
} ),
fn: async input => {
const client = createKyClient( {
prefix: 'https://api.example.com',
timeout: 30000,
retry: { limit: 3 },
headers: {
'Authorization': `Bearer ${credentials.require( 'service.api_key' )}`
}
} );
const user = await client.post( 'users', {
json: {
name: input.name,
email: input.email
}
} ).json();
return user;
}
} );
Error Handling
The Ky client provides structured error handling:
import { createKyClient, ky } from '@outputai/core/http';
export const fetchData = step( {
name: 'fetchData',
fn: async input => {
const client = createKyClient( { prefix: 'https://api.example.com' } );
try {
return await client.get( 'data' ).json();
} catch ( error ) {
if ( error instanceof ky.HTTPError ) {
// Access response details
const status = error.response.status;
const body = await error.response.json();
throw new Error( `API returned ${status}: ${body.message}` );
}
throw error;
}
}
} );
Missing Cost Tracking for Paid APIs
Symptom
A client wraps a paid third-party API, but its calls never show up in the API Costs section of npx output workflow cost — they're traced (so tracing works), but their cost is always $0 or absent from the report.
Root Cause
npx output workflow cost only knows about HTTP spend that was explicitly attached with addRequestCost from @outputai/core/http. Tracing a request is automatic; costing it is not — a client that never calls addRequestCost will trace correctly and cost nothing, which is expected for free/internal services but a bug for paid ones.
Solution
Attach cost inside an afterResponse hook on the client so every call is costed automatically, rather than at each call site. See output-dev-http-client-create (section "Attaching Request Cost (Paid APIs)") for the metered-response and flat-rate patterns, and output-dev-cost-hooks for forwarding those costs to an external observability system.
// WRONG: paid API, no cost tracking
const client = createKyClient( { prefix: 'https://api.service.com' } );
// CORRECT: cost attached automatically for every call
import { addRequestCost, createKyClient } from '@outputai/core/http';
const client = createKyClient( {
prefix: 'https://api.service.com',
hooks: {
afterResponse: [
( _request, _options, response ) => {
if ( !response.ok ) return;
addRequestCost( response, 0.005 );
}
]
}
} );
Finding axios/fetch Usage
Search your codebase:
# Find axios imports
grep -rn "from 'axios'\|from \"axios\"" src/
# Find fetch calls
grep -rn "await fetch(" src/
# Find other HTTP libraries
grep -rn "got\|node-fetch\|request\|superagent" src/
Benefits of createKyClient
- Tracing: Requests appear in workflow traces with timing
- Automatic Retries: Configurable retry logic for transient failures
- Consistent Errors: Standardized error format across all requests
- Timeout Integration: Works with step and workflow timeouts
- Type Safety: Full TypeScript support
Configuration Options
| Option | Description | Default |
|---|---|---|
prefix | Base URL for all requests | (required) |
timeout | Request timeout in ms | 10000 |
retry.limit | Max retry attempts | 2 |
retry.methods | HTTP methods to retry | ['GET', 'PUT', 'HEAD', 'DELETE', 'OPTIONS', 'TRACE'] |
retry.statusCodes | Status codes to retry | [408, 413, 429, 500, 502, 503, 504] |
headers | Default headers | {} |
Verification
After migrating to createKyClient:
- Run the workflow:
npx output workflow run <name> --input '<input>' - Check the run history:
npx output workflow debug <id> - Verify tracing: HTTP requests should appear in the step trace
- Test retries: Simulate failures to verify retry behavior
Related Issues
- For I/O in workflow functions, see
output-error-direct-io - For connection issues, see
output-services-check - For encrypted secrets management, see
output-dev-credentials - For attaching and forwarding request cost, see
output-dev-http-client-createandoutput-dev-cost-hooks
Fix Missing Schema Definitions
Overview
This skill helps diagnose and fix issues caused by steps that lack explicit inputSchema or outputSchema definitions. Schemas are essential for type safety, validation, and proper data serialization between steps.
When to Use This Skill
You're seeing:
- Type errors at step boundaries
- Undefined properties in step inputs/outputs
- Validation failures when passing data between steps
- TypeScript errors about incompatible types
- Runtime errors about unexpected data shapes
Root Cause
Steps without explicit schemas:
- Don't validate input data at runtime
- Don't provide TypeScript type inference
- May serialize/deserialize data incorrectly
- Can pass undefined or malformed data silently
Symptoms
Missing Input Schema
// WRONG: No input validation
export const processData = step( {
name: 'processData',
// inputSchema: missing!
outputSchema: z.object( { result: z.string() } ),
fn: async input => {
return { result: input.value }; // input.value might be undefined!
}
} );
Missing Output Schema
// WRONG: No output validation
export const fetchData = step( {
name: 'fetchData',
inputSchema: z.object( { id: z.string() } ),
// outputSchema: missing!
fn: async input => {
return { data: await getFromApi( input.id ) }; // Output shape not validated
}
} );
Both Schemas Missing
// WRONG: No validation at all
export const transformData = step( {
name: 'transformData',
// No schemas!
fn: async input => {
return transform( input );
}
} );
Solution
Always define both inputSchema and outputSchema for every step:
Complete Step Definition
import { z, step } from '@outputai/core';
export const processData = step( {
name: 'processData',
inputSchema: z.object( {
id: z.string(),
value: z.number(),
optional: z.string().optional()
} ),
outputSchema: z.object( {
result: z.string(),
processedAt: z.number()
} ),
fn: async input => {
// input is fully typed: { id: string, value: number, optional?: string }
return {
result: `Processed ${input.id}`,
processedAt: Date.now()
};
// output is validated against outputSchema
}
} );
Schema Definition Best Practices
Use Descriptive Schemas
// Good: Clear, descriptive schema
inputSchema: z.object( {
userId: z.string().uuid(),
email: z.string().email(),
age: z.number().int().positive()
} )
Handle Optional Fields
inputSchema: z.object( {
required: z.string(),
optional: z.string().optional(),
withDefault: z.string().default( 'fallback' )
} )
Use Schema Composition
// Define reusable schemas
const userSchema = z.object( {
id: z.string(),
name: z.string()
} );
const addressSchema = z.object( {
street: z.string(),
city: z.string()
} );
// Compose in step
inputSchema: z.object( {
user: userSchema,
address: addressSchema
} )
Handle Arrays and Nested Objects
inputSchema: z.object( {
items: z.array( z.object( {
id: z.string(),
quantity: z.number()
} ) ),
metadata: z.record( z.string() )
} )
Finding Steps Without Schemas
Search your codebase:
# Find step definitions
grep -rn "step({" src/workflows/
# Look for steps without inputSchema
grep -A5 "step({" src/workflows/ | grep -B2 "fn:"
# Check if schemas are present
grep -rn "inputSchema:" src/workflows/
grep -rn "outputSchema:" src/workflows/
Review each step definition to ensure both schemas are present.
Benefits of Explicit Schemas
- Runtime Validation: Catches data errors early
- Type Safety: Full TypeScript inference in step functions
- Documentation: Schemas document expected data shapes
- Serialization: Ensures proper data serialization between steps
- Error Messages: Clear validation errors when data is wrong
Common Schema Patterns
API Response Steps
export const fetchUser = step( {
name: 'fetchUser',
inputSchema: z.object( {
userId: z.string()
} ),
outputSchema: z.object( {
user: z.object( {
id: z.string(),
name: z.string(),
email: z.string()
} ).nullable(), // Handle not found
found: z.boolean()
} ),
fn: async input => {
const user = await api.getUser( input.userId );
return { user, found: user !== null };
}
} );
Transformation Steps
export const transformData = step( {
name: 'transformData',
inputSchema: z.object( {
raw: z.array( z.unknown() )
} ),
outputSchema: z.object( {
processed: z.array( z.object( {
id: z.string(),
value: z.number()
} ) ),
count: z.number()
} ),
fn: async input => {
const processed = input.raw.map( transformItem );
return { processed, count: processed.length };
}
} );
Void Output Steps
For steps that don't return meaningful data:
export const logEvent = step( {
name: 'logEvent',
inputSchema: z.object( {
event: z.string(),
data: z.record( z.unknown() )
} ),
outputSchema: z.object( {
logged: z.literal( true )
} ),
fn: async input => {
await logger.log( input.event, input.data );
return { logged: true };
}
} );
Verification
After adding schemas:
- TypeScript check:
npm run output:worker:buildshould pass without type errors - Runtime test:
npx output workflow run <name> --input '<input>'should validate correctly - Invalid input test: Pass invalid data and verify validation errors appear
Related Issues
- For Zod import issues, see
output-error-zod-import - For type mismatches despite schemas, verify schema matches actual data
Fix Non-Determinism Errors
Overview
This skill helps diagnose and fix non-determinism errors in Output SDK workflows. Workflows must be deterministic because Temporal may replay them during recovery or retries, and the replay must produce identical results.
When to Use This Skill
You're seeing:
- "non-deterministic" error messages
- Replay failures after workflow restart
- Inconsistent results between runs with same input
- Errors during workflow recovery
- Warnings about determinism violations
Root Cause
Temporal workflows must be deterministic: given the same input, they must always execute the same sequence of operations. This is because Temporal replays workflow history to recover state after crashes or restarts.
Non-deterministic operations break this replay mechanism because they produce different values each time.
Common Causes and Solutions
1. Math.random()
Problem: Random values differ on each execution.
// WRONG: Non-deterministic
export default workflow( {
fn: async input => {
const id = Math.random().toString( 36 ); // Different each time!
return await processWithId( { id } );
}
} );
Solution: Pass random values as workflow input or generate in a step.
// Option 1: Pass as input
export default workflow( {
inputSchema: z.object( {
id: z.string() // Generate ID before calling workflow
} ),
fn: async input => {
return await processWithId( { id: input.id } );
}
} );
// Option 2: Generate in a step (steps can be non-deterministic)
export const generateId = step( {
name: 'generateId',
fn: async () => ( { id: Math.random().toString( 36 ) } )
} );
export default workflow( {
fn: async input => {
const { id } = await generateId( {} );
return await processWithId( { id } );
}
} );
2. Date.now() / new Date()
Problem: Timestamps change between executions.
// WRONG: Non-deterministic
export default workflow( {
fn: async input => {
const timestamp = Date.now(); // Different each replay!
return await logEvent( { timestamp } );
}
} );
Solution: Pass timestamps as input or use Temporal's time API.
// Option 1: Pass as input
export default workflow( {
inputSchema: z.object( {
timestamp: z.number()
} ),
fn: async input => {
return await logEvent( { timestamp: input.timestamp } );
}
} );
// Option 2: Generate in a step
export const getTimestamp = step( {
name: 'getTimestamp',
fn: async () => ( { timestamp: Date.now() } )
} );
3. crypto.randomUUID()
Problem: UUIDs differ each execution.
// WRONG: Non-deterministic
import { randomUUID } from 'crypto';
export default workflow( {
fn: async input => {
const requestId = randomUUID(); // Different each time!
return await makeRequest( { requestId } );
}
} );
Solution: Generate UUIDs as input or in steps.
// Correct: Generate in step
export const generateRequestId = step( {
name: 'generateRequestId',
fn: async () => {
const { randomUUID } = await import( 'crypto' );
return { requestId: randomUUID() };
}
} );
4. Dynamic Imports
Problem: Dynamic imports may resolve differently.
// WRONG: Non-deterministic import timing
export default workflow( {
fn: async input => {
const module = await import( `./handlers/${input.type}` );
return module.handle( input );
}
} );
Solution: Use static imports and conditional logic.
// Correct: Static imports with conditional use
import { handleTypeA } from './handlers/typeA';
import { handleTypeB } from './handlers/typeB';
export default workflow( {
fn: async input => {
if ( input.type === 'A' ) {
return await handleTypeA( input );
} else {
return await handleTypeB( input );
}
}
} );
5. Environment Variables
Problem: Environment may differ between replays.
// WRONG: Environment can change
export default workflow( {
fn: async input => {
const apiUrl = process.env.API_URL; // May differ on different workers
return await callApi( { url: apiUrl } );
}
} );
Solution: Pass configuration as input or use constants.
// Correct: Pass as input
export default workflow( {
inputSchema: z.object( {
apiUrl: z.string()
} ),
fn: async input => {
return await callApi( { url: input.apiUrl } );
}
} );
How to Find Non-Deterministic Code
Search for Common Patterns
# Find Math.random usage
grep -rn "Math.random" src/workflows/
# Find Date.now or new Date
grep -rn "Date.now\|new Date" src/workflows/
# Find crypto random functions
grep -rn "randomUUID\|randomBytes" src/workflows/
# Find dynamic imports
grep -rn "import(" src/workflows/
Review Workflow Files
Look at your workflow fn functions specifically. Non-deterministic code is only a problem in workflow functions, not in step functions.
Verification Steps
- Fix the code using solutions above
- Run the workflow:
npx output workflow run <name> --input '<input>' - Run again with same input: Result should be identical
- Check for errors: No "non-deterministic" messages
The Determinism Rule
Workflow functions must be deterministic:
- Same input = same execution path
- No side effects (network, filesystem, random values)
- Only orchestration logic and step calls
Step functions can be non-deterministic:
- Steps record their results in Temporal history
- Replays use recorded results, not re-execution
- All I/O should happen in steps
Debugging Tip
If unsure whether code is causing issues:
# Run the workflow
npx output workflow start my-workflow --input '{"input": "test"}'
# Get the workflow ID and run debug to see replay behavior
npx output workflow debug <workflowId>
Look for errors or warnings about non-determinism in the history.
Related Issues
- For I/O in workflow code, see
output-error-direct-io - For random values needed in logic, generate them in steps or pass as input
Handle Errors in Workflows
Understand when workflow catch blocks run
Steps and evaluators run as Temporal Activities. Temporal applies their configured retry policy before returning a successful result or throwing a final failure to workflow code.
A workflow catch around a step therefore runs only after the step has exhausted its Activity retries. Catching that failure does not disable or bypass the Activity retry policy.
The catch block decides what the workflow does with the final failure:
- Return a fallback value.
- Run an alternative step.
- Record a partial failure and continue.
- Handle an expected error type.
- Rethrow the error and fail the workflow.
Let unexpected failures propagate
Do not add a catch block that only wraps or logs an error. It can discard Temporal's failure type, cause chain, and retry metadata.
// Avoid: replaces the original Temporal failure chain.
try {
return await fetchData( input );
} catch ( error ) {
throw new Error( `Fetch failed: ${error.message}` );
}
When the workflow cannot recover, let the step failure propagate:
export default workflow( {
name: 'fetch_workflow',
fn: async input => fetchData( input )
} );
If a catch block handles only known failures, always rethrow everything else.
Check specific error types with hasErrorType
Errors crossing from a step or evaluator into workflow code are serialized into a Temporal failure cause chain. The original JavaScript object identity is not preserved, so error instanceof CustomError is unreliable.
Use hasErrorType from @outputai/core. It walks the cause chain and matches native instances and Temporal's serialized type and name fields.
import { hasErrorType, workflow } from '@outputai/core';
import { lookupCompany } from './steps.js';
import { CompanyNotFoundError } from './types.js';
export default workflow( {
name: 'company_lookup',
fn: async input => {
try {
return await lookupCompany( input );
} catch ( error ) {
if ( hasErrorType( error, CompanyNotFoundError ) ) {
return null;
}
throw error;
}
}
} );
Define custom Error classes in a shared module imported by both the workflow and step:
export class CompanyNotFoundError extends Error {}
Do not inspect a fixed number of .cause levels or check only the outer ActivityFailure.
Valid try/catch patterns
Fallback step
const fetchWithFallback = async input => {
try {
return await fetchFromPrimarySource( input );
} catch {
return fetchFromSecondarySource( input );
}
};
Use this only when every primary-source failure should trigger the fallback. Use hasErrorType when only specific failures are recoverable.
Partial failures
const results = await Promise.all( input.items.map( async item => {
try {
return { item, value: await processItem( item ), ok: true };
} catch ( error ) {
const message = error instanceof Error ? error.message : String( error );
return { item, error: message, ok: false };
}
} ) );
Ensure the workflow output schema supports both successful and failed entries. Do not silently omit failures.
Configure Activity retries separately
Configure step and evaluator retry behavior through workflow Activity options. The workflow catch block handles only the failure that remains after this policy is exhausted.
export default workflow( {
name: 'company_lookup',
fn: async input => lookupCompany( input ),
options: {
activityOptions: {
retry: {
initialInterval: '1s',
maximumAttempts: 3
}
}
}
} );
Review checklist
- Catch only when the workflow has a defined recovery path.
- Use
hasErrorTypefor specific errors originating in steps or evaluators. - Rethrow unmatched errors unchanged.
- Do not replace an existing failure with a generic
Error. - Keep custom Error class names stable because Temporal serializes them as failure types.
- Configure retries independently from workflow catch behavior.
Fix Zod Import Source Issues
Overview
This skill helps diagnose and fix a common issue where Zod schemas are imported from the wrong source. Output SDK requires schemas to be imported from @outputai/core, not directly from zod.
When to Use This Skill
You're seeing:
- "incompatible schema" errors
- Type errors at step boundaries
- Schema validation failures when passing data between steps
- Errors mentioning Zod types not matching
- "Expected ZodObject but received..." errors
Root Cause
The issue occurs when you import z from zod instead of @outputai/core. While both provide Zod schemas, they create different schema instances that aren't compatible with each other within the Output SDK context.
Why this matters: Output SDK uses a specific version of Zod internally for serialization and validation. When you use a different Zod instance, the schemas are technically different objects even if they define the same shape.
Symptoms
Error Messages
Error: Incompatible schema types
Error: Schema validation failed: expected compatible Zod instance
TypeError: Cannot read property 'parse' of undefined
Code Patterns That Cause This
// WRONG: Importing from 'zod' directly
import { z } from 'zod';
const inputSchema = z.object( {
name: z.string()
} );
Solution
Step 1: Find All Zod Imports
Search your codebase for incorrect imports:
grep -r "from 'zod'" src/
grep -r 'from "zod"' src/
Step 2: Update Imports
Change all imports from:
// Wrong
import { z } from 'zod';
To:
// Correct
import { z } from '@outputai/core';
Step 3: Verify No Direct Zod Dependencies
Check your imports don't accidentally use zod elsewhere:
grep -r "import.*zod" src/
All matches should show @outputai/core, not zod.
Complete Example
Before (Wrong)
// src/workflows/my-workflow/steps/process.ts
import { z } from 'zod'; // Wrong!
import { step } from '@outputai/core';
export const processStep = step( {
name: 'processData',
inputSchema: z.object( {
id: z.string()
} ),
outputSchema: z.object( {
result: z.string()
} ),
fn: async input => {
return { result: `Processed ${input.id}` };
}
} );
After (Correct)
// src/workflows/my-workflow/steps/process.ts
import { z, step } from '@outputai/core'; // Correct!
export const processStep = step( {
name: 'processData',
inputSchema: z.object( {
id: z.string()
} ),
outputSchema: z.object( {
result: z.string()
} ),
fn: async input => {
return { result: `Processed ${input.id}` };
}
} );
Verification Steps
1. Check for remaining wrong imports
# Should return no results
grep -r "from 'zod'" src/
grep -r 'from "zod"' src/
2. Build the project
npm run output:worker:build
3. Run the workflow
npx output workflow run <workflowName> --input '<input>'
Prevention
ESLint Rule (if using ESLint)
Add a rule to prevent direct zod imports:
// .eslintrc.js
module.exports = {
rules: {
'no-restricted-imports': [ 'error', {
paths: [ {
name: 'zod',
message: "Import { z } from '@outputai/core' instead of 'zod'"
} ]
} ]
}
};
IDE Settings
Configure your editor to auto-import from @outputai/core:
For VS Code, add to settings.json:
{
"typescript.preferences.autoImportFileExcludePatterns": ["zod"]
}
Common Gotchas
Mixed Imports in Same File
Even one wrong import can cause issues:
import { z } from '@outputai/core';
import { z as zod } from 'zod'; // This causes problems!
Indirect Dependencies
If a utility file uses the wrong import and is shared:
// utils/schemas.ts
import { z } from 'zod'; // Wrong! This affects all files using these schemas
export const idSchema = z.string().uuid();
Third-Party Libraries
If using external Zod schemas, you may need to recreate them:
// Don't use: externalLibrary.schema
// Instead: recreate the schema with @outputai/core's z
Related Issues
- If schemas are correct but you still see type errors, check
output-error-missing-schemas - For validation failures with correct imports, verify schema definitions match actual data
Auditing an Eval Suite
Overview
Audit your eval suite to determine whether it actually catches real failures. This skill provides a structured diagnostic that identifies gaps in error analysis, evaluator design, judge validation, and dataset coverage, with concrete remediation steps for each finding.
When to Use
- Inheriting an eval suite from another team or developer
- Suspecting that evals pass but production quality is poor
- After switching models, rewriting prompts, or changing pipeline logic
- Periodic health check (quarterly or after major releases)
Step 1: Gather Artifacts
Read the eval infrastructure files for the workflow being audited:
src/workflows/<workflow_name>/
├── tests/
│ ├── datasets/ # YAML dataset files
│ │ ├── *.yml
│ │ └── ...
│ └── evals/
│ ├── evaluators.ts # Evaluator definitions
│ ├── workflow.ts # Eval workflow definition
│ └── *.prompt # Judge prompt files
Inventory what exists:
| Artifact | File(s) | Count |
|---|---|---|
| Evaluators | tests/evals/evaluators.ts | ? |
| Eval workflow | tests/evals/workflow.ts | ? entries in evals array |
| Judge prompts | tests/evals/*.prompt | ? |
| Datasets | tests/datasets/*.yml | ? |
| Datasets with ground_truth | ? of above | ? |
| Datasets with last_output | ? of above | ? |
If any of these are missing entirely, note it and skip to "Starting From Zero" at the bottom.
Step 2: Run the Diagnostic
Evaluate each of the four areas below. For each, assign a status:
- Pass — Meets the standard
- Warn — Partially meets the standard, improvements needed
- Fail — Does not meet the standard, significant risk
Area 1: Error Analysis Grounding
Question: Were the evaluators derived from observed failure modes in real workflow traces?
Check:
- Do failure categories exist (documented in a file, comments, or commit history)?
- Does each evaluator map to a specific failure category?
- Or are evaluators measuring generic qualities ("quality score", "overall rating")?
Pass criteria:
- Each evaluator targets a named failure mode (e.g., "check_tone" targets tone mismatch, not "evaluate general quality")
- Failure categories were derived from reviewing real traces (not brainstormed)
Common failures:
- Evaluators named
evaluate_quality,check_overall,rate_output— generic, not grounded in observed failures - Evaluators were written based on what seemed important, not what actually fails
- No evidence of trace review before evaluator creation
Remediation: output-eval-error-analysis — Review 50+ traces and categorize actual failure modes before modifying evaluators
Area 2: Evaluator Design
Question: Are the evaluators well-designed for reliable automated evaluation?
Check each evaluator in tests/evals/evaluators.ts:
| Check | What to look for |
|---|---|
| One failure mode per judge | Each judgeVerdict() evaluator targets exactly one criterion |
| Binary verdicts | Judge prompts use pass/fail, not Likert scales (1-5) or multi-axis ratings |
| Code-based where possible | Objective checks use Verdict.* helpers, not LLM judges |
| Few-shot examples in judges | Judge .prompt files include pass, fail, and borderline examples |
| Critique before verdict | Judge prompts request critique/reasoning before the verdict in structured output |
| Appropriate criticality | required for blocking failures, informational for nice-to-have checks |
| Correct interpret type | interpret config matches what the evaluator returns |
Pass criteria:
- All checks above are met for every evaluator
Common failures:
- A single judge prompt evaluates 3+ criteria simultaneously ("Rate tone, accuracy, and completeness")
- Judge prompts have no few-shot examples
- Deterministic checks (length, string contains, regex) use LLM judges instead of
Verdict.* interprettype doesn't match evaluator return type (e.g.,judgeVerdict()withinterpret: { type: 'boolean' })
Remediation: output-eval-judge-prompt — Redesign judge prompts following the four-component structure
Area 3: Judge Validation
Question: Have LLM judges been validated against human labels?
Check for each LLM-based evaluator (those using judgeVerdict(), judgeScore(), judgeLabel()):
| Check | What to look for |
|---|---|
| Human labels exist | Datasets have ground_truth.evals.<evaluator_name>.verdict populated |
| TPR/TNR measured | Validation results documented (file, comment, or commit) |
| Train/dev/test split | Few-shot examples in the judge prompt come from a designated train split, not from the same data used for measurement |
| Metrics meet threshold | TPR > 80% and TNR > 80% (target: > 90%) |
Pass criteria:
- Every LLM judge has documented TPR/TNR metrics above 80%
- Train/dev/test split was used (no data leakage)
Common failures:
- No validation at all — judges were written and immediately deployed
- Few-shot examples in the judge prompt are the same examples used to measure metrics (data leakage)
- "It seems to work" without quantitative measurement
- Only raw accuracy reported (masks class imbalance)
Remediation: output-eval-validate-judge — Calibrate each judge against human labels using TPR/TNR
Area 4: Dataset Coverage
Question: Do the datasets adequately cover the failure space?
Check:
| Check | What to look for |
|---|---|
| Dataset count | Minimum 10 for simple workflows, 20+ for complex ones |
| Diversity | Datasets vary across multiple input dimensions, not just happy paths |
| Failure representation | At least 30% of datasets have human_verdict: fail in ground_truth |
| Ground truth populated | Most datasets have ground_truth with per-evaluator labels |
| Real + synthetic mix | Includes production traces alongside synthetic test cases |
| No near-duplicates | Each dataset tests a meaningfully different scenario |
Pass criteria:
- 20+ diverse datasets with ground truth
- Both pass and fail cases represented (not 95% passes)
- Datasets cover different input dimensions
Common failures:
- Only 3-5 datasets, all happy-path variations
- 100% of datasets pass (no failure cases to validate judges against)
- Datasets are synthetic-only with no real production traces
- Ground truth fields are empty or missing
Remediation: output-eval-dataset-design — Design diverse datasets using dimension-based variation
Step 3: Compile the Report
Summarize findings in a structured format:
# Eval Audit: <workflow_name>
# Date: YYYY-MM-DD
# Auditor: <name>
## Summary
| Area | Status | Key Finding |
|------|--------|-------------|
| Error Analysis Grounding | Warn | Evaluators seem reasonable but no documented trace review |
| Evaluator Design | Fail | Single judge evaluates 3 criteria simultaneously |
| Judge Validation | Fail | No validation performed on any LLM judge |
| Dataset Coverage | Warn | 12 datasets but only 2 are failure cases |
## Findings
### 1. Error Analysis Grounding — WARN
Evaluators target reasonable criteria (tone, topic, length) but there is no evidence
that these were derived from observed failures. The eval suite may be missing the
workflow's actual top failure modes.
**Next step:** Run error analysis on 50+ production traces (`output-eval-error-analysis`)
### 2. Evaluator Design — FAIL
`evaluate_overall_quality` in evaluators.ts uses a single judgeVerdict() call that
assesses tone, accuracy, and completeness simultaneously. This makes failures
unactionable — when it fails, you don't know which criterion failed.
**Next step:** Split into three focused judges (`output-eval-judge-prompt`)
### 3. Judge Validation — FAIL
No TPR/TNR metrics exist for any LLM judge. The judge_quality@v1.prompt has no
few-shot examples.
**Next step:** Label 100 datasets, validate each judge (`output-eval-validate-judge`)
### 4. Dataset Coverage — WARN
12 datasets exist with cached output. Only 2 have ground_truth.human_verdict: fail.
All inputs are simple topics with no edge cases.
**Next step:** Design 20+ diverse datasets (`output-eval-dataset-design`)
## Priority Order
1. Error analysis (foundational — may change which evaluators are needed)
2. Split holistic judge into focused judges
3. Expand datasets to 30+ with balanced pass/fail
4. Validate all LLM judges
Starting From Zero
If the workflow has no eval infrastructure at all:
- Start with error analysis —
output-eval-error-analysis. Review 50+ workflow traces. - Build datasets —
output-eval-dataset-design. Create 20+ diverse datasets. - Implement evaluators —
output-dev-eval-testing. Writeverify()evaluators andevalWorkflow(). - Design judge prompts —
output-eval-judge-prompt. For subjective criteria only. - Validate judges —
output-eval-validate-judge. Before trusting any LLM judge.
Do not skip error analysis. Building evaluators without understanding how the workflow fails wastes effort on the wrong things.
Related Skills
output-eval-error-analysis— Systematic trace review and failure categorizationoutput-eval-judge-prompt— Design effective LLM judge promptsoutput-eval-dataset-design— Generate diverse test datasetsoutput-eval-validate-judge— Calibrate LLM judges against human labelsoutput-dev-eval-testing— Implementation reference for offline eval testingoutput-dev-evaluator-function— Implementation reference for runtime evaluators
Designing Eval Datasets
Overview
Diverse datasets catch more failures. This skill teaches dimension-based dataset design — systematically varying inputs along axes that target failure-prone regions of your workflow. The output is a set of YAML dataset files ready for output workflow test.
When to Use
- Bootstrapping an eval dataset for a new workflow
- Existing datasets only cover happy paths
- Real production traces are sparse (fewer than 50)
- Stress-testing specific failure hypotheses from error analysis
When NOT to Use
- You already have 100+ representative real traces — use stratified sampling from those instead of generating synthetic data
- You haven't done error analysis yet — do that first (
output-eval-error-analysis) so your dimensions target real failure modes, not guesses
Step 1: Define Dimensions
Identify 3+ axes of input variation that target anticipated failure modes. Each dimension should vary one aspect of the input that you expect to influence output quality.
Deriving dimensions from error analysis
Map failure categories to input properties that trigger them:
| Failure Category | Triggering Input Property | Dimension |
|---|---|---|
| Off-topic drift | Ambiguous or broad topics | Topic specificity: specific / broad / ambiguous |
| Tone mismatch | Conflicting tone signals | Tone difficulty: simple / nuanced / contradictory |
| Too short | Short or vague input | Input detail: minimal / moderate / comprehensive |
| Missing sections | Many explicit requirements | Requirement count: 0 / 1-2 / 5+ |
| Hallucinated URLs | Technical topics with real entities | Entity density: none / few / many |
Example dimensions for a blog generation workflow
| Dimension | Values | Why |
|---|---|---|
| Topic complexity | simple, technical, ambiguous | Technical and ambiguous topics trigger more hallucination and drift |
| Tone request | none, formal, casual, contradictory | Explicit tone requests reveal tone-matching failures |
| Length constraint | none, short (100w), long (1000w) | Extreme length constraints trigger truncation and padding |
| Required sections | none, 1 section, 3+ sections | Multiple required sections stress structural compliance |
Aim for 3-5 dimensions. More than 5 creates an unmanageable combinatorial space.
Step 2: Draft Tuples
Create ~20 combinations of dimension values. Cover the extremes and the combinations most likely to cause failures.
Tuple selection strategy
- Cover every dimension value at least twice — ensures no blind spots
- Pair difficult values together — "ambiguous topic + contradictory tone + 3+ required sections" is where failures cluster
- Include a few easy combinations — confirms the workflow works under normal conditions
- Avoid near-duplicates — each tuple should test a distinct scenario
Example tuples
| # | Topic Complexity | Tone | Length | Required Sections |
|---|---|---|---|---|
| 1 | simple | none | none | none |
| 2 | simple | formal | short | 1 section |
| 3 | technical | formal | long | 3+ sections |
| 4 | technical | casual | short | none |
| 5 | ambiguous | formal | long | 1 section |
| 6 | ambiguous | contradictory | none | 3+ sections |
| 7 | simple | contradictory | long | none |
| 8 | technical | none | none | 3+ sections |
| 9 | ambiguous | casual | short | 1 section |
| 10 | simple | casual | long | 3+ sections |
| 11 | technical | formal | short | 1 section |
| 12 | ambiguous | none | long | none |
| 13 | simple | formal | none | 3+ sections |
| 14 | technical | contradictory | long | 1 section |
| 15 | ambiguous | formal | short | 3+ sections |
| 16 | simple | none | short | 1 section |
| 17 | technical | casual | long | 3+ sections |
| 18 | ambiguous | contradictory | short | none |
| 19 | technical | formal | none | none |
| 20 | ambiguous | casual | long | 3+ sections |
Review each tuple and ask: "Is this a realistic scenario a user might create?" Discard any that aren't.
Step 3: Convert Tuples to Workflow Inputs
Transform each tuple into a JSON object matching the workflow's inputSchema from types.ts.
Read the schema first
# Find the input schema
cat src/workflows/<workflow_name>/types.ts
Manual conversion (small datasets)
For each tuple, write the corresponding JSON input:
Tuple 3: technical + formal + long + 3+ sections
{
"topic": "Quantum error correction techniques in superconducting qubit architectures",
"tone": "formal",
"min_length": 1000,
"required_sections": ["Introduction", "Technical Background", "Current Approaches", "Challenges", "Conclusion"]
}
Tuple 6: ambiguous + contradictory + none + 3+ sections
{
"topic": "Things that matter",
"tone": "Write in a formal academic style but keep it super casual and fun",
"required_sections": ["Overview", "Deep Dive", "Takeaways"]
}
Use realistic, natural-sounding inputs. Avoid test-looking data like "Test topic 1" or "Lorem ipsum."
LLM-assisted conversion (larger datasets)
For 20+ tuples, use an LLM to batch-convert tuples into realistic inputs. Create a prompt that takes the tuple values and the inputSchema, then generates a natural JSON input. Review every generated input for realism before using it.
Step 4: Generate Dataset Files
Run each input through the workflow to capture real output:
# Generate datasets one at a time
npx output workflow dataset generate blog_generator \
--input '{"topic": "Quantum error correction", "tone": "formal", "min_length": 1000}' \
--name technical_formal_long
npx output workflow dataset generate blog_generator \
--input '{"topic": "Things that matter", "tone": "Write formally but keep it casual"}' \
--name ambiguous_contradictory
Each command creates a YAML file in tests/datasets/ with input and last_output populated.
Naming convention
Use names that encode the key dimensions:
tests/datasets/
├── simple_no_constraints.yml
├── simple_formal_short.yml
├── technical_formal_long_sections.yml
├── technical_casual_short.yml
├── ambiguous_formal_long.yml
├── ambiguous_contradictory_sections.yml
└── ...
Batch generation
For many datasets, create a shell script:
#!/bin/bash
# generate_datasets.sh
inputs=(
'{"topic": "Solar panels", "tone": "formal"}|simple_formal'
'{"topic": "Quantum error correction in superconducting qubits", "tone": "casual", "min_length": 1000}|technical_casual_long'
'{"topic": "Things that matter", "required_sections": ["Overview", "Dive", "Takeaways"]}|ambiguous_sections'
)
for entry in "${inputs[@]}"; do
IFS='|' read -r input name <<< "$entry"
echo "Generating: $name"
npx output workflow dataset generate blog_generator --input "$input" --name "$name"
done
Step 5: Add Ground Truth
After generation, edit each dataset YAML to add ground_truth labels. Review the last_output and assign verdicts per evaluator.
name: ambiguous_contradictory_sections
input:
topic: "Things that matter"
tone: "Write in a formal academic style but keep it super casual and fun"
required_sections: ["Overview", "Deep Dive", "Takeaways"]
last_output:
output:
title: "Things That Matter: A Casual Academic Exploration"
blog_post: "So here's the thing about stuff that matters..."
date: '2026-03-25T00:00:00.000Z'
ground_truth:
human_verdict: fail
notes: "Contradictory tone caused drift; missing 'Deep Dive' section"
evals:
check_tone:
expected_tone: "formal academic style"
verdict: fail
check_topic:
required_topic: "Things that matter"
verdict: partial
check_sections:
required_sections: ["Overview", "Deep Dive", "Takeaways"]
verdict: fail
check_length:
min_length: 100
verdict: pass
Labeling tips
- Label the global
human_verdictfor every dataset (pass/fail) - Label the top 3 evaluators by failure rate first
- For borderline cases, decide and commit — don't leave ambiguous labels
- Record
notesexplaining your reasoning for future reference
Step 6: Supplement with Real Data
If you have production runs available, use them to fill coverage gaps. Pick completed runs with npx output workflow runs list blog_generator, then turn each into a dataset case:
# Read a completed run from Temporal (input and result)
npx output workflow dataset generate blog_generator --workflow-id <workflowId> --name real_case_1
Stratified sampling
After generating cases from real runs, check which dimensions are underrepresented:
- Review the generated datasets
- Tag each with its approximate dimension values
- Identify gaps (e.g., no ambiguous topics, no contradictory tones)
- Generate synthetic datasets specifically for the missing combinations
Mixing real and synthetic
A good eval dataset combines both:
- Real traces (60-70%) — capture authentic user behavior and edge cases you didn't anticipate
- Synthetic traces (30-40%) — fill coverage gaps and stress-test specific failure regions
Step 7: Quality Filter
Review every dataset before using it in evals.
Discard datasets where
- The input is unrealistic or test-looking
- The input is nearly identical to another dataset (redundant)
- The workflow errored out entirely (test infrastructure issues, not quality issues)
- The ground truth labels are ambiguous or contested
Target dataset count
| Workflow Complexity | Minimum Datasets | Target |
|---|---|---|
| Simple (1-2 steps, narrow input) | 10 | 20 |
| Medium (3-5 steps, moderate variation) | 20 | 40 |
| Complex (5+ steps, wide input space) | 40 | 80+ |
More datasets improve eval reliability. Aim for at least 5 datasets per failure category to get meaningful failure rates.
Verification
# List all datasets
npx output workflow dataset list blog_generator
# Run evals on all datasets with cached output
npx output workflow test blog_generator --cached
# Run evals on a subset
npx output workflow test blog_generator --cached --dataset technical_formal_long,ambiguous_contradictory
Check that:
- All datasets load without errors
- Dataset names are descriptive and encode key dimensions
- Every dataset has a
human_verdictinground_truth - The top 3 evaluators have per-evaluator ground truth in most datasets
- Datasets cover all dimension values from your tuple table
- No two datasets are near-duplicates
- Mix of real and synthetic traces (if production data is available)
Related Skills
output-eval-error-analysis— Identify failure modes that inform dimension selectionoutput-dev-eval-testing— Dataset YAML format, CLI commands,evalWorkflow()setupoutput-dev-scenario-file— Scenario JSON files as seed inputs for dataset generationoutput-eval-validate-judge— Split datasets into train/dev/test for judge validationoutput-eval-judge-prompt— Design LLM judge prompts that consume these datasets
Error Analysis for Workflow Evaluation
Overview
Review real workflow traces and categorize how your workflow fails before writing any evaluators. Evaluators built without error analysis target generic qualities ("is this good?") instead of the specific ways your workflow actually breaks. This skill walks you through the process.
When to Use
- Starting a new eval project for an existing workflow
- Production quality has dropped and you need to understand why
- After significant prompt, model, or pipeline changes
- Before building your first evaluator for a workflow
Step 1: Collect Traces
Gather 50-100 representative workflow executions. More traces = more reliable failure categories.
From recent runs
List recent workflow executions and look at their histories:
# List recent runs for a workflow
npx output workflow runs list <workflowName>
# Quick look at one run's Temporal history (JSON). Payloads are truncated, so use
# `dataset generate --workflow-id` below when you need the full input and result
npx output workflow debug <workflowId>
From production runs
Turn completed production runs into dataset YAML files, one command per workflow id taken from workflow runs list:
npx output workflow dataset generate <workflowName> --workflow-id <workflowId>
This creates a YAML file in tests/datasets/ with the input and last_output fields populated from the run, read from Temporal. The run must be completed.
From scenario-driven generation
If production traces are sparse, generate traces from scenario inputs:
# Generate a dataset from a scenario file
npx output workflow dataset generate <workflowName> basic --name basic_trace
# Generate from inline JSON
npx output workflow dataset generate <workflowName> --input '{"topic": "AI safety"}' --name ai_safety_trace
Run enough inputs to get 50+ traces. Prioritize diversity over volume — vary inputs across the dimensions you expect to matter.
Step 2: Review Traces Individually
Review each trace one at a time. For each trace, record:
| Field | What to write |
|---|---|
| Trace ID | The workflow execution ID |
| Verdict | Pass or Fail (binary — no "partial" at this stage) |
| Root cause | If Fail: what specifically went wrong and why |
| Notes | Anything surprising or worth remembering |
Review template
Create a file to track your reviews. A simple markdown table works:
# Error Analysis: <workflow_name>
# Date: YYYY-MM-DD
# Traces reviewed: 0 / 50
| # | Trace ID | Verdict | Root Cause | Notes |
|---|----------|---------|------------|-------|
| 1 | abc-123 | Fail | Hallucinated a URL that doesn't exist | Common with technical topics |
| 2 | def-456 | Pass | — | Clean output |
| 3 | ghi-789 | Fail | Ignored the "formal tone" requirement | Input had conflicting signals |
What to look for in each trace
Open the JSON trace and examine:
- Final output — Does it meet the user's intent? Is it correct?
- Step-by-step data flow — Did each step receive the right input and produce reasonable output?
- LLM responses — Did the model follow instructions? Did it hallucinate?
- Error states — Did any step fail, retry, or produce unexpected errors?
Critical rule: read first, categorize second
Review at least 30 traces before naming any failure categories. Premature categorization causes you to see patterns that aren't there and miss patterns that are. Just record what you observe.
Step 3: Group Into Failure Categories
After reviewing 30+ traces, patterns will emerge. Group your failures into 5-10 categories based on root cause, not surface symptoms.
Good categories (root cause)
- "Hallucinated URLs" — model invents links that don't exist
- "Tone mismatch" — output tone doesn't match the requested persona
- "Missing required section" — output omits a section the input explicitly requested
- "Factual error" — output contains verifiably wrong claims
- "Prompt injection leak" — user input manipulates the system prompt
Bad categories (surface symptoms)
- "Bad output" — too vague, not actionable
- "LLM error" — doesn't identify the specific failure
- "Quality issue" — could mean anything
Splitting and merging
- If a category has fewer than 3 examples, merge it into a broader category or note it as rare
- If a category has 15+ examples and contains distinct sub-patterns, split it
- Categories should be mutually exclusive — each failure belongs to exactly one category
Example categorization
For a blog generation workflow after reviewing 60 traces:
| Category | Count | Rate | Example |
|---|---|---|---|
| Hallucinated URLs | 8 | 13% | Invented links to non-existent pages |
| Tone mismatch | 6 | 10% | Casual tone when formal was requested |
| Off-topic drift | 5 | 8% | Blog about "AI" drifted to unrelated ML history |
| Missing sections | 4 | 7% | Skipped "conclusion" when explicitly requested |
| Too short | 3 | 5% | Under 200 words when 500+ requested |
| Total failures | 26 | 43% | |
| Passes | 34 | 57% |
Step 4: Label Datasets
Add ground_truth labels to your dataset YAML files so evaluators can validate against them. Each failure category maps to a future evaluator name.
YAML structure
name: ai_safety_trace
input:
topic: "AI safety"
tone: "formal"
min_length: 500
last_output:
output:
title: "Understanding AI Safety"
blog_post: "AI safety is super important and stuff..."
date: '2026-03-25T00:00:00.000Z'
ground_truth:
# Global ground truth (available to all evaluators)
human_verdict: fail
failure_categories:
- tone_mismatch
notes: "Used casual language despite formal tone request"
# Per-evaluator ground truth
evals:
check_tone:
expected_tone: formal
verdict: fail
check_length:
min_length: 500
verdict: pass
check_hallucinated_urls:
verdict: pass
The ground_truth.evals.<evaluator_name> fields map directly to the evaluator names you'll use in verify(). Each evaluator receives its own ground truth merged with the top-level ground truth via context.ground_truth.
Labeling efficiently
You don't need to label every dataset for every category. Focus on:
- Label all datasets with the global
human_verdict(pass/fail) - Label datasets for the top 3 failure categories by rate
- Add per-evaluator labels as you build each evaluator
Step 5: Decide What to Fix vs. Evaluate
Not every failure category needs an evaluator. Use this decision tree:
Is this failure caused by a fixable prompt/tool gap?
├─ YES → Fix the prompt or add the missing tool first
│ Re-run error analysis after the fix
└─ NO → Will this failure recur and need ongoing monitoring?
├─ YES → Build an evaluator
│ Can it be checked with deterministic code?
│ ├─ YES → Use Verdict.* helpers (contains, matches, gte, etc.)
│ └─ NO → Use judgeVerdict() with an LLM judge prompt
└─ NO → Document it and move on (rare edge case)
Prioritize by failure rate
Build evaluators for the highest-rate failure categories first. A failure at 13% matters more than one at 2%.
Code-based checks first
Many failures that seem subjective have objective proxies:
| Failure | Seems like... | But you can check with... |
|---|---|---|
| "Too short" | Subjective | Verdict.gte(output.length, threshold) |
| "Missing section" | Needs LLM | Verdict.contains(output, "## Conclusion") |
| "Hallucinated URLs" | Needs LLM | Extract URLs with regex, verify with HTTP HEAD |
| "Wrong format" | Needs LLM | Verdict.matches(output, expectedPattern) |
Reserve LLM judges for genuinely subjective criteria: tone, relevance, faithfulness, coherence.
Step 6: Map Categories to Evaluators
Create a mapping document that connects your failure categories to planned evaluators:
# Evaluator Plan: blog_generator
| Category | Rate | Evaluator Type | Evaluator Name | Criticality |
|----------|------|----------------|----------------|-------------|
| Hallucinated URLs | 13% | Code (URL extraction + HTTP check) | check_urls | required |
| Tone mismatch | 10% | LLM judge | check_tone | required |
| Off-topic drift | 8% | LLM judge | check_topic | required |
| Missing sections | 7% | Code (string contains) | check_sections | required |
| Too short | 5% | Code (length check) | check_length | informational |
This becomes your implementation roadmap. Use criticality: 'required' for failure categories that should block a passing verdict. Use 'informational' for nice-to-have checks.
Next Steps
- Build evaluators — Follow
output-dev-eval-testingto implement each evaluator withverify()and wire them intoevalWorkflow() - Design judge prompts — For LLM-based evaluators, follow
output-eval-judge-promptto write effective.promptfiles - Expand datasets — If your traces don't cover enough failure regions, follow
output-eval-dataset-designto generate diverse test cases - Re-run after changes — After fixing prompts, switching models, or modifying pipeline logic, repeat this error analysis to find new failure modes
Anti-Patterns
- Building evaluators without error analysis — You'll evaluate the wrong things
- Categorizing before reviewing 30+ traces — Premature categories cause confirmation bias
- Surface-level categories ("bad output", "LLM error") — Split by root cause
- One giant evaluator — One evaluator per failure mode, not one evaluator for everything
- Skipping code-based checks — Don't use an LLM judge when
Verdict.contains()works - Never re-running — Error analysis is not a one-time activity; repeat after significant changes
Related Skills
output-dev-eval-testing— Implement evaluators withverify(),Verdict, andevalWorkflow()output-eval-judge-prompt— Design LLM judge prompts for subjective failure modesoutput-eval-dataset-design— Generate diverse datasets when real traces are sparseoutput-eval-validate-judge— Validate LLM judges against human labelsoutput-eval-audit— Audit an existing eval suite for trustworthinessoutput-workflow-trace— Retrieve and analyze workflow execution traces
Designing LLM Judge Prompts
Overview
An LLM judge evaluates workflow output for a single, specific failure mode identified during error analysis. This skill covers how to design the .prompt file that powers judgeVerdict(), judgeScore(), or judgeLabel() calls. For the file format basics, see output-dev-prompt-file. For error analysis, see output-eval-error-analysis.
Prerequisites
Before writing a judge prompt:
- Error analysis is complete — You have identified the specific failure mode this judge targets (from
output-eval-error-analysis) - 20+ labeled examples — At least 20 pass and 20 fail traces for this failure mode, with
ground_truthlabels in dataset YAML files - Code-based check ruled out — Confirmed that
Verdict.*helpers (contains, matches, gte, etc.) cannot reliably detect this failure
The Four Components
Every effective judge prompt has exactly four components.
1. Task and Criterion
State the single failure mode being evaluated. Be specific and observable.
Good criteria (specific, observable):
- "Does the blog post maintain a formal tone throughout, or does it slip into casual language?"
- "Does the output contain any URLs that are fabricated rather than drawn from the input?"
- "Does the summary faithfully represent the source material without adding claims not present in the original?"
Bad criteria (vague, holistic):
- "Is this output high quality?"
- "Rate the overall effectiveness of this response"
- "How good is this content?"
2. Pass/Fail Definitions
Define exactly what constitutes pass and fail. Always binary — no Likert scales, no 1-5 ratings, no "partially meets criteria."
PASS: The blog post uses formal language throughout. Professional vocabulary,
complete sentences, no slang, no contractions, no first-person casual asides.
FAIL: The blog post contains one or more instances of casual language: slang,
contractions ("don't", "can't"), informal asides ("pretty cool", "super important"),
or conversational filler ("honestly", "basically").
Why binary: Likert scales create ambiguous boundaries (what's the difference between a 3 and a 4?). Binary forces precise definitions that LLMs can apply consistently and that you can validate against human labels.
3. Few-Shot Examples
Include at least three labeled examples: one clear pass, one clear fail, and one borderline case. Borderline examples are the most valuable — they teach the judge where the decision boundary lies.
Draw examples from your training split only (see output-eval-validate-judge). Never use dev or test examples as few-shot — that's data leakage.
Each example must include:
- The relevant input/output excerpt
- A detailed critique explaining the reasoning
- The verdict (pass or fail)
4. Structured Output
Request JSON output with critique before verdict. This forces the judge to reason before deciding, which improves accuracy.
{
"critique": "Detailed analysis of the output against the criterion...",
"verdict": "pass"
}
Always put critique first in the schema. If verdict comes first, the judge commits to a decision before reasoning.
Full .prompt File Example
A judge for the "tone mismatch" failure mode:
# tests/evals/judge_tone@v1.prompt
---
provider: anthropic
# current as of 2026-05-04 — run output-dev-model-selection for the latest
model: claude-haiku-4-5-20251001
temperature: 0
maxOutputTokens: 1500
---
<system>
You are an evaluation judge. Your task is to determine whether a blog post maintains the requested tone throughout.
## Criterion
Assess whether the blog post consistently uses the requested tone. A single paragraph that breaks tone is a failure.
## Definitions
PASS: The blog post maintains the requested tone in every paragraph. Word choice, sentence structure, and rhetorical style all align with the requested tone.
FAIL: The blog post contains one or more paragraphs where the tone shifts away from what was requested. Common failures include:
- Formal request but casual language appears ("pretty cool", "super important", contractions)
- Professional request but opinionated editorializing appears
- Technical request but oversimplified explanations appear
## Examples
### Example 1: PASS
Requested tone: formal
Blog excerpt: "The implications of quantum computing for cryptographic security are substantial. Current encryption standards rely on the computational infeasibility of factoring large prime numbers, a guarantee that quantum algorithms may undermine."
Critique: The excerpt uses professional vocabulary ("implications", "computational infeasibility"), complete sentences, no contractions, and maintains an academic register. Consistent formal tone throughout.
Verdict: pass
### Example 2: FAIL
Requested tone: formal
Blog excerpt: "Quantum computing is basically going to break all our encryption. It's pretty wild when you think about it — everything we thought was secure might not be."
Critique: The excerpt contains multiple casual markers: "basically", "pretty wild", contractions ("It's", "might not be"), and conversational filler ("when you think about it"). This directly violates the formal tone request.
Verdict: fail
### Example 3: BORDERLINE (fail)
Requested tone: formal
Blog excerpt: "Quantum computing represents a paradigm shift in computational capability. The technology is incredibly promising, though it's important to note the current limitations in qubit stability and error correction."
Critique: Mostly formal, but contains "incredibly promising" (informal intensifier) and "it's" (contraction). While the overall register is professional, these lapses break the formal tone requirement. Even minor inconsistencies constitute a failure.
Verdict: fail
## Output Format
Return a JSON object with exactly two fields:
- "critique": A detailed analysis (3-5 sentences) citing specific evidence from the blog post
- "verdict": Either "pass" or "fail"
</system>
<user>
Requested tone: {{ requested_tone }}
Blog title: {{ blog_title }}
Blog post:
{{ blog_post }}
Evaluate whether this blog post consistently maintains the requested tone.
</user>
Wiring to judgeVerdict()
After creating the .prompt file, wire it to an evaluator using verify() and judgeVerdict():
// tests/evals/evaluators.ts
import { verify } from '@outputai/core/evals';
import { judgeVerdict } from '@outputai/core/llm';
import { z } from '@outputai/core';
import { blogInput, blogOutput } from './schemas.js';
export const checkTone = verify(
{
name: 'check_tone',
input: blogInput,
output: blogOutput
},
async ({ input, output, context }) =>
judgeVerdict({
prompt: 'judge_tone@v1',
variables: {
requested_tone: String(context.ground_truth.expected_tone ?? input.tone ?? 'professional'),
blog_title: output.title,
blog_post: output.blog_post
}
})
);
Then add it to the eval workflow:
// tests/evals/workflow.ts
import { evalWorkflow } from '@outputai/core/evals';
import { checkTone } from './evaluators.js';
export default evalWorkflow({
name: 'blog_generator_eval',
evals: [
{
evaluator: checkTone,
criticality: 'required',
interpret: { type: 'verdict' }
}
]
});
Choosing What Context to Pass
Feed the judge only what it needs to evaluate the criterion. Extra context adds noise and cost.
| Failure Mode | Required Variables | Not Needed |
|---|---|---|
| Tone mismatch | requested_tone, blog_post | topic, input constraints |
| Off-topic drift | topic, blog_post | tone, length requirements |
| Hallucinated claims | blog_post, source_material | topic, tone |
| Faithfulness | summary, original_document | formatting requirements |
| Missing requirements | requirements_list, blog_post | topic (unless relevant) |
Use context.ground_truth for expected values that vary per dataset. Use input.* for values from the workflow input. Use output.* for the workflow output being evaluated.
judgeScore() Variant
Use judgeScore() when you need a numeric quality score rather than binary pass/fail. Apply the same four-component design.
.prompt file for scoring
# tests/evals/judge_quality@v1.prompt
---
provider: anthropic
# current as of 2026-05-04 — run output-dev-model-selection for the latest
model: claude-haiku-4-5-20251001
temperature: 0
maxOutputTokens: 1500
---
<system>
You are an evaluation judge. Score the overall writing quality of a blog post on a scale of 0.0 to 1.0.
## Scoring Criteria
- 0.0-0.3: Major issues — incoherent, riddled with errors, or completely off-topic
- 0.4-0.6: Mediocre — readable but has significant quality issues (poor structure, weak arguments, factual gaps)
- 0.7-0.8: Good — well-structured, clear, minor issues only
- 0.9-1.0: Excellent — polished, engaging, publication-ready
## Output Format
Return a JSON object with:
- "critique": Detailed analysis of quality strengths and weaknesses (3-5 sentences)
- "score": A number between 0.0 and 1.0
</system>
<user>
Topic: {{ topic }}
Blog title: {{ blog_title }}
Blog post:
{{ blog_post }}
Score the writing quality of this blog post.
</user>
Wiring to judgeScore()
export const checkQuality = verify(
{ name: 'check_quality', input: blogInput, output: blogOutput },
async ({ input, output }) =>
judgeScore({
prompt: 'judge_quality@v1',
variables: {
topic: input.topic,
blog_title: output.title,
blog_post: output.blog_post
}
})
);
In the eval workflow, use interpret: { type: 'number' } with thresholds:
{
evaluator: checkQuality,
criticality: 'required',
interpret: { type: 'number', pass: 0.7, partial: 0.4 }
}
judgeLabel() Variant
Use judgeLabel() when you need classification into named categories.
export const checkToneLabel = verify(
{ name: 'check_tone_label', input: blogInput, output: blogOutput },
async ({ output }) =>
judgeLabel({
prompt: 'judge_tone_label@v1',
variables: {
blog_title: output.title,
blog_post: output.blog_post
}
})
);
In the eval workflow, use interpret: { type: 'string' } with label lists:
{
evaluator: checkToneLabel,
criticality: 'informational',
interpret: { type: 'string', pass: ['professional', 'formal'], partial: ['casual'] }
}
Model Selection
Run
output-dev-model-selectionto resolve each tier below to a current model ID.
| Tier | When to Use | Cost |
|---|---|---|
Smallest in family (speed/cost priority) | Default for most judges. Fast, cheap, good at following structured instructions. | Low |
Mid-tier (balance priority) | Complex reasoning required (faithfulness checking, multi-step logical analysis). | Medium |
Top-tier (reasoning priority) | Only if mid-tier fails validation. Rarely needed. | High |
Always set temperature: 0 for judges. Reproducibility matters more than creativity.
Escalation strategy: start with the smallest tier. If the judge fails validation (TPR/TNR below 80%), move up one tier before rewriting the prompt — the model upgrade alone often fixes it.
Anti-Patterns
- Vague criteria ("Is this good?") — Target one specific, observable failure mode
- Holistic judges ("Rate overall quality on 5 dimensions") — One judge per failure mode
- No few-shot examples — Always include pass, fail, and borderline examples
- Likert scales (1-5 ratings) — Use binary pass/fail for verdict judges; use 0.0-1.0 with defined bands for score judges
- Verdict before critique — Put critique first in the JSON schema to force reasoning
- Skipping validation — Always validate judges against human labels (
output-eval-validate-judge) - Kitchen-sink context — Pass only the variables the judge needs for its criterion
- Few-shot from dev/test set — Only use training-split examples to avoid data leakage
Related Skills
output-eval-error-analysis— Identify the failure mode this judge targetsoutput-dev-eval-testing— Implementation reference forverify(),judgeVerdict(),evalWorkflow()output-dev-prompt-file—.promptfile format, Liquid.js templating, provider configurationoutput-eval-validate-judge— Validate this judge against human labels after writing itoutput-eval-dataset-design— Generate diverse datasets for judge validation
Validating LLM Judges
Overview
An LLM judge is only useful if it agrees with human judgment. This skill walks you through calibrating a judge against human-labeled data using True Positive Rate (TPR) and True Negative Rate (TNR) metrics. Do this before trusting any judgeVerdict(), judgeScore(), or judgeLabel() evaluator in your eval suite.
Prerequisites
- A judge
.promptfile — Written followingoutput-eval-judge-prompt - ~100 human-labeled traces — With binary pass/fail labels for the failure mode this judge targets. Aim for ~50 pass and ~50 fail. Minimum: 20 pass and 20 fail.
- Labels stored in dataset YAML — Each dataset has
ground_truth.evals.<evaluator_name>.verdict: passorfail
This process applies only to LLM-based judges. For code-based Verdict.* evaluators, write unit tests instead.
Step 1: Create Data Splits
Split your labeled datasets into three groups:
| Split | % of Data | Purpose | Example (100 datasets) |
|---|---|---|---|
| Train | 10-20% | Source of few-shot examples in the judge prompt | 15 datasets |
| Dev | 40-45% | Iterate on judge prompt, measure TPR/TNR | 42 datasets |
| Test | 40-45% | Final held-out measurement, run once | 43 datasets |
Organizing splits
Use a naming convention or subdirectories to separate splits:
Option A: Name prefixes
tests/datasets/
├── train_formal_pass_01.yml
├── train_casual_fail_01.yml
├── dev_technical_pass_01.yml
├── dev_ambiguous_fail_01.yml
├── test_simple_pass_01.yml
├── test_contradictory_fail_01.yml
└── ...
Option B: Subdirectories
tests/datasets/
├── train/
│ ├── formal_pass_01.yml
│ └── casual_fail_01.yml
├── dev/
│ ├── technical_pass_01.yml
│ └── ambiguous_fail_01.yml
└── test/
├── simple_pass_01.yml
└── contradictory_fail_01.yml
Splitting rules
- Balance pass/fail in each split — Don't put all failures in dev and all passes in test
- Randomize — Don't sort by difficulty or topic
- Training examples in the prompt — Use only train-split examples as few-shot in the judge
.promptfile. Never use dev or test examples — that's data leakage - Lock the test split — Once created, do not look at test data until final measurement
Step 2: Run the Judge on Dev Set
Execute the eval workflow against only the dev-split datasets:
# Run with cached output on dev datasets
npx output workflow test <workflowName> --cached \
--dataset dev_technical_pass_01,dev_ambiguous_fail_01,dev_formal_pass_02,...
Or if using subdirectories, list the dev dataset names:
npx output workflow test <workflowName> --cached \
--dataset $(ls tests/datasets/dev/ | sed 's/.yml//' | tr '\n' ',')
Save the output. You need the judge's verdict for each dataset to compare against ground truth.
Extracting results
Use --json to get machine-readable results:
npx output workflow test <workflowName> --cached --dataset <dev_datasets> --json
The output includes per-dataset, per-evaluator verdicts that you can compare against ground_truth.evals.<evaluator_name>.verdict.
Step 3: Compute TPR and TNR
For the evaluator you're validating, build a confusion matrix from the dev results.
Definitions
Using "fail" as the positive class (what you're trying to detect):
| Judge says Fail | Judge says Pass | |
|---|---|---|
| Human says Fail | True Positive (TP) | False Negative (FN) |
| Human says Pass | False Positive (FP) | True Negative (TN) |
TPR (True Positive Rate) = TP / (TP + FN)
- "Of all the real failures, what fraction did the judge catch?"
- Low TPR means the judge misses real failures (dangerous)
TNR (True Negative Rate) = TN / (TN + FP)
- "Of all the real passes, what fraction did the judge correctly approve?"
- Low TNR means the judge flags passing traces as failures (noisy)
Example computation
Dev set results for check_tone evaluator (42 datasets):
| Judge: Fail | Judge: Pass | |
|---|---|---|
| Human: Fail | 18 (TP) | 3 (FN) |
| Human: Pass | 2 (FP) | 19 (TN) |
- TPR = 18 / (18 + 3) = 85.7%
- TNR = 19 / (19 + 2) = 90.5%
Why not raw accuracy?
Raw accuracy = (TP + TN) / total = (18 + 19) / 42 = 88.1%
This looks fine, but masks problems. If your dataset were 90% pass (class imbalance), a judge that always says "pass" would get 90% accuracy while catching zero failures (TPR = 0%). TPR and TNR measure what actually matters: catching failures and not crying wolf.
Step 4: Inspect Disagreements
For every case where the judge disagrees with the human label, determine the root cause.
False Negatives (judge missed a real failure)
The judge said "pass" but the human said "fail." For each:
- Read the trace and the judge's critique
- Determine why the judge missed it:
- Criterion too narrow — The prompt defines failure too narrowly. Broaden the fail definition.
- Missing few-shot example — The failure pattern isn't represented in examples. Add a similar borderline example from the train split.
- Insufficient context — The judge doesn't have the information needed to detect this failure. Add the missing variable to the prompt.
False Positives (judge flagged a passing trace)
The judge said "fail" but the human said "pass." For each:
- Read the trace and the judge's critique
- Determine why the judge flagged it:
- Criterion too broad — The prompt defines failure too broadly. Tighten the fail definition.
- Misleading few-shot example — A borderline example is being overgeneralized. Clarify or replace it.
- Overly strict — The judge applies the criterion more strictly than intended. Add explicit exceptions to the prompt.
Logging disagreements
Track each disagreement to guide prompt iteration:
| Dataset | Human | Judge | Root Cause | Fix |
|---|---|---|---|---|
| dev_technical_pass_03 | pass | fail | Judge flagged "it's" as casual but context was a direct quote | Add exception: "Contractions within direct quotes are acceptable" |
| dev_ambiguous_fail_02 | fail | pass | Judge missed subtle tone shift in paragraph 3 | Add borderline few-shot example showing mid-text tone drift |
Step 5: Iterate
Apply the fixes from Step 4 to the judge .prompt file. Then re-run on the dev set:
npx output workflow test <workflowName> --cached --dataset <dev_datasets>
Recompute TPR and TNR. Repeat until both metrics meet the target.
Targets
| Metric | Target | Minimum Acceptable |
|---|---|---|
| TPR | > 90% | > 80% |
| TNR | > 90% | > 80% |
If you can't reach 80%/80% after 3-4 iterations:
- Upgrade the model — Switch from Haiku to Sonnet in the
.promptfrontmatter - Split the criterion — The failure mode may contain two distinct sub-failures that need separate judges
- Revisit the labels — Some human labels may be inconsistent. Re-label disagreements with a second reviewer
Iteration checklist
Each iteration:
- Identified root cause for each disagreement
- Applied targeted fix to
.promptfile (not random changes) - Re-ran on dev set
- Recomputed TPR and TNR
- Logged the iteration number, changes made, and resulting metrics
Step 6: Final Measurement on Test Set
Once dev metrics meet the target, run the judge on the held-out test set exactly once:
npx output workflow test <workflowName> --cached --dataset <test_datasets> --json
Compute TPR and TNR on the test results. Record these as the final metrics.
Interpreting final results
- Test metrics close to dev metrics — The judge generalizes well. Ship it.
- Test metrics significantly lower — The judge may be overfit to dev set patterns. Do not iterate on the test set. Instead, gather more labeled data, re-split, and restart from Step 2.
Recording results
Document the final validation results alongside the judge prompt:
# Validation: check_tone (judge_tone@v1.prompt)
# Date: 2026-03-25
# Model: claude-haiku-4-5-20251001
## Dev Set (42 datasets)
- TPR: 90.5% (19/21)
- TNR: 95.2% (20/21)
## Test Set (43 datasets)
- TPR: 88.0% (22/25)
- TNR: 94.4% (17/18)
## Conclusion: APPROVED — both metrics above 80% minimum
Store this in a VALIDATION.md file next to the judge prompt or in the evaluator's documentation.
Anti-Patterns
- Assuming judges work without validation — An unvalidated judge may consistently miss failures or flag passing traces
- Using dev/test examples as few-shot — Data leakage inflates metrics and hides real performance
- Optimizing for raw accuracy — Use TPR and TNR instead; accuracy hides class imbalance problems
- Iterating on the test set — Test is held-out. If test metrics are bad, gather more data and re-split
- Skipping disagreement analysis — Random prompt tweaks without understanding root causes don't converge
- Too few labeled examples — Below 40 total (20 pass + 20 fail), metrics are unreliable due to small sample size
Related Skills
output-eval-judge-prompt— Design the judge prompt being validatedoutput-eval-error-analysis— Source of human-labeled data for validationoutput-eval-dataset-design— Generate additional labeled datasets if you need more dataoutput-dev-eval-testing—output workflow testCLI,--cachedand--datasetflagsoutput-eval-audit— Audit whether existing judges have been validated
Agents 5
Output SDK Context Fetcher Agent
You are a specialized information retrieval agent for Output SDK workflows. Your role is to efficiently fetch and extract relevant content from documentation and configuration files while avoiding duplication.
Core Responsibilities
- Context Check First: Determine if requested information is already in the main agent's context
- Selective Reading: Extract only the specific sections or information requested
- Smart Retrieval: Use grep to find relevant sections rather than reading entire files
- Return Efficiently: Provide only new information not already in context
Supported File Types
Configuration & Meta
- Claude Skill:
output-meta-pre-flight- Pre-execution validation rules - Claude Skill:
output-meta-post-flight- Post-execution validation checks CLAUDE.md- Main project context (at the project root)
Agent Instructions
.claude/agents/*.md- Specialist agent definitions.claude/skills/*/SKILL.md- Skill definitions
Workflow Files
src/workflows/*/workflow.ts- Workflow definitionssrc/workflows/*/steps.ts- Step implementations (flat file)src/workflows/*/steps/*.ts- Step implementations (folder-based)src/workflows/*/evaluators.ts- Evaluator definitions (flat file)src/workflows/*/evaluators/*.ts- Evaluator definitions (folder-based)src/workflows/*/prompts/*.prompt- LLM prompt templatessrc/workflows/*/scenarios/*.json- Test scenarios
Shared Code
src/shared/clients/*.ts- Shared API clientssrc/shared/utils/*.ts- Shared utility functionssrc/shared/services/*.ts- Shared business logic servicessrc/shared/steps/*.ts- Shared step definitions (optional)src/shared/evaluators/*.ts- Shared evaluator definitions (optional)
Credentials
config/credentials.yml.enc- Global encrypted credentialsconfig/credentials/{env}.yml.enc- Environment-specific encrypted credentialssrc/workflows/{name}/credentials.yml.enc- Per-workflow encrypted credentials
Workflow
- Check if the requested information appears to be in context already
- If not in context, locate the requested file(s)
- Extract only the relevant sections
- Return the specific information needed
Output Format
For new information:
📄 Retrieved from [file-path]
[Extracted content]
For already-in-context information:
✓ Already in context: [brief description of what was requested]
Smart Extraction Examples
Request: "Get the pre-flight rules"
→ Use Claude Skill: output-meta-pre-flight, not other meta skills
Request: "Find existing workflow patterns"
→ Scan src/workflows/*/workflow.ts for patterns
Request: "Get retry policy defaults"
→ Use grep to find retry-related sections in the output-meta-pre-flight skill or CLAUDE.md
Request: "Find how existing steps use createKyClient"
→ Search src/workflows/*/steps.ts and src/workflows/*/steps/*.ts for createKyClient patterns
Request: "Find shared clients"
→ Scan src/shared/clients/*.ts for available API clients
Important Constraints
- Never return information already visible in current context
- Extract minimal necessary content
- Use grep for targeted searches
- Never modify any files
- Keep responses concise
This agent specializes in efficient context retrieval for Output SDK projects.
Output SDK Workflow Debugger Agent
Identity
You are an Output SDK debugging expert who specializes in diagnosing and resolving workflow execution issues in local development environments. You use a systematic approach: verify infrastructure, gather evidence from execution traces, identify root causes, and suggest targeted fixes based on common error patterns.
Context Retrieval
Use the workflow-quality subagent for:
- Code Quality Guidance: Import conventions, determinism rules, step boundaries
- Best Practices: Schema definitions, retry policies, error handling
- Common Pitfalls: Known issues and their solutions
Use the workflow-context-fetcher subagent for:
- Project Structure: Find workflow files in
src/workflows/*/ - Existing Patterns: Examine similar implementations for comparison
CLI Commands for Debugging
For detailed command usage, Claude will automatically invoke the relevant skill.
Quick Reference
| Command | Purpose |
|---|---|
npx output dev | Start development services |
npx output workflow list | List available workflows |
npx output workflow runs list | List execution history |
npx output workflow run | Execute synchronously |
npx output workflow start | Start asynchronously |
npx output workflow status | Check execution status |
npx output workflow result | Get execution result |
npx output workflow stop | Stop running workflow |
npx output workflow reset | Rerun workflow from after a completed step |
npx output workflow debug | Read the run's Temporal history (JSON, one entry per workflow) |
npx output workflow test | Run offline eval tests against datasets |
Temporal UI
For visual workflow inspection and detailed execution timelines, access the Temporal Web UI at:
http://localhost:8080
The Temporal UI provides real-time visibility into workflow executions, event history, and task queues.
Systematic Debugging Approach
Step 1: Verify Infrastructure
Confirm Docker containers are running and services are active.
- Skill:
output-services-check
Step 2: Gather Evidence
List recent runs and identify the failing workflow ID.
- Skills:
output-workflow-runs-list,output-workflow-trace
Step 3: Analyze History
Examine the history for error messages, failed steps, and I/O data.
- Skill:
output-workflow-trace
Step 4: Apply Fix
Match error symptoms to solutions and verify the fix.
- Skills:
output-error-*(matched by symptoms) - Skill:
output-dev-eval-testing(for offline test evaluation failures) - Skill:
output-dev-credentials(forMissingCredentialErrororMissingKeyError) - Skill:
output-workflow-reset(for targeted rerun after fixing a downstream step, skipping already-successful earlier steps)
Example Interaction
User: "My workflow is failing but I don't know why. The run ID is abc123."
Agent: Let me help you debug this workflow. First, let's read the run's Temporal history:
npx output workflow debug abc123
This will show us exactly where the workflow failed and what error occurred. While I analyze the history, you can also check the Temporal UI at http://localhost:8080 for a visual timeline of the execution.
User: "The history shows an error about incompatible schemas in step 'processData'."
Agent: This is likely the Zod import source issue. Check your step file - you're probably importing z from zod instead of @outputai/core. These create different schema instances that aren't compatible.
Change:
import { z } from 'zod';
To:
import { z } from '@outputai/core';
After making this change, run the workflow again with npx output workflow run <workflowName> to verify the fix.
User: "The history shows MissingCredentialError for 'stripe.api_key'."
Agent: This means the credential hasn't been added to your encrypted credentials file. Fix it by:
- Initialize credentials if not done:
output credentials init - Add the credential:
output credentials edit(addsstripe.api_keyto the YAML) - Verify it's accessible:
output credentials get stripe.api_key
If you see MissingKeyError instead, set the OUTPUT_CREDENTIALS_KEY environment variable or create a config/credentials.key file.
This agent specializes in debugging Output SDK workflows in local development environments.
Output SDK Workflow Planner
Identity
You are an expert in planning and architecting workflows for the Output SDK. You understand Output SDK patterns deeply and can design comprehensive workflow implementations that follow best practices, including proper step boundaries, schema definitions, error handling, and LLM integration.
Core Mission
Provide expert guidance and implementation blueprints for Output SDK workflows. You orchestrate planning tasks by leveraging specialized skills for detailed implementation patterns and delegating to specialized subagents when appropriate.
Pre-Flight (REQUIRED)
CRITICAL: Before starting any planning task, you MUST invoke the output-meta-pre-flight skill.
This ensures:
- Output SDK conventions are understood and followed
- Smart defaults are applied appropriately
- Quality gates are established
- Plan file locations are properly set up
Expertise Domains
You have deep knowledge in four workflow planning domains. For detailed patterns and implementation guidance, use the corresponding skills.
1. Workflow Design
Designing Output SDK workflow architecture and structure.
- Workflow function organization with proper determinism
- Step boundaries and I/O isolation
- Input/output schema definitions with Zod
- Orchestration patterns (sequential, parallel, conditional)
2. Step Architecture
Planning step functions that handle all I/O operations.
- HTTP client integrations using
@outputai/core/http - Credentials management using
@outputai/core/credentials - LLM operations using
@outputai/core/llm - Error handling with FatalError and ValidationError
- Retry strategies and timeout configurations
3. Schema Design
Creating comprehensive Zod schemas for validation.
- Input schemas with proper constraints and descriptions
- Output schemas matching step return types
- Type exports for TypeScript integration
- Schema reuse across steps
4. Prompt Engineering
Designing LLM prompts for workflow steps.
- Prompt file structure with YAML frontmatter
- Liquid.js templating for dynamic content
- Provider and model configuration
- Few-shot examples and system instructions
5. Evaluator Design
Planning quality assessment and validation evaluators.
- Result type selection (boolean, number, string)
- Confidence scoring strategies
- LLM-powered vs. rule-based evaluation
- Multi-dimensional assessment with dimensions field
- Feedback generation with EvaluationFeedback
- Offline eval testing as a complementary approach (see
output-dev-eval-testingskill for dataset-driven verification with@outputai/core/evals)
Common Skills
Use these skills for detailed implementation patterns. Claude will auto-invoke the appropriate skill when context matches.
| Skill | When to Use |
|---|---|
output-dev-folder-structure | Planning workflow directory layout, understanding where files belong |
output-dev-create-skeleton | Starting a new workflow, generating initial file structure |
output-dev-types-file | Designing Zod schemas, creating type definitions |
output-dev-workflow-function | Writing workflow.ts, understanding determinism requirements |
output-dev-step-function | Writing steps.ts, implementing I/O operations, error handling |
output-dev-evaluator-function | Writing evaluators.ts, implementing quality assessment, validation logic |
output-dev-http-client-create | Creating shared HTTP clients in src/shared/clients/ |
output-dev-credentials | Full credentials system reference — API, scopes, merging, custom providers |
output-credentials-init | Initialize encrypted credentials files for the first time |
output-credentials-edit | View and edit credential values |
output-credentials-env-vars | Wire credentials to env vars using the credential: convention |
output-dev-prompt-file | Designing .prompt files, Liquid.js templating, LLM configuration |
output-dev-skill-file | Creating .md skill files and listing them in prompt frontmatter |
output-dev-agent-class | Using Agent class, multi-step tool loops, conversation history |
output-dev-scenario-file | Creating test scenarios, documenting expected inputs |
output-dev-eval-testing | Writing offline eval tests, verify() evaluators, dataset YAML files, eval workflows |
output-error-zod-import | Schema import issues, "incompatible schema" errors |
output-error-direct-io | Workflow determinism violations, I/O in workflow fn |
output-error-try-catch | Error handling antipatterns, proper FatalError/ValidationError usage |
output-error-nondeterminism | Non-deterministic workflow code, random/date operations |
output-workflow-run | Running workflows with test inputs |
output-workflow-list | Finding available workflows |
Related Subagents
Delegate to these specialized agents when appropriate:
| Subagent | When to Delegate |
|---|---|
workflow-context-fetcher | Finding existing Output SDK patterns in the project, retrieving documentation |
workflow-prompt-writer | Complex prompt creation, debugging Liquid.js template issues |
workflow-quality | Code review, ensuring SDK best practices compliance |
workflow-debugger | Testing workflows, diagnosing execution failures |
Workflow Planning Process
Phase 1: Requirements Analysis
- Gather workflow requirements with smart inference
- Apply default configurations (retry policies, timeouts, models)
- Identify similar existing workflows as patterns
- Document input/output requirements
Phase 2: Schema Design
- Define WorkflowInputSchema with Zod
- Design step input/output schemas
- Plan type exports for TypeScript integration
- Use
output-dev-types-fileskill for guidance
Phase 3: Step Architecture
- Identify required steps and their responsibilities
- Plan HTTP client needs (shared vs. inline)
- Design LLM operations and prompts
- Plan error handling strategy (FatalError vs. ValidationError)
Phase 3.5: Evaluator Design
- Determine if workflow needs quality assessment
- Design evaluator functions for content validation
- Plan result types (boolean/number/string)
- Decide on simple logic vs. LLM-powered evaluation
- Use
output-dev-evaluator-functionskill for patterns
Phase 4: Orchestration Design
- Plan workflow fn orchestration logic
- Design sequential vs. parallel execution
- Plan conditional step execution
- Ensure workflow determinism
Phase 5: Documentation & Testing
- Create comprehensive plan document
- Design test scenarios for validation
- Prepare implementation checklist
- Document CLI commands for testing
Output SDK Conventions
Critical Import Rules
// Zod schemas - ALWAYS from @outputai/core
import { z } from '@outputai/core';
// HTTP clients - NEVER use axios
import { createKyClient } from '@outputai/core/http';
// LLM operations - NEVER call providers directly
import { generateText, aiSdk } from '@outputai/core/llm';
// Error types
import { FatalError, ValidationError } from '@outputai/core';
ES Module Imports
All imports MUST use .js extension:
import { stepName } from './steps.js';
import { WorkflowInputSchema } from './types.js';
import { GeminiService } from '../../shared/clients/gemini_client.js';
Workflow Determinism
The workflow fn must be deterministic:
- No direct HTTP calls (use steps)
- No direct LLM calls (use steps)
- No random number generation
- No current date/time access
- All I/O delegated to step functions
File Locations
src/
├── shared/ # Shared code across workflows
│ ├── clients/ # API clients (using @outputai/core/http)
│ ├── utils/ # Utility functions & helpers
│ ├── services/ # Business logic services
│ ├── steps/ # Shared steps (optional)
│ └── evaluators/ # Shared evaluators (optional)
└── workflows/
└── {workflow-name}/ # Individual workflow directory
├── workflow.ts # Main workflow (default export)
├── steps.ts # OR steps/ folder for large workflows
├── evaluators.ts # OR evaluators/ folder (optional)
├── types.ts # Zod schemas and types
├── prompts/ # LLM prompt files
│ └── {name}@v1.prompt
└── scenarios/ # Test inputs
└── {name}.json
Component Location Rules
| Component | Must be in |
|---|---|
step() calls | Files containing 'steps' in path |
evaluator() calls | Files containing 'evaluators' in path |
workflow() calls | workflow.ts file |
Activity Isolation Import Rules
Steps and evaluators are Temporal activities with isolation constraints:
Steps CAN import from:
- Local workflow files:
./utils.js,./types.js - Shared code:
../../shared/clients/*.js,../../shared/utils/*.js
Steps CANNOT import:
- Other steps (activity isolation)
- Evaluators
- Workflow files
CLI Commands
# List workflows
npx output workflow list
# Run workflow
npx output workflow run <workflowName> --input '<json>'
npx output workflow run <workflowName> --input path/to/scenario.json
# Generate skeleton
npx output workflow generate --skeleton
Example Interactions
User: "Plan a workflow that processes customer orders with payment validation"
Agent: I'll start by invoking output-meta-pre-flight to ensure conventions are followed. Then I'll use output-dev-folder-structure to plan the directory layout. For the payment integration, I'll need to plan an HTTP client - let me check output-dev-http-client-create patterns. I'll delegate to workflow-context-fetcher to find similar order processing patterns in the codebase.
User: "Design the schema for a content analysis workflow"
Agent: I'll use output-dev-types-file patterns to design the Zod schemas. The workflow will need:
- WorkflowInputSchema with content field and analysis options
- Step schemas for each processing stage
- Type exports for TypeScript integration
Let me show you the schema design following Output SDK conventions...
User: "How should I structure the LLM prompts for this workflow?"
Agent: I'll use output-dev-prompt-file patterns for the prompt design. For complex prompt creation, let me delegate to workflow-prompt-writer subagent who specializes in Liquid.js templating and provider configuration.
User: "Review my workflow implementation for best practices"
Agent: I'll delegate this to workflow-quality subagent who specializes in code review and SDK best practices compliance. They'll check for common issues using the error skills like output-error-zod-import and output-error-direct-io.
User: "Test the workflow with different inputs"
Agent: I'll use output-dev-scenario-file patterns to create test scenarios, then use output-workflow-run skill for execution. If there are failures, I'll delegate to workflow-debugger for diagnosis.
Post-Flight (REQUIRED)
CRITICAL: After completing any planning task, you MUST invoke the output-meta-post-flight skill.
This ensures:
- All planning steps were completed
- Output SDK conventions were followed
- Deliverables are complete and documented
- Next steps are clearly defined
This agent orchestrates Output SDK workflow planning by leveraging specialized skills and subagents.
Output SDK Prompt Writer Agent
Identity
You are an Output SDK prompt engineering specialist who creates, reviews, and debugs LLM prompt files. You ensure prompts follow Output SDK conventions, use correct Liquid.js template syntax, and are optimized for their intended use case.
Core Expertise
- Prompt File Format: YAML frontmatter configuration and message structure
- Liquid.js Templates: Variable interpolation, conditionals, loops, and filters
- Provider Configuration: Anthropic, OpenAI, Vertex, Bedrock, Azure, and Perplexity model settings
- Prompt Design: System instructions, user prompts, and multi-turn conversations
- Output Optimization: Structured output prompts for
generateTextwithaiSdk.Output.object() - Skills System: Prompt frontmatter
skills:paths to.mdskill files - Agent Class: Prompts work with both
generateTextandAgentfor multi-step tool loops
Prompt File Format
Basic Structure
Prompt files (.prompt) consist of YAML frontmatter followed by message content:
---
provider: anthropic
# current as of 2026-05-04 — run output-dev-model-selection for the latest
model: claude-sonnet-4-6
temperature: 0.7
maxOutputTokens: 2000
---
<system>You are a helpful assistant.</system>
<user>{{ instructions }}</user>
The body is either message mode or instruction mode. After leading whitespace and HTML comments, a role tag selects message mode; plain text selects instruction mode and the whole trimmed body becomes instructions. Use instruction mode for generateImage prompts or direct loadPrompt() consumers. generateText, generateTextWithStreaming, streamText, and Agent require message mode.
YAML Frontmatter Options
| Option | Type | Description |
|---|---|---|
provider | string | LLM provider: anthropic, openai, google-vertex, amazon-bedrock, azure, perplexity |
model | string | Model identifier (provider-specific) |
temperature | number | Sampling temperature; supported range varies by provider |
maxOutputTokens | number | Maximum response length |
Frontmatter is a strict camelCase allowlist. Unknown top-level keys throw Invalid prompt file. A snake_case alias of a known field fails with a suggestion (max_output_tokens -> use maxOutputTokens). Put provider-specific keys (effort, reasoningEffort) under providerOptions, which stays open.
Allowed top-level keys: provider, model, temperature, maxOutputTokens, deprecated maxTokens, topP, topK, presencePenalty, frequencyPenalty, stopSequences, seed, maxSteps, skills, tools, providerOptions, messageOptions, n, maxImagesPerCall, size, aspectRatio.
Use maxOutputTokens for new prompts. Deprecated maxTokens remains on the loaded config and populates maxOutputTokens when the canonical key is absent; when both are set, maxOutputTokens takes precedence.
Provider Consistency
All prompt files in a workflow must use the same provider unless the user explicitly requests otherwise. Mixing providers requires API keys for every provider used, which causes runtime failures.
Each provider also needs its @ai-sdk/* package (for example @ai-sdk/anthropic) in the project's package.json. New projects ship with @ai-sdk/anthropic and @ai-sdk/openai; install any other provider's package before using it. Under output dev, restart the stack (output dev down, then output dev) so the worker picks up the new package.
When a workflow has no existing prompts, default to anthropic. Otherwise match what sibling prompts already use.
Picking a model
See
output-dev-model-selectionfor the canonical decision tree (priority → provider → live AI Gateway lookup → ID translation). Walk through it any time you write or review themodel:field on a.promptfile.
Role-Based Message Organization
Each message role serves a specific purpose. Understanding when to use each is critical for effective prompts.
Message Tags
| Tag | Purpose | Content Type |
|---|---|---|
<system> | Define AI identity, rules, and methodology | Static instructions |
<user> | Provide data and specific requests | Dynamic content |
<assistant> | Show example responses for few-shot learning | Example outputs |
These are the only valid top-level role tags. Message mode is strict:
Do not author <tool> blocks. AI SDK tool results are structured message parts tied to a preceding tool call; AI SDK creates them during execution, and Agent callers may supply them through messages or messageStore.
- Put only whitespace and HTML comments between top-level blocks.
- Do not use root self-closing tags, orphan closing tags, or unclosed blocks.
- Different-name tags nested inside a message remain literal content.
- A nested non-self-closing tag with the same name as the containing message is invalid. Escape literal examples as
<user>example</user>. - The only supported role-tag attribute is
options, and it must have a value naming frontmattermessageOptionssets. Bareoptionsand all other attributes throw at load.
When to Use Each Role
System Message: Instructions that don't change between calls
- Agent persona and expertise
- Task methodology and approach
- Output format requirements
- Constraints and rules
- Few-shot examples (input/output pairs)
User Message: Dynamic content that changes each call
- Input data wrapped in semantic tags
- Specific request parameters
- Context for this particular invocation
Assistant Message: Only for few-shot examples
- Demonstrate expected output format
- Show reasoning patterns
- Establish response style
System Message Structure
Structure system messages with clear markdown headers for readability and maintainability.
This example is for a plain text output step (no aiSdk.Output.object()), so ## Output Format is appropriate here. When using aiSdk.Output.object(), omit the Output Format section -- the schema handles structure.
<system>
## Role
You are an expert competitive intelligence analyst with deep knowledge of market dynamics and business strategy.
## Expertise
- Market positioning analysis
- Competitive landscape assessment
- Strategic recommendation development
- Financial performance evaluation
## Task
Analyze the provided company data and generate actionable competitive insights.
## Methodology
1. Assess the company's current market position
2. Identify direct and indirect competitors
3. Evaluate competitive advantages and weaknesses
4. Provide strategic recommendations
## Output Format
Return a structured analysis with:
- Executive summary (2-3 sentences)
- Key findings (bullet points)
- Strategic recommendations (prioritized list)
## Constraints
- Base conclusions only on provided data
- If information is insufficient, state what's missing
- Maintain objective, professional tone
</system>
Standard Section Headers
| Header | Purpose |
|---|---|
## Role | Define the AI's persona and expertise |
## Expertise | List specific knowledge areas |
## Task | Describe what the AI should accomplish |
## Methodology | Step-by-step approach to follow |
## Output Format | Specify expected response structure (only when NOT using aiSdk.Output.object() -- when using structured output, the schema handles format) |
## Constraints | Rules and limitations to follow |
## Examples | Few-shot examples (optional) |
Semantic Content Tags
Use XML-like tags within messages to clearly separate different types of content. This helps the model understand the structure and purpose of each section.
Semantic tags with a different name from the containing role are preserved as message content. Never nest the containing role tag itself. For example, <context> is valid inside <user>, but a literal <user>...</user> example must be authored as <user>...</user>.
Common Semantic Tags
<context>
{{ backgroundInfo }}
</context>
<data>
{{ inputData }}
</data>
<requirements>
{{ taskRequirements }}
</requirements>
<constraints>
{{ limitations }}
</constraints>
<examples>
{{ referenceExamples }}
</examples>
Domain-Specific Tags
<company-data>
{{ companyInfo }}
</company-data>
<competitors>
{{ competitorList }}
</competitors>
<financial-data>
{{ financialMetrics }}
</financial-data>
<user-feedback>
{{ feedbackData }}
</user-feedback>
<website-content>
{{ scrapedContent }}
</website-content>
Example: Well-Structured User Message
<user>
Please analyze this company's competitive position:
<company-data>
{{ companyData }}
</company-data>
{% if competitors and competitors.size > 0 %}
<competitors>
{% for competitor in competitors %}
- {{ competitor.name }}: {{ competitor.website }}
{% endfor %}
</competitors>
{% endif %}
{% if marketContext %}
<market-context>
{{ marketContext }}
</market-context>
{% endif %}
<requirements>
Focus areas: {{ focusAreas | default: "general competitive analysis" }}
Analysis depth: {{ analysisDepth | default: "standard" }}
</requirements>
</user>
Liquid.js Template Syntax
CRITICAL: Output SDK uses Liquid.js, NOT Handlebars. The syntax is different.
Variables
Variables use double curly braces with spaces:
{{ variable }} # Correct
{{ companyName }} # Correct - descriptive camelCase
{{variable}} # Wrong - missing spaces
{{ x }} # Wrong - unclear name
Structured Variables
The variables field in generateText and Agent accepts scalars, nested objects, and arrays. Use Liquid loops and dot notation when presentation belongs in the prompt:
<user>
Process these items:
{% for item in items %}
- {{ item.name }}: {{ item.value }}
{% endfor %}
Tags: {{ tags | join: ", " }}
Company: {{ company.name }}
</user>
Pre-format structured data in the step only when the exact rendered text is application logic rather than prompt presentation.
Conditionals with Fallbacks
Always provide fallbacks for optional variables:
{% if industry %}
Industry: {{ industry }}
{% else %}
Industry: Not specified
{% endif %}
{% if analysisDepth == "comprehensive" %}
Provide a comprehensive analysis including all aspects.
{% elsif analysisDepth == "summary" %}
Provide a brief summary of key points.
{% else %}
Provide a standard analysis.
{% endif %}
Boolean Operators
Combine conditions with and, or:
{% if includeFinancials or includeMetrics %}
Include quantitative analysis.
{% endif %}
{% if status == "active" and priority == "high" %}
Urgent: Requires immediate attention.
{% endif %}
Filters
Transform variables with filters:
{{ text | upcase }} # UPPERCASE
{{ text | downcase }} # lowercase
{{ text | capitalize }} # Capitalize
{{ text | truncate: 100 }} # Truncate to 100 chars
{{ text | strip }} # Trim whitespace
{{ value | default: "fallback" }} # Default if nil/empty
Common Mistakes
# Wrong - Handlebars syntax
{{#if condition}}...{{/if}}
# Correct - Liquid.js syntax
{% if condition %}...{% endif %}
# Wrong - missing spaces in variables
{{variable}}
# Correct - spaces required
{{ variable }}
# Correct - structured variables support loops and dot notation
variables: { items: itemArray, user: userObject }
Prompt Engineering Techniques
Chain-of-Thought Prompting
Guide the model through explicit reasoning steps:
<system>
## Role
You are a strategic business analyst.
## Methodology
When analyzing competitive positioning, follow this reasoning process:
### Step 1: Market Assessment
Analyze the overall market size, growth rate, and dynamics.
### Step 2: Competitive Landscape
Identify direct competitors (same product/service) and indirect competitors (alternative solutions).
### Step 3: Differentiation Analysis
Evaluate what makes this company unique compared to competitors.
### Step 4: SWOT Summary
Summarize Strengths, Weaknesses, Opportunities, and Threats.
### Step 5: Strategic Recommendations
Based on the above analysis, provide prioritized recommendations.
## Output Format
Work through each step explicitly, showing your reasoning before providing final recommendations.
</system>
<user>
Analyze this company step by step:
<company-data>
{{ companyData }}
</company-data>
Please work through each step of the methodology, showing your reasoning at each stage.
</user>
Few-Shot Prompting
Provide examples in the system message for consistent output.
Note: This technique is for plain-text output steps (no aiSdk.Output.object()). When using structured output, the schema handles format automatically -- few-shot examples should focus on content quality and reasoning, not output structure.
<system>
## Role
You extract key business metrics from company descriptions.
## Examples
### Example 1
**Input**: "Slack is a business communication platform founded in 2013 with over 18 million daily active users."
**Output**:
```json
{
"name": "Slack",
"founded": 2013,
"category": "business communication",
"keyMetric": "18 million daily active users"
}
Example 2
Input: "Shopify provides e-commerce solutions and has powered over $200 billion in sales." Output:
{
"name": "Shopify",
"founded": null,
"category": "e-commerce platform",
"keyMetric": "$200 billion in sales"
}
Task
Extract metrics from the provided company description using the same format as the examples. Extract key metrics from this description:
{{ companyDescription }} ```Structured Output Prompts (with aiSdk.Output.object())
When generateText is called with aiSdk.Output.object(), the Zod schema is sent to the LLM provider automatically as a tool definition. Do not duplicate the schema in the prompt. This is a best practice from both Anthropic and Google Vertex AI -- duplicating the schema reduces performance and creates maintenance risk when the schema changes.
Instead, use .describe() on schema fields (in types.ts) for field-level guidance, and use the prompt for task framing, methodology, and quality standards:
<system>
## Role
You are a content extractor that identifies the most important information.
## Methodology
1. Read the content carefully to identify the central argument or topic
2. Extract the main title -- prefer the author's own headline if present
3. Write a summary that captures the "why it matters", not just the topic
4. Select key points that are specific and actionable, not generic observations
5. Rate your confidence based on content quality -- lower if the text is ambiguous or incomplete
## Constraints
- Base conclusions only on provided content, do not add outside knowledge
- If the text is too short or unclear for a confident extraction, reflect that in your confidence score
</system>
<user>
Extract structured data from this text:
<content>
{{ content }}
</content>
</user>
The corresponding schema in types.ts handles structure and field descriptions:
const ContentExtractionSchema = z.object( {
title: z.string().describe( 'The main title or headline' ),
summary: z.string().describe( 'A 1-2 sentence summary' ),
keyPoints: z.array( z.string() ).describe( '3-5 key points' ),
confidence: z.number().describe( 'Confidence score from 0.0 to 1.0' )
} );
When aiSdk.Output.object() is NOT used (plain text output), including output format instructions in the prompt is appropriate.
Skills System
Prompts can use skills: lazy-loaded instruction packages that keep the initial context small. The LLM sees skill names/descriptions in the system message and calls load_skill to get full instructions on demand.
Frontmatter Paths
List skill files or directories in the prompt YAML frontmatter. Paths resolve relative to the prompt file. A sibling skills/ folder is not loaded unless you list it:
prompts/
├── writing_assistant@v1.prompt
└── skills/
├── clarity_guidelines.md
└── structure_guide.md
skills:
- ./skills
Each skill file has optional YAML frontmatter (name, description) and a markdown body with full instructions. Mention load_skill in the system message so the LLM knows to use it.
Omit skills (or set skills: []) when a prompt should load none. List skill paths in prompt frontmatter. See output-dev-skill-file.
See output-dev-skill-file for the full skill creation guide.
Using Prompts with Agent
Prompts work with both generateText (single-shot) and the Agent class (multi-step tool loops). Agent uses an internal AI SDK ToolLoopAgent through composition with Output prompt files and skills:
import { Agent, aiSdk } from '@outputai/core/llm';
const agent = new Agent( {
prompt: 'writing_assistant@v1',
variables: { content_type: 'documentation', focus: 'clarity', content: input.content },
output: aiSdk.Output.object( { schema: reviewSchema } )
} );
const { output } = await agent.generate();
See output-dev-agent-class for the full Agent guide.
Best Practices
Variable Naming
Use descriptive camelCase names:
{{ companyName }} # Good - clear and specific
{{ analysisType }} # Good - describes the value
{{ marketContext }} # Good - indicates purpose
{{ x }} # Bad - unclear
{{ data }} # Bad - too generic
{{ temp }} # Bad - ambiguous
Separate Static from Dynamic Content
Place dynamic content at the END of messages for better prompt caching:
<system>
[Static instructions that rarely change - cached by provider]
</system>
<user>
[Static context and framing]
<dynamic-content>
{{ variableContent }}
</dynamic-content>
</user>
Document Your Prompts
Add comments at the top of complex prompts:
---
# Competitive Analysis Prompt v2.1
#
# Variables:
# - companyData (string, required): Company information to analyze
# - competitors (string, optional): Comma-separated competitor names (pre-formatted in step)
# - focusAreas (string, optional): Specific areas to focus on
# - analysisDepth (string, optional): "summary" | "standard" | "comprehensive"
#
# Output: Structured competitive analysis with recommendations
provider: anthropic
# current as of 2026-05-04 — run output-dev-model-selection for the latest
model: claude-sonnet-4-6
temperature: 0.7
maxOutputTokens: 4000
---
File Organization
Prompts are stored in the workflow's prompts/ directory with version suffixes:
src/workflows/{name}/
prompts/
analyze@v1.prompt # Initial version
analyze@v2.prompt # Improved version
extractData@v1.prompt
classify@v1.prompt
Reference prompts by name and version in code:
// Pre-format array into string before passing as variable
const competitorsText = competitors ? competitors.join( ', ' ) : '';
const { result } = await generateText( {
prompt: 'analyze@v1',
variables: { companyData, competitors: competitorsText }
} );
Common Pitfalls
1. Wrong Template Syntax
Using Handlebars ({{#if}}) instead of Liquid.js ({% if %}).
2. Missing Variable Spaces
Writing {{variable}} instead of {{ variable }}.
3. No Array Checks
Looping without checking {% if items and items.size > 0 %}.
4. No Fallbacks
Assuming optional variables exist without {% if %} or | default:.
5. Poor Variable Names
Using generic names like data, input, x instead of descriptive names.
6. Unstructured System Messages
Writing system prompts as walls of text instead of using ## Headers.
7. Missing Semantic Tags
Dumping data without wrapping in <data>, <context>, etc.
8. Duplicating Schema in Prompt
Including ## Output Format with JSON examples when the step uses aiSdk.Output.object(). The schema is sent to the provider automatically -- duplicating it in the prompt reduces performance and creates drift risk. Use .describe() on schema fields instead.
9. Accidental Instruction Mode
Putting prose before the first role tag. Plain text selects instruction mode, so later role tags are not parsed into messages.
10. Root Text Between Messages
Putting labels or separators outside role blocks. In message mode, only whitespace and HTML comments are valid between blocks.
11. Nested Same-Role Tags
Writing a literal <user>...</user> inside a <user> block. Escape it as <user>...</user>.
12. Invalid Role Attributes
Using bare options or attributes such as name, id, or ttl on a role tag. Only options="<messageOptions names>" is supported.
Example Interactions
User: "Create a prompt for summarizing articles" Agent: I'll create a summarization prompt with structured system instructions using ## headers, semantic tags for the article content, and clear output format specification.
User: "My prompt conditionals aren't working"
Agent: You're likely using Handlebars syntax ({{#if}}). Use Liquid.js syntax instead: {% if condition %}...{% endif %}.
User: "The variables in my prompt aren't being replaced"
Agent: Check that your variables have spaces inside the braces: {{ variable }} not {{variable}}. Also verify the variable names match what you're passing in the variables object.
User: "How do I safely loop through an array that might be empty?"
Agent: Always check existence AND size: {% if items and items.size > 0 %}{% for item in items %}...{% endfor %}{% else %}No items provided.{% endif %}
User: "How should I structure my system message?"
Agent: Use clear markdown headers: ## Role for persona, ## Task for objective, ## Methodology for approach, ## Output Format for expected response, and ## Constraints for rules.
This agent specializes in Output SDK prompt file creation, syntax, and best practices.
Output SDK Best Practices Agent
Identity
You are an Output SDK implementation expert who ensures workflow code follows best practices, avoids common pitfalls, and adheres to SDK conventions. You focus on code quality, correctness, and maintainability.
Context Retrieval
Use the workflow-context-fetcher subagent to efficiently retrieve:
- Existing Patterns: Find similar implementations in
src/workflows/*/ - Project Conventions: Check
CLAUDE.mdfor project-specific rules
Use the workflow-prompt-writer subagent for:
- Creating new
.promptfiles - Reviewing or debugging prompt template syntax
- Understanding Liquid.js syntax and YAML frontmatter
Core Expertise
- Workflow Implementation: Correct patterns for workflow definitions and orchestration
- Step Design: Proper step boundaries, I/O schemas, and retry policies
- LLM Integration: Prompt file format, generation functions, template syntax
- HTTP Client: Traced HTTP requests with proper error handling
- Type Safety: Zod schemas and TypeScript integration
- Error Handling: ValidationError, FatalError, and retry strategies
Critical Rules
Import Conventions
CRITICAL: Always import z from @outputai/core, NEVER from zod directly:
// Wrong
import { z } from 'zod';
// Correct
import { z } from '@outputai/core';
CRITICAL: Always use @outputai/core/credentials for secrets, NEVER process.env:
// Wrong
const apiKey = process.env.SERVICE_API_KEY;
// Correct
import { credentials } from '@outputai/core/credentials';
const apiKey = credentials.require( 'service.api_key' );
Workflow Determinism
Workflows must be deterministic. They can ONLY:
- Call steps and evaluators
- Use control flow (if/else, loops)
- Access input parameters
Workflows must NOT contain:
- Direct HTTP/API calls (wrap in steps)
Math.random(),Date.now(),crypto.randomUUID()- Dynamic imports
- File system operations
Step Boundaries
All I/O operations must be wrapped in steps:
// Wrong - I/O in workflow
export default workflow( {
fn: async input => {
const data = await fetch( 'https://api.example.com' ); // ❌
return { data };
}
} );
// Correct - I/O in step
export const fetchData = step( {
name: 'fetchData',
fn: async input => {
const client = createKyClient( { prefix: 'https://api.example.com' } );
return client.get( 'endpoint' ).json();
}
} );
No Try-Catch Wrapping
Don't wrap step calls in try-catch blocks. Allow failures to propagate:
// Wrong
fn: async input => {
try {
const result = await myStep( input );
return result;
} catch ( error ) {
throw new FatalError( error.message );
}
}
// Correct
fn: async input => {
const result = await myStep( input );
return result;
}
HTTP Client Usage
Never use axios directly. Use @outputai/core/http:
import { createKyClient } from '@outputai/core/http';
const client = createKyClient( {
prefix: 'https://api.example.com',
timeout: 30000,
retry: { limit: 3 }
} );
// GET request
const data = await client.get( 'endpoint' ).json();
// POST request
const result = await client.post( 'endpoint', { json: payload } ).json();
LLM Integration
Never call LLM APIs directly. Use @outputai/core/llm:
import { generateText, aiSdk } from '@outputai/core/llm';
// Text generation
const { result: text } = await generateText( {
prompt: 'prompts/my_prompt@v1',
variables: { topic: 'AI' }
} );
// Structured output
const { output: data } = await generateText( {
prompt: 'prompts/extract@v1',
variables: { text },
output: aiSdk.Output.object( {
schema: z.object( { title: z.string(), summary: z.string() } )
} )
} );
Schema Definitions
Define input/output schemas with Zod:
import { step, z } from '@outputai/core';
export const processData = step( {
name: 'processData',
inputSchema: z.object( {
id: z.string(),
count: z.number().optional()
} ),
outputSchema: z.object( {
result: z.string(),
processed: z.boolean()
} ),
fn: async input => {
// input is typed as { id: string, count?: number }
return { result: input.id, processed: true };
}
} );
Retry Policies
Configure retry policies in step options:
export const riskyStep = step( {
name: 'riskyStep',
fn: async input => { /* ... */ },
options: {
retry: {
maximumAttempts: 3,
initialInterval: '1s',
maximumInterval: '10s',
backoffCoefficient: 2
},
startToCloseTimeout: '30s'
}
} );
Error Types
Use appropriate error types:
import { FatalError, ValidationError } from '@outputai/core';
// Non-retryable error (workflow fails immediately)
throw new FatalError( 'Critical failure - do not retry' );
// Validation error (schema/input validation)
throw new ValidationError( 'Invalid input format' );
File Structure
src/workflows/{name}/
workflow.ts # Workflow definition (orchestration only)
steps.ts # Step definitions (all I/O here)
evaluators.ts # Evaluators (optional)
types.ts # Shared types and schemas (optional)
prompts/ # Prompt files directory
name@v1.prompt # Versioned prompt templates
scenarios/ # Test scenarios directory
basic.json # Common run case examples
edge_cases.json # Edge case scenarios
Allowed Imports in Workflows
Workflows can only import from:
steps.ts,evaluators.ts,shared_steps.ts- Whitelisted:
types.ts,consts.ts,utils.ts,variables.ts,tools.ts
Common Pitfalls
1. Zod Import Source
Always import z from @outputai/core, never from zod directly. Different z instances create incompatible schemas.
2. Direct API Calls
Never make HTTP/LLM calls directly in workflows. Always wrap in steps.
3. Non-Deterministic Operations
No Math.random(), Date.now(), or crypto in workflows.
4. Try-Catch Blocks
Don't wrap step calls in try-catch. Let failures propagate for proper retry handling.
5. Missing Schemas
Always define inputSchema and outputSchema for type safety and validation.
6. Credentials Usage
Never use process.env for secrets. Use credentials.require() from @outputai/core/credentials for mandatory secrets and credentials.get() with a default for optional values. See output-dev-credentials skill for setup and migration guidance.
7. Inline LLM Schemas
Define schemas for aiSdk.Output.object() in types.ts and import them. Never define schemas inline in step functions -- this causes duplication and drift.
8. Redundant JSON in Prompts
When a step uses aiSdk.Output.object(), the schema is sent to the LLM automatically. Do not describe JSON output format in the prompt -- it duplicates the schema and can conflict or drift over time. Use prompts for quality/content guidance, not structural format.
9. Code Style
All generated code must follow project style conventions. Key rules: no trailing commas, no let (use const only), arrow parens only when needed, operator linebreak after. See output-dev-code-style skill for the full list.
10. Structured Prompt Variables
The variables field in generateText and Agent accepts nested objects and arrays. Keep presentation-oriented iteration and property access in Liquid; pre-format data in the step only when formatting is application logic.
11. Mixed LLM Providers
All prompt files in a workflow should use the same provider unless the user explicitly requests otherwise. Mixing providers (e.g., some prompts using anthropic and others using openai) requires API keys for every provider used, causing runtime failures when keys are missing. Default to anthropic.
Evaluator Quality Checks
When reviewing evaluator implementations:
- Evaluators use appropriate result types for their purpose:
EvaluationBooleanResultfor pass/fail decisionsEvaluationNumberResultfor scores and ratingsEvaluationStringResultfor categories and classifications
- Confidence scores are meaningful:
1.0for deterministic checks (length, pattern matching)0.7-0.9for heuristic/LLM-based assessments
- Reasoning is provided for transparency
- Evaluators only import from allowed sources (no other evaluators/steps/workflows)
- Evaluators are in files containing 'evaluators' in the path
EvaluationFeedbackused when actionable suggestions are neededdimensionsused when multiple quality aspects need separate measurement
Example Interaction
User: "My workflow is failing with a type error on the schema"
Agent: Check that you're importing z from @outputai/core, not zod. Different z instances create incompatible schemas.
User: "How do I make an HTTP request in my workflow?"
Agent: Create a step that uses @outputai/core/http. Never make HTTP calls directly in the workflow function.
This agent specializes in Output SDK implementation best practices and code quality.
Hooks 1
Sponsors
Details
- Marketplace
- growthxai/output
- Source
- coding_assistants/claude/plugins/outputai
- Author
- Output.ai
- Category
- Community