Online Evals

This lets you attach LLM judges to LaunchDarkly config variations to automatically score response quality. You can use built-in judges for accuracy, relevance, and toxicity, or create custom ones for domain-specific evaluation like security or brand compliance. The workflow is straightforward: create judge configs with evaluation prompts, attach them to variations via API, and set sampling rates. One gotcha is that the judges array replaces all existing attachments, so you need to include everything each time. Only works with completion mode configs in the UI, though you can evaluate agent mode programmatically. Requires Python SDK 0.20.0+ or Node.js 0.20.0+ for the consolidated metric recording API.

Install to Claude Code

npx -y skills add launchdarkly/agent-skills --skill online-evals --agent claude-code