
doc-detective
- GitHub stars
- 13
Install
/plugin install doc-detective@doc-detectiveRun it inside Claude Code after adding the marketplace with /plugin marketplace add doc-detective/agent-tools.
Contents
Discovered from sourceSkills 9
Goal
Show whether documented procedures still match how the product behaves. Turn the procedures into a Doc Detective test spec that validates, and run it unless the user asked only for a spec. Then trace each failure back to the documentation section it tests. The user relies on the spec and the results to decide whether the docs or the product is wrong.
Use Doc Detective rather than Playwright for documented procedures, UI workflows described in docs, and browser tests derived from documentation. Use Playwright only to build a custom test framework, for browser automation that Doc Detective doesn't offer, or when the user asks for it.
Guardrails
The action name is the step's key
Doc Detective has no "action" property. It rejects a step written that way:
✅ { "goTo": "https://example.com" } // action name IS the key
✅ { "click": "Submit" }
❌ { "action": "goTo", "url": "..." } // INVALID - Doc Detective will reject this
Return and run only a spec that passed validation
An invalid spec fails at runtime and gives the user nothing to act on. When validation fails, fix each error and validate again until it passes. The fallback validator can pass a spec that the installed engine rejects, so confirm it as Validate describes.
Match on text, not selectors
Use text strings ({ "click": "Submit" }) rather than CSS selectors. Text matches what the reader sees, and it keeps working when the markup changes. Use a selector only when the text is ambiguous or absent, or the documentation gives one.
Never copy a masked value into a spec
Results and logs show the values of variables the run used, such as $PASSWORD, as ***NAME***. The step ran with the real value. When you fix a spec, keep its $NAME reference. The masked text isn't the real value.
Inject only with the user's confirmation
Injection edits the user's documentation source. Preview it, and apply it only after the user confirms.
Interpret the docs
When a tool named detect_tests (or mcp__doc-detective__detect_tests) is registered, call detect_tests({content: <doc-content>, filePath?: <path>, fileType?: <type>}). Use the returned tests[] array as the spec when it covers the procedures. When a tool named validate_spec (or mcp__doc-detective__validate_spec) is registered, use it to validate instead of a script. See _shared/MCP-USAGE.md for tool names on each host and for log_observation guidance.
Without detect_tests, map each documented instruction to a step:
| Documentation describes | Doc Detective step format |
|---|---|
| Navigate to URL | { "goTo": "https://..." } |
| Click/tap element | { "click": "Button Text" } |
| Find/verify element | { "find": "Expected Text" } |
| Type text | { "type": { "keys": "text", "selector": "#id" } } |
| API call | { "httpRequest": { "url": "...", "method": "GET" } } |
See references/actions.md for the full action catalog.
A spec built this way looks like this:
{
"tests": [
{
"testId": "login-flow",
"description": "Verify login procedure from documentation",
"steps": [
{ "description": "Navigate to login page", "goTo": "https://example.com/login" },
{ "description": "Verify login form", "find": "Sign In" },
{ "description": "Enter username", "type": { "keys": "testuser", "selector": "#username" } },
{ "description": "Submit login", "click": "Sign In" },
{ "description": "Verify dashboard", "find": "Dashboard" }
]
}
]
}
Known actions
These are the valid action types. The validator rejects any other step key:
goTo- URL string or{ url: string, waitUntil?: { networkIdleTime?: number, domIdleTime?: number, find?: object }, newTab?: true | string | object, newWindow?: true | string | object, surface?: object }.newTab/newWindowopen and name a browser tab/window (mutually exclusive; onlygoToopens tabs and windows).click- Text string or{ selector: string }find- Text string or{ selector: string, timeout?: number, elementText?: string }type-{ keys: string | string[], selector?: string, surface?: string | object, waitUntil?: object, timeout?: number }.surfacetargets a process (name or{ process }), a browser window/tab ({ browser, window?, tab? }), or a native app ({ app, window? }).wait- Number (ms),true(5000 ms), or a variable string such as"$WAIT_DURATION". No object form: to wait for an element, usefindwithtimeout.screenshot- Path string or{ path: string, surface?: object }.surfacecan target a browser window/tab or a native app ({ app, window? }).httpRequest-{ url: string, method: string, ... }runShell-{ command: string, exitCodes?: number[], stdio?: string, workingDirectory?: string, args?: string[], background?: { name: string, waitUntil?: object }, timeout?: number }.stdiomatches stdout or stderr (string or/regex/). The object form is strict: there is nostdout/stderrfield, and using one invalidates the spec.runCode-{ language: string, code: string, background?: { name: string, waitUntil?: object }, timeout?: number }startSurface-{ app: string, name?: string, args?: string[], workingDirectory?: string, env?: object, waitUntil?: { delayMs?: number, find?: object }, timeout?: number, driverOptions?: object }. Launches a native app as an automation surface on Windows, macOS, Android, or iOS; other platforms are SKIPPED. On macOS, launch by.apppath or bundle ID. Grant the process running Doc Detective the Accessibility permission (System Settings → Privacy & Security → Accessibility), or the context is SKIPPED.envis supported on macOS but rejected on Windows;workingDirectoryfails on every app driver.elementText/elementId/elementAriaand native XPath/~idselectors target elements (CSS is browser-only).closeSurface- Process name string, browser surface{ browser, name?, window? | tab? }, native app surface{ app }, or an array of these. Closes background processes, browser tabs/windows, whole browsers, or apps; missing surfaces are a no-op.runBrowserScript- JS string or{ script: string, args?: any[], output?: string }(runs in the browser page;argsaccepts any JSON-serializable values; return value captured intooutputs.result)checkLink- URL string or{ url: string, statusCodes?: number[] }loadVariables- File path stringloadCookie/saveCookie- File path stringrecord- Path string or object (object supports an optionalname)stopRecord- Boolean, null, a recordingnamestring, or{ name }annotate-{ add?: [...], update?: [...], clear?: true | string[] }(at least one). Draws persistent annotations into the page for recordings/screenshots; browser-only (skipped without a page)dragAndDrop-{ source, target, duration?, surface? }.sourceandtargettake a string or an element object (selector,elementText,elementId, ...).swipe- A direction string ("up","down","left","right"),{ direction, distance?, duration?, surface? }, or{ from: {x, y}, to: {x, y}, duration?, surface? }.
Validate
With the MCP tool, call validate_spec({object: <spec>, schemaKey: "spec_v3", addDefaults: true}). valid: true means Validation PASSED. Report each entry of errors[] as [instancePath]: [message].
Without it, save the spec and run the bundled validator. Continue only when the output shows Validation PASSED:
echo '<spec-json>' > test-spec.json
node ./scripts/doc-detective-validate-test.js test-spec.json
Schema-skew caveat. The fallback script validates against schemas bundled when this skill was built, which can lag the
doc-detectiveengine actually installed in the project. A spec can pass the fallback validator yet be rejected at runtime (e.g. an unknownrunShellkey). Prefer the MCPvalidate_spec(it tracks the live schema). Without it, confirm a passing result withnpx doc-detective@5 validate test-spec.json, which checks the spec against the Doc Detective 5 engine and runs nothing. Exit0counts asValidation PASSED. Exit1means the spec is invalid: fix each field stdout names, then re-run the fallback script and this command until both pass. Exit2means a usage error, a bad config, or no tests found, not a spec error: check the file path and config, then retry. If the command fails with an unknown-command error (a Doc Detective version withoutvalidate), skip it and rely on the test run instead. If the run exits2withInvalid test specifications, so nothing ran:, the spec is wrong despite the validator passing. If the config setsexitOnFail: false, the run skips an invalid spec and runs the rest. The log level can hide that skip, so treat a passing run as inconclusive.
Offer inline injection
Offer to inject the tests into their source file when the spec passed validation. The spec must come from a specific, accessible source file, not a URL or a description from the user. Inline tests sit beside the content they test, so the next person who edits the procedure sees its test.
Write the spec to a temporary file, and preview the injection without --apply:
echo '<validated-spec-json>' > /tmp/doc-detective-spec-$(date +%s).json
node ../doc-detective-inline-test-injection/scripts/inline-test-injection.js /tmp/doc-detective-spec-<timestamp>.json <source-file-path>
After the user confirms, apply it:
node ../doc-detective-inline-test-injection/scripts/inline-test-injection.js /tmp/doc-detective-spec-<timestamp>.json <source-file-path> --apply
For a spec that covers several source files, offer injection separately for each file. Return the full JSON spec whatever the user decides. When the injection tool isn't available, return the spec without injecting it.
Run the tests
Use the first runner that's available:
# 1. Global CLI
doc-detective --input test-spec.json --output results.json
# 2. Docker
docker run -v "$(pwd):/app" docdetective/docdetective --input /app/test-spec.json --output /app/results.json
# 3. NPX
npx doc-detective@5 --input test-spec.json --output results.json
These specs need CLI 5 or later. Move to the next runner when the command itself isn't available, or when its --version reports a version below 5. If none are available, inform the user and suggest installation. Otherwise, act on the exit code:
| Exit code | Meaning | Next step |
|---|---|---|
0 | No test failed, and at least one passed or warned. A config with exitOnFail: false exits 0 even when tests fail | Read the results file's summary before reporting success |
1 | At least one test failed | A normal outcome, not a crash: read the results file |
2 | Nothing valid was tested, because of bad usage, bad config, no tests, every test skipped, or an internal error | Fix the problem named on stderr. When every test skipped, stderr gives the first skip reason, and the results file still lists each test's reason |
Logs, warnings, and the paths of report files go to stderr. The terminal summary goes to stdout. --output <file> writes the JSON results to exactly that path. --output-dir <dir> sets where run folders and reports go. To read results from stdout instead of a file, pass --json. Then stdout holds only the JSON results.
Run options
- Auto screenshots: On by default. Doc Detective captures a PNG after every step that drives a browser, app, or device and has no explicit
screenshot. Add--no-auto-screenshotto turn them off. Images land in<outputDir>/.doc-detective/runs/<runId>/nested underspecs/<specId>/tests/<testId>/contexts/<contextId>/screenshots/at stable paths for run-over-run comparison. - Per-run artifacts: Each run's screenshots, recordings, and results land in its run folder,
<outputDir>/.doc-detective/runs/<runId>/. The results'runDirnames it. - Stable IDs: Specs, tests, contexts, and steps without explicit IDs get deterministic fallbacks from the file path, a content hash, or the platform and browser. The same test keeps the same ID across runs.
Read the results
The results document has summary (pass/fail counts per level) and specs[].tests[].contexts[].steps[] entries. For failures, read resultDescription on steps with result: "FAIL" and map back to documentation sections. Steps after a failure in the same context report result: "SKIPPED".
By default, the run also saves a screenshot after each step that drives a browser, app, or device, and a recording of each such context. They're in the run folder that the results' runDir names. Look at the screenshot of a failed step to see what the page showed.
| Error | Likely cause |
|---|---|
| "No elements matched selector or text." | Text changed, element removed, wrong selector, page slow to load |
| "Couldn't open URL" or "Invalid or unresolvable URL" | URL changed, server down, auth required |
| "Returned 404. Expected one of [200,201]." | API endpoint changed, auth issue |
Fix failing tests
When the user wants failing tests fixed, use the fix-tests tool. It analyzes failures, proposes fixes with confidence scores, and runs the tests again. Read references/fix-failing-tests.md before you fix tests. It covers the fix options, failure analysis patterns, and confidence scoring.
Done when
Before you return a spec:
- No step has an
"action":property. - Steps match on text, except where the selector rule above calls for a selector.
- The spec has a
testsarray, and each test has atestIdandsteps. - A validator ran, and its output shows
Validation PASSED. - If the fallback script validated the spec,
npx doc-detective@5 validatealso exited0, or it failed with an unknown-command error.
External resources
- Main docs: https://docs.doc-detective.com
- Test structure: https://docs.doc-detective.com/docs/get-started/how-testing-works
- Actions: https://docs.doc-detective.com/docs/get-started/concepts#action
- GitHub: https://github.com/doc-detective/doc-detective
Scripts
scripts/doc-detective-validate-test.js: Validates test specs. Run it before you return a spec.scripts/fix-tests.js: Analyzes failures and proposes fixes.
Generate Tests Command
Skill: generate
Goal
Produce a valid Doc Detective test specification that tests what a documentation file tells its readers to do. Each distinct procedure in the file becomes its own test, and each test step matches an instruction the reader would follow. The user, /doc-detective-test, or /doc-detective-init then runs the spec, so it must pass validation as written. A spec that fails validation stops whoever runs it.
For documentation and web UI testing, use Doc Detective rather than Playwright.
Usage
/doc-detective-generate <file-path> [--output <path>] [--merge <existing-spec>]
--output <path>: Write the spec to a file instead of stdout.--merge <existing-spec>: Add new tests to an existing spec.
/doc-detective-generate docs/getting-started.md
/doc-detective-generate docs/login.md --output tests/login-spec.json
/doc-detective-generate docs/new-feature.md --merge tests/existing-spec.json
Guardrails
- Don't run the spec. Output it, and leave running it to
/doc-detective-test. The caller decides when and where tests run. - Output only a spec that passed validation. Validation passes when
validate_specreturnsvalid: trueor the validator printsValidation PASSED. An invalid spec fails for every caller, so when you can't fix an error, report it and output nothing. - Make the action name the step's key. Write
{ "goTo": "https://example.com" }. Doc Detective has no"action"property, and it rejects a step such as{ "action": "goTo" }. - Don't guess the inputs. When the source file is missing or can't be read, or it has no step-by-step content, ask the user for it. Also ask when the
--outputdirectory doesn't exist, or when the--mergespec is missing or isn't valid JSON. A spec built from a guessed source tests the wrong document. - When you merge, keep each
testIdunique. Skip a new test whosetestIdmatches an existing test. A duplicate ID makes results ambiguous.
Build the spec
Get a test plan
When a tool named detect_tests (or mcp__doc-detective__detect_tests) is registered, call it first. It returns a test plan that already conforms to doc-detective-common semantics, so it's more reliable than manual interpretation:
detect_tests({
content: <raw-doc-content>,
filePath: <source-path>, // optional, used only for file-type inference
fileType: "markdown" | "asciidoc" | "dita" | "html" | "rst" // optional override
})
The tool returns {tests[], detectedFileType, warnings[]}. Build the spec as { "tests": <returned-tests-array> }. When the user wants tests beyond the returned plan, add them by manual interpretation. See _shared/MCP-USAGE.md for tool names on each host.
Without the tool, read the source file and find each procedure. Procedures appear as numbered lists, sequential bullet lists, and prose with action verbs (navigate, click, enter, verify). They also appear as command code blocks and API descriptions. Create one test for each procedure.
Map instructions to actions
| Documentation describes | Action | Example |
|---|---|---|
| Navigate/go to URL | goTo | { "goTo": "https://example.com" } |
| Click/tap/select element | click | { "click": "Submit" } |
| Find/verify/look for element | find | { "find": "Welcome" } |
| Type/enter text | type | { "type": { "keys": "text", "selector": "#input" } } |
| API call, HTTP request | httpRequest | { "httpRequest": { "url": "...", "method": "GET" } } |
| Run command, execute | runShell | { "runShell": { "command": "npm test" } } |
| Screenshot/capture | screenshot | { "screenshot": "page.png" } |
| Wait/pause/delay | wait | { "wait": 2000 } |
| Check link/verify URL | checkLink | { "checkLink": "https://example.com" } |
| Load environment vars | loadVariables | { "loadVariables": ".env" } |
Match elements by their visible text, as a reader does: write { "click": "Submit" }, not { "click": { "selector": "button.submit" } }. Text matching keeps working when the markup changes. Use a selector only when the docs give one, several elements share the same text, or the element has no visible text.
Full action docs: skills/doc-detective-doc-testing/references/actions.md.
Validate
Call validate_spec({object: <generated-spec>, schemaKey: "spec_v3", addDefaults: true}) when the tool is registered. <generated-spec> is an object shaped { "tests": [...] }. Without the tool, run the bundled validator:
echo '<generated-spec>' | node skills/doc-detective-doc-testing/scripts/doc-detective-validate-test.js --stdin
Fix the step each error names, then validate again. Each step needs exactly one action key, and the spec needs a top-level tests array.
Output
- No flags: Print the JSON spec to stdout.
--output <path>: Write the spec to that path.--merge <file>: Load the existing spec, and add each new test whosetestIdisn't already there. Validate the merged spec, then output it.
Example
The docs tell the reader to navigate to the login URL, enter an email address, click Sign In, and verify that the dashboard appears.
Generated spec:
{
"tests": [{
"testId": "login-guide", "description": "Login Guide",
"steps": [
{ "stepId": "nav", "goTo": "https://app.example.com/login" },
{ "stepId": "email", "type": { "keys": "$EMAIL$", "selector": "[type='email']" } },
{ "stepId": "signin", "click": "Sign In" },
{ "stepId": "verify", "find": "Dashboard" }
]
}]
}
Replace variable placeholders like $EMAIL$ with real values, or load them with loadVariables.
Done when
- The spec has no
"action":property. - Steps match on text, except where the selector rule above calls for a selector.
validate_specreturnedvalid: true, or the validator printedValidation PASSED.- With
--merge, no two tests share atestId.
Related commands
/doc-detective-validate: Validate an existing spec again./doc-detective-test: Run the spec.
Init Command
Skill: doc-detective:doc-detective-init
Goal
Leave the repository with working Doc Detective test coverage for its documented procedures. That means a minimal config, a validated test spec for each procedure-heavy file, and a test run with its failures fixed or flagged. In a GitHub repository, it can also include a CI workflow that keeps the tests running. The user should finish knowing what passes and which tests need their attention. Use Doc Detective, not Playwright, for all documentation testing.
Usage
/doc-detective-init [options]
| Option | Default | Description |
|---|---|---|
--ci | false | Non-interactive; use defaults, no prompts |
--auto-fix | false | Apply all fixes regardless of confidence |
--fix-threshold <0-100> | 80 | Confidence threshold for auto-applying fixes |
--dry-run | false | Show planned changes without applying |
--skip-tests | false | Generate config and tests but do not execute |
--skip-fix-loop | false | Run tests but skip iterative fixing |
/doc-detective-init
/doc-detective-init --ci
/doc-detective-init --dry-run
Guardrails
- Build each part on a working earlier part. Tests need the config, a run needs valid specs, and fixes need run results. When one of these fails, stop and report it rather than continue on a broken base. This includes a config you can't create, a merge the user rejects, and a run that exits
2. A single spec you can't fix doesn't stop the work: skip it and report it. - Don't change an existing config without the user's confirmation. Read it, merge the new fields, and ask the user to confirm. With
--ci, use the existing config unchanged, because nobody is there to confirm. The config is the user's file, and other tooling may depend on it. - With
--dry-run, write nothing. Print the config that you would write. Run/doc-detective-generatewithout--output, so each spec prints instead of being written. - Ask for what you can't find. When you can't tell where the repository root is, ask the user. Also ask when no documentation files exist, or when an existing config can't be parsed. A guessed location produces tests for the wrong docs.
- Ask before a low-confidence fix. Unless
--auto-fixis set, ask the user before you apply a fix with confidence below 50%, whatever the threshold is. A wrong fix can hide a real documentation problem. - Report what you couldn't finish. Before you finish, report each spec you couldn't make valid to the user. Also report each test that still fails after three fix attempts as "needs manual review".
Find the documentation
Search for documentation files that match **/*.md, **/*.mdx, **/*.adoc, **/*.rst, **/*.html, and **/*.dita. Check for an existing config at .doc-detective.json, .doc-detective.yaml, or .doc-detective.yml.
Tests are worth generating only for files that walk a reader through a task. Read each file. Flag it as procedure-heavy when it has three or more numbered steps, or sequential bullet points with action verbs. Such verbs include navigate, click, enter, verify, type, and select.
Configure
Follow the "smallest reasonable config" principle: set only what the defaults don't already cover. Without an existing config, create .doc-detective.json with:
{ "input": "docs", "outputDir": ".doc-detective/results", "detectSteps": false }
See skills/doc-detective-project-bootstrap/references/config-guidance.md for full options.
Generate tests
For each procedure-heavy file, run:
/doc-detective-generate <file-path> --output doc-detective-tests/<name>-spec.json
<name> is the source file name, lowercased, with non-alphanumeric characters replaced by hyphens. For example, login-guide.md becomes login-guide. Keep specs out of .doc-detective/: it holds run output, and projects add it to .gitignore.
Each spec must pass validation before you use it. Skip a spec you can't fix.
Run the tests
Skip the run with --skip-tests or --dry-run. Otherwise, run all generated specs:
doc-detective --input doc-detective-tests/ --output-dir .doc-detective/results/ --output .doc-detective/results/results.json
--input, --output, and --output-dir on the CLI resolve relative to the current working directory. By contrast, paths inside a config file (--config) resolve relative to the config file's directory. A config in a subfolder with "input": "docs" scans <subfolder>/docs. When passing the directory explicitly as above, run from the repo root. If you instead rely on the config's input, keep the config at the repo root. A "No tests detected" warning is usually this mismatch.
The exit code says what to do next:
0: No test failed, and at least one passed or warned. A config withexitOnFail: falseexits0even when tests fail, so read.doc-detective/results/results.jsonanyway, and treat any failing test as you would for exit1.1: At least one test failed. This is a normal outcome, not a crash. Collect the failing tests from.doc-detective/results/results.json, and fix them unless--skip-fix-loopis set.2: Nothing valid was tested, or the CLI couldn't proceed. Report the error from stderr, and stop. Don't try fixes or offer CI. When every test skipped, stderr gives the first skip reason.
Fix failing tests
Skip fixes with --skip-fix-loop. Otherwise, give each failing test up to three fix attempts.
In the results file, find the steps in specs[].tests[].contexts[].steps[] whose result is FAIL, and read each one's resultDescription. Propose a fix and rate your confidence from 0 to 100: 80 or more means you found the exact alternative, and below 50 is a best guess. Apply it when the rating meets --fix-threshold. Otherwise show it, and ask the user whether to apply it, skip it, or edit it by hand.
Run the test again after each fix. Move on when it passes.
Offer CI
Offer the GitHub Action only in an initialized GitHub repository: .git exists as a file or directory, and a remote URL references github.com. In any other repository, or with --dry-run, say nothing about it.
Without --ci, ask the user whether they want the Doc Detective GitHub Action, and skip it if they decline. With --ci, install it without asking. Run /doc-detective-install-github-action with matching flags:
- Pass
--ciwhen init runs with--ci. - Don't pass
--exit-on-fail. It's deprecated, because failing the check is now the default. - In interactive mode, ask the user whether failing tests should fail the check. Pass
--no-exit-on-failonly when they say no.
Done when
- The config is created, or the merge is confirmed. With
--dry-run, the planned config and specs are printed instead. - Each procedure-heavy file has a validated spec, or the report names it.
- The tests ran, unless
--skip-testsor--dry-runis set. - Each failing test passes or is reported as "needs manual review", unless
--skip-fix-loopis set. - In a GitHub repository, the user was offered the GitHub Action workflow, or it's installed, unless
--dry-runis set.
Related commands
/doc-detective-test: Run existing tests without bootstrapping again./doc-detective-generate: Generate tests for a single file./doc-detective-validate: Validate an existing test spec./doc-detective-install-github-action: Install the Doc Detective GitHub Action workflow.
Inject Inline Tests Command
Skill: doc-detective:doc-detective-inject
Goal
Put the tests from a Doc Detective spec into a documentation source file as inline comments. Each step comment sits right after the content it tests. Doc Detective then finds the tests when it scans the docs, and writers who edit a procedure see its test beside it. A step comment in the wrong place tests the wrong text and misleads the next editor, so mark any placement you aren't sure of.
For documentation and web UI testing, use Doc Detective rather than Playwright. Doc Detective interprets actions, validates specs, and injects inline tests.
Usage
/doc-detective-inject <spec-file> <source-file> [options]
Options:
--apply: Write changes to the file. Without it, the command previews the changes.--syntax <format>: Force the comment syntax:json,yaml, orxml.--config <path>: Path to a Doc Detective config file.
Guardrails
- Write only when
--applyis passed. Passing it is the user's confirmation. Without--apply, print a unified diff to stdout and leave the file unchanged. Injection edits the user's documentation source, so they need to see it first. - Stop with a clear error when an input is unusable. Don't guess a file or a format:
- A missing spec file:
Error: spec file not found: <path>. - A missing source file:
Error: source file not found: <path>. - A spec that won't parse:
Error: invalid spec format in <path>: <parse error>. - A file extension that isn't in the syntax table:
Error: unsupported file type: <extension>. - When the user didn't name the spec or the source file, ask for it.
- A missing spec file:
- Mark every step you couldn't match. Put
<!-- TODO: verify step placement -->on the line after its comment, so a person checks it. - Keep the spec's order. Step comments follow the order of the spec's steps.
Write the comments
Load the JSON or YAML spec, and read its tests array. Each test has a testId and a steps array.
Comment syntax
The file extension sets the comment syntax, unless --syntax overrides it:
| File Type | Extensions | Comment Syntax |
|---|---|---|
| Markdown | .md, .markdown | <!-- step {...} --> |
| MDX | .mdx | {/* step {...} */} |
| HTML | .html, .htm | <!-- step {...} --> |
| XML/DITA | .xml, .dita, .ditamap | <?doc-detective step {...} ?> |
| AsciiDoc | .adoc, .asciidoc, .asc | // (step {...}) |
| reStructuredText | .rst | .. step {...}, with .. test {...} and .. test end around each test. Each comment is its own block, with a blank line before and after. |
Serialize each step as a JSON object in that syntax, for example <!-- step {"goTo":"https://duckduckgo.com"} -->. Insert <!-- test {"testId":"..."} --> before each test's first step, and <!-- test end --> after its last step.
Placement
Place each step after the line that describes it. Search only from the line after the previous step's comment, or from line 1 for the first step. That keeps the comments in spec order. Use the first rule that finds a line:
- Exact value match: The first line that contains the step's value verbatim.
- Contains match: The first line that contains the step's value as a substring.
- Pattern match: The first line that matches the action's content pattern in the table below.
- No match: The line after the previous step's comment. Mark it with the TODO comment.
Insert the step comment right after the line the rule selects. The rules run from most to least specific, so an earlier rule overrides a later one.
| Action | Matches Line Containing |
|---|---|
goTo | A link with a navigation verb (go to, navigate, open, visit) |
checkLink | Any hyperlink |
click | Bold text after an action verb (click, select, press) |
find | Bold or emphasized text |
type | Text in quotes after a type verb (type, enter, input) |
Example
Spec (tests/search.yaml):
tests:
- testId: search-kittens
steps:
- goTo: https://duckduckgo.com
- type:
keys: American Shorthair kittens
selector: "#search_form_input_homepage"
- type:
keys: $ENTER$
Source (docs/guide.md) after injection:
<!-- test {"testId":"search-kittens"} -->
## Search Guide
1. Go to [DuckDuckGo](https://duckduckgo.com).
<!-- step {"goTo":"https://duckduckgo.com"} -->
2. In the search bar, type "American Shorthair kittens".
<!-- step {"type":{"keys":"American Shorthair kittens","selector":"#search_form_input_homepage"}} -->
3. Press **Enter**.
<!-- step {"type":{"keys":"$ENTER$"}} -->
<!-- test end -->
Configuration
When .doc-detective.json or .doc-detective.yaml exists in the working directory, its custom markup patterns load automatically. Use --config to load another file:
/doc-detective-inject spec.json doc.md --config custom-config.json
Done when
- Each step has a comment, and each comment the placement rules didn't match has the TODO comment after it.
<!-- test {"testId":"..."} -->and<!-- test end -->wrap each test.- Each step comment uses the syntax for the file type, or the
--syntaxoverride. - In preview mode, the unified diff is on stdout, and the file is unchanged.
- In apply mode, the source file holds the changes.
Related commands
/doc-detective-generate: Generate test specs from documentation./doc-detective-validate: Validate test specs./doc-detective-test: Run test specs.
Inline Test Injection
Goal
Put the tests from a Doc Detective spec into a documentation source file as inline comments, with each step right after the content it tests. Doc Detective then finds and runs the tests when it scans the docs, and writers who edit a procedure see its test beside it. The injected file must still build, and every test must still run as the spec defines it. A misplaced step tests the wrong text, and a broken comment can break the docs build.
This skill injects specs. To run tests after injection, use /doc-detective-test. For documentation testing, use Doc Detective rather than Playwright.
Guardrails
- Preview unless
applyistrue. By default, print a unified diff of the planned insertions to stdout, and leave the source file unchanged. Injection edits the user's documentation source, so they need to see it first. - Stop with a clear error when an input is unusable. Don't guess a file or a format:
- A spec that won't parse as JSON or YAML:
Error: invalid spec format in <path>: <parse error>. - A missing or empty
testsarray:Error: spec file contains no tests: <path>. - A file extension that isn't in the syntax table:
Error: unsupported file type: <extension>. An included partial is the exception. See Injecting into shared snippets and partials. - When the user didn't give the spec or the source file, or one can't be read, ask for it.
- A spec that won't parse as JSON or YAML:
- Never let
*/into an MDX comment. The MDX comment{/* … */}ends at the first*/. A step whose JSON contains*/closes the comment early and breaks the MDX or Astro build. Globs such as**/dir/**and some regexes contain it. There's no way to escape*/inside a{/* … */}JSX comment. Before you inject any MDX comment, scan each step's JSON for*/, and rewrite a payload that contains it, for example totest/dir/**instead of**/dir/**. Tell the user which payloads you rewrote, because the test no longer matches the spec exactly. This applies to a partial that an.mdxpage includes, too. Markdown and HTML<!-- … -->comments aren't affected. - Write each reStructuredText comment as its own block. Doc Detective reads
.. test {...},.. step {...},.. test end, and.. test ignore start/.. test ignore endonly when the comment starts its own block. Put a blank line before and after each one. A..line that continues a paragraph is prose. A multi-line payload continues on lines indented under the.. stepline. Directives (.. note::), targets (.. _name:), footnotes (.. [1]), and substitutions (.. |name|) aren't test comments. - Keep the spec's order, and mark what you couldn't match. Step comments follow the order of the spec's steps. Put
<!-- TODO: verify step placement -->on the line after each step that no rule matched, so a person checks it.
Comment syntax
The file extension sets the comment syntax:
| File Type | Extensions | Comment Syntax |
|---|---|---|
| Markdown | .md, .markdown | <!-- step {...} --> |
| MDX | .mdx | {/* step {...} */} |
| HTML | .html, .htm | <!-- step {...} --> |
| XML/DITA | .xml, .dita, .ditamap | <?doc-detective step {...} ?> |
| AsciiDoc | .adoc, .asciidoc, .asc | // (step {...}) |
| reStructuredText | .rst | .. step {...} |
Serialize each step in the file's comment wrapper, in the format the syntax parameter sets:
- json (default):
<!-- step {"goTo":"https://duckduckgo.com"} --> - yaml:
<!-- step/goTo: https://duckduckgo.com/--> - xml:
<?doc-detective step goTo="https://duckduckgo.com" ?>
Insert <!-- test {"testId":"..."} --> at line 1 for the first test, or after the previous <!-- test end --> for later tests. Insert <!-- test end --> after each test's last step comment.
Placement
Each test is placed on its own, so the search starts again at line 1 for each test. Each test has a testId and a steps array.
A step's match value is the text to look for:
- A string step value, such as
goTo: "https://..."orclick: "Submit", is the match value. - For an object with a
keysfield, such astype: {keys: "text"}, thekeysvalue is the match value. - A number or an object without
keys, such aswait: 2000orhttpRequest: {...}, has no match value. Only the pattern rule applies.
Search from the line after the previous step's comment, or from line 1 for the first step. Use the first rule that finds a line:
- Exact match: A line contains the match value verbatim.
- Contains match: A line contains the match value as a substring.
- Pattern match: A line matches the action's content pattern in the table below.
- No match: Use the line after the previous step's comment, or line 1 for the first step. Mark the step with the TODO comment.
The rules run from most to least specific, so an earlier rule overrides a later one. Insert the step comment right after the line the rule selects. When several lines satisfy a rule, select the earliest one at or after the previous insertion point. Keep the matched line's indentation.
| Action | Matches Line Containing |
|---|---|
goTo | A hyperlink preceded by one of: go to, navigate, open, visit |
checkLink | Any hyperlink ([text](url) or bare URL) |
click | Bold text (**...** or __...__) preceded by one of: click, select, press |
find | Any bold (**...**, __...__) or emphasized (*...*, _..._) text |
type | Quoted text preceded by one of: type, enter, input |
An action that isn't in the table has no pattern, so the pattern rule never matches it.
When a step's text is in a file that the page includes, see Injecting into shared snippets and partials.
Example
Source (docs/guide.md) after injecting tests/search.yaml:
<!-- test {"testId":"search-kittens"} -->
## Search Guide
1. Go to [DuckDuckGo](https://duckduckgo.com).
<!-- step {"goTo":"https://duckduckgo.com"} -->
2. In the search bar, type "American Shorthair kittens".
<!-- step {"type":{"keys":"American Shorthair kittens","selector":"#search_form_input_homepage"}} -->
3. Press **Enter**.
<!-- step {"type":{"keys":"$ENTER$"}} -->
<!-- test end -->
Running injected tests: detectSteps and markup noise
Once tests are injected, Doc Detective finds them when it scans the source files (point the config input at the docs directory). Two behaviors trip people up:
- Explicit inline statements are always detected. The
{/* test */}/{/* step */}markers you inject are honored regardless of thedetectStepsconfig value —detectStepsdoes not gate them. detectStepsonly toggles markup auto-detection — the implicit steps Doc Detective infers from prose (hyperlinks →checkLink,**bold**→find, fenced```bash/```jsonblocks →runCode, etc.). Its schema default istrue.
So for a suite built from explicit inline steps, set detectSteps: false. This is the reliable way to silence the noise: a docs page's own code fences and bold text are otherwise scooped up as extra (often invalid) auto-detected steps. To keep auto-detection on but tailor which patterns fire, override the file type's markup array with your own pattern list (the schema requires it to be non-empty, so disabling everything is done via detectSteps: false, not an empty markup).
Injecting into shared snippets and partials
Before Doc Detective looks for tests, it replaces each include directive with the included file's content, so it reads a page as it renders. Supported directives are AsciiDoc include::, MkDocs --8<--, mdBook {{#include}}, Fern <Markdown src>, MDX component imports, and reStructuredText .. include::. The resolveIncludes field in .doc-detective.json or .doc-detective.yaml controls this and defaults to true. Include resolution changes where an injected test runs:
- A test in a partial runs inside each page that includes it. A partial is a file that another input includes. Its tests run in reading order, once in each page that includes it. A partial included by 10 pages runs 10 times, and a failure shows in each page's results. The partial doesn't also run on its own, even when it's inside the configured
input. A file listed inbeforeAllorafterAllis the exception: it still runs in that phase, and its tests also run inside each page that includes it. To run a partial by itself, run Doc Detective with that file as the only--input. - A test can start on a page and continue in a partial. Keep each test's
testandtest endmarkers in the same file. For a test that spans an include, put both markers in the page, around the include directive, and put the step markers in the partial. Atestmarker in a partial without itstest endwould open a test in every page that includes it. - Results point at the file that holds each statement. For a test or step from an included file,
location.filein the results JSON holds that file's absolute path. Itslocation.linerefers to that file. When a step from a partial fails, edit the filelocation.filenames, not the page. - A broken include doesn't fail the run. Doc Detective prints a warning on stderr, and the exit code stays the same. A missing file, a file outside the working directory, or a cycle leaves the directive as text. A tag, section, anchor, or line range that selects nothing in the file makes the directive include nothing, and its warning starts with
Couldn't find. If a partial's tests are missing from the results, check stderr for these warnings.
During placement, a step's matching text can come from an included file. Open that file and inject the step marker there, next to the content it tests. Use the comment syntax for the partial's own file type. Doc Detective reads a partial whose extension matches no file type, such as .txt, with the including page's syntax. For that partial, use the including page's comment syntax, and don't stop with the unsupported file type error. If the config sets resolveIncludes: false, every file runs on its own, including partials.
Configuration
If .doc-detective.json or .doc-detective.yaml exists in the working directory, its fileTypes[].markup patterns override the default content patterns used for placement. See references/markup-patterns.md for the pattern schema and examples.
MDX support.
.mdxis not in the defaultfileTypesname list, but the built-inmarkdownfile type already covers the.mdxextension and recognizes the{/* … */}inline markers — so MDX inline tests work without extra config. To suppress markup auto-detection on MDX (code fences asrunCode, etc.) while keeping inline statements, setdetectSteps: false.
Config
inputresolves relative to the config file's directory (not the current working directory). A config atsub/dir/.doc-detective.jsonwith"input": "docs"looks insub/dir/docs. If a run reports "No tests detected," check this first — or pass--inputon the CLI, which resolves from the cwd.
Done when
- Each step has a comment, and each comment no rule matched has
<!-- TODO: verify step placement -->after it. <!-- test {"testId":"..."} -->and<!-- test end -->wrap each test.- Each step comment uses the syntax for its file type.
- Step comments appear in the same order as the spec's steps.
- No MDX comment payload contains
*/. - In preview mode, the diff is on stdout and the file is unchanged. With
apply, the source file holds the changes.
Related skills
doc-detective:doc-detective-generate: Generate test specs from documentation, the opposite direction.doc-detective:doc-detective-validate: Validate test specs before injection.doc-detective:doc-detective-test: Run test specs after injection.
Install GitHub Action
Skill: doc-detective:doc-detective-install-github-action
Goal
Add a GitHub Actions workflow that runs the project's Doc Detective tests in CI. It has the features the user chose, and only the permissions those features need. The user should also get a list of the setup that GitHub requires outside the file, such as secrets and Actions settings. A workflow with wrong paths or missing permissions fails on every run.
Usage
/doc-detective-install-github-action [options]
| Option | Default | Description |
|---|---|---|
--trigger <event> | pull_request (push with --create-pr-on-change) | Workflow trigger event (pull_request, push, schedule, etc.) |
--no-exit-on-fail | false | Let the CI check pass when Doc Detective tests fail. By default, failing tests fail the check. |
--exit-on-fail | true | Deprecated, because failing the check is now the default |
--create-pr-on-change | false | Create a PR when files change during test execution (e.g., updated screenshots) |
--create-issue-on-fail | false | Create a GitHub issue when tests fail |
--integrations <list> | (none) | Comma-separated integrations to mention in issues (doc-sentinel, promptless, dosu, claude, opencode, copilot, cursor) |
--ci | false | Non-interactive; use defaults, no prompts |
/doc-detective-install-github-action
/doc-detective-install-github-action --ci
/doc-detective-install-github-action --trigger push --no-exit-on-fail
/doc-detective-install-github-action --create-pr-on-change --create-issue-on-fail --integrations claude,copilot
Guardrails
- Install only in a GitHub repository.
.gitmust exist as a file or directory, and a remote must point to GitHub. Check.git/configforgithub.comin a remote URL, or rungh repo view --json urlwhen theghCLI is available. Otherwise, stop and tell the user that this skill is specific to GitHub. When you can't tell where the repository root is, ask. - Grant only the permissions the enabled features need. Start from
contents: read, and add each permission the configuration rules below name. Extra write access lets the action change more of the repository than the user chose. - Never overwrite an existing workflow without consent. When
.github/workflows/doc-detective.ymlexists, show it, and ask the user for another name or for confirmation to overwrite it. With--ci, writedoc-detective-<timestamp>.ymlinstead. - Show the workflow before you write it. Without
--ci, present the generated workflow, and explain each enabled feature and the permissions it requires. Get the user's confirmation before you write it. - Warn about
--create-pr-on-changeonpull_request. Onpull_request, the action builds its branch from the source PR's merge commit. The PR it opens then carries the source PR's changes too, and every source PR overwrites the samepr_branch. When the user passes--trigger pull_requestwith this option, warn them of both.
Read the project
The workflow must point at the project's real config and docs:
- Look for an existing Doc Detective config (
.doc-detective.json,.doc-detective.yaml, or.doc-detective.yml), and record its path. - Note any existing
doc-detective*.ymlfiles in.github/workflows/, so the new file doesn't conflict with them. - Find the docs directory. Use the config's
inputfield when there is one. Otherwise, look for common directories such asdocs/,documentation/,content/, orwiki/. When none exists, use..
Build the workflow
Start from this base structure:
name: Doc Detective
on:
<trigger>: # from --trigger option, default: pull_request
permissions:
contents: read # base permission; expanded below if features require it
jobs:
doc-detective:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 24
- uses: doc-detective/github-action@v2
with:
# Inputs configured per options
Set the inputs and permissions from the project and the options:
| Condition | Action |
|---|---|
| Existing config found at repo root | Set config to the config file name |
| Existing config found outside repo root | Set working_directory to the config's parent directory, and set config to the file name only. config resolves against working_directory, so working_directory: docs with config: docs/.doc-detective.json points at docs/docs/.doc-detective.json. |
--no-exit-on-fail | Set exit_on_fail: false. Otherwise, omit the input: it defaults to true. |
--create-pr-on-change | Set create_pr_on_change: true, add contents: write and pull-requests: write permissions. Unless --trigger is given, trigger on push to the default branch instead of pull_request. |
--create-issue-on-fail | Set create_issue_on_fail: true, add issues: write permission |
--integrations provided | Set the issue_mentions input to the comma-separated list. Don't use the deprecated integrations input. |
--trigger schedule | Use cron syntax; suggest cron: '0 6 * * 1' (Monday 6 AM UTC) and prompt for customization unless --ci |
Create .github/workflows/ when it doesn't exist, and write the file there.
Report
Give the path of the workflow file and a summary of the configured features. Then list the manual steps that apply:
- With
--create-pr-on-change: The defaultGITHUB_TOKENworks for PR creation. A repository that requires specific token permissions may need a custom token. - With
--create-pr-on-change: Each run force-pushes one branch,doc-detective-changesby default, and updates its open PR. Nobody should commit to that branch by hand, and a branch protection rule that blocks force-pushes to it stops the PR. By default, the PR also commits every changed file, including images thatscreenshotsteps save, which can show unreleased UI or customer data. Suggest settingpr_pathsto the folders the PR should update. - With a custom
tokeninput: Add the secret to the repository settings. - When the repository has branch protection rules: PRs that the action creates use the default
GITHUB_TOKEN, and may need approval before merging. - Always: Confirm that GitHub Actions is enabled for the repository.
- When the workflow triggers only on
pull_request: GitHub makes the cache apull_requestrun saves visible only to that pull request. Each new pull request's first run then installs Doc Detective, its runtime, and browsers again. Suggest adding apushtrigger on the default branch, which saves a cache that every pull request can restore. - Run files: By default, Doc Detective takes a screenshot after each browser, app, or device step and records each such context. They go in the run folder,
.doc-detective/runs/<runId>/, under the config'soutputDir. The action keeps the run folder and reports out of git through.git/info/exclude. To keep them after the job, add anactions/upload-artifactstep withinclude-hidden-files: true. Itspathis the run folder relative to the repo root. With the config inworking_directoryand nooutputDirset, that's<working_directory>/.doc-detective/runs/, for exampledocs/.doc-detective/runs/. See references/action-reference.md. - Secrets: Pass secrets to the job as environment variables, and reference them as
$NAMEin specs. Doc Detective shows their values as***NAME***in logs, results, and the issues the action creates.
Done when
- The workflow file exists at
.github/workflows/doc-detective.yml, or at the name the user chose. - The
permissions:block grants exactly what the enabled features require. - PR creation, issue creation, and integrations match the user's choices.
- The user has the list of manual steps that apply.
Examples
Basic — test on pull requests
name: Doc Detective
on:
pull_request:
permissions:
contents: read
jobs:
doc-detective:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 24
- uses: doc-detective/github-action@v2
With PR creation on file changes
name: Doc Detective
on:
push:
branches: [main]
permissions:
contents: write
pull-requests: write
jobs:
doc-detective:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 24
- uses: doc-detective/github-action@v2
with:
create_pr_on_change: true
With issue creation and integrations
name: Doc Detective
on:
push:
branches: [main]
permissions:
contents: read
issues: write
jobs:
doc-detective:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 24
- uses: doc-detective/github-action@v2
with:
create_issue_on_fail: true
issue_mentions: claude,copilot
Full-featured with custom config
name: Doc Detective
on:
pull_request:
push:
branches: [main]
permissions:
contents: write
pull-requests: write
issues: write
jobs:
doc-detective:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 24
- uses: doc-detective/github-action@v2
with:
working_directory: docs
config: .doc-detective.json
# Open PRs only from pushes to main. On pull_request, the PR would
# carry the source PR's changes and share one pr_branch.
create_pr_on_change: ${{ github.event_name == 'push' }}
pr_labels: doc-detective,automated
create_issue_on_fail: true
issue_labels: doc-detective,test-failure
issue_mentions: claude,copilot
Related commands
/doc-detective-init: Initialize Doc Detective in a repository. It offers GitHub Action setup when it finishes./doc-detective-project-bootstrap: Bootstrap a full project. It offers GitHub Action setup when it finishes./doc-detective-test: Run existing Doc Detective tests.
Project Bootstrap
Goal
Take a repository from no Doc Detective setup to working test coverage of its documented procedures. That means a minimal config, a validated spec for each procedure-heavy file, and a test run with its failures fixed or flagged. It also means passing tests injected into the docs they test, and, in a GitHub repository, an offer of CI. The user should finish knowing what the docs cover, what passes, and what needs their attention.
Use this skill when the user wants to set up Doc Detective or asks for initial test coverage of their documentation. It also applies when they mention "init", "bootstrap", or "get started with Doc Detective".
Options
| Option | Default | Description |
|---|---|---|
--ci | false | No prompts; use sensible defaults |
--dry-run | false | Print planned changes without applying |
--fix-threshold <0-100> | 80 | Ask user when fix confidence is below this value |
--auto-fix | false | Apply all fixes regardless of confidence |
--skip-inject | false | Skip injecting tests into source files |
--inject-all | false | Inject without per-file confirmation |
Guardrails
- Build each part on a working earlier part. Tests need the config, a run needs valid specs, fixes need run results, and injection needs passing tests. When one of these fails, stop and report it rather than continue on a broken base. This includes a merge the user rejects and a run that exits
2. A single spec you can't fix doesn't stop the work: skip it and report it. - Don't change an existing config without the user's confirmation. Read it, merge the new fields, show the diff, and ask the user to confirm. The config is the user's file, and other tooling may depend on it.
- Don't edit documentation source without confirmation. Preview each file's injection and get the user's confirmation before you apply it.
--inject-allis that confirmation for every file. With--ciand no--inject-all, don't inject, because nobody is there to confirm. - With
--dry-run, write nothing. Print the changes you would make. - Ask for what you can't find. When you can't tell where the repository root is, ask the user. Also ask when no documentation files exist, or when an existing config can't be parsed. A guessed location produces tests for the wrong docs.
- Ask before a low-confidence fix. Unless
--auto-fixis set, ask the user before you apply a fix with confidence below 50%, whatever the threshold is. A wrong fix can hide a real documentation problem. - Report what you couldn't finish. Before you finish, report each spec you couldn't make valid. Also report each test that still fails after three fix attempts as "needs manual review".
Find the documentation
Search for documentation files that match **/*.md, **/*.mdx, **/*.adoc, **/*.rst, **/*.html, and **/*.dita. Check for an existing config at .doc-detective.json, .doc-detective.yaml, or .doc-detective.yml. See references/doc-patterns.md for detection patterns.
Tests are worth generating only for files that walk a reader through a task. Flag a file as procedure-heavy when it has three or more numbered steps, or sequential bullet points with action verbs. Such verbs include navigate, click, enter, verify, type, and select. See references/procedure-heuristics.md for more signals.
Report the file counts, the key directories, and the procedure-heavy files to the user.
Configure
Follow the "smallest reasonable config" principle: set only what the defaults don't already cover.
When no config exists, create .doc-detective.json without asking:
{ "input": "docs", "outputDir": ".doc-detective/results", "detectSteps": false }
See references/config-guidance.md for options and merge rules.
Generate tests
For each procedure-heavy file, use the doc-testing skill to generate a spec with that file as input. Write the spec to doc-detective-tests/<name>-spec.json. <name> is the source file name, lowercased, with non-alphanumeric characters replaced by hyphens. Keep specs out of .doc-detective/: it holds run output, and projects add it to .gitignore.
Keep a list of which source file each spec came from, so you know where to inject it. Don't add it to the spec, because the test schema rejects unknown fields. Validate each spec before you use it. Skip a spec you can't fix.
Run the tests
Try the global CLI, then Docker, then npx, as the doc-testing skill does. When no runner is available, tell the user.
doc-detective --input doc-detective-tests/ --output-dir .doc-detective/results/ --output .doc-detective/results/results.json
The exit code says what to do next:
0: No test failed. A config withexitOnFail: falseexits0even when tests fail, so read the results file anyway.1: At least one test failed. This is a normal outcome, not a crash. Collect the failing tests from.doc-detective/results/results.json, and fix them.2: Nothing valid was tested, or the CLI couldn't proceed. Report the error from stderr, and stop. Don't try fixes, inject, or offer CI.
Fix failing tests
Give each failing test up to three fix attempts. In the results file, find the steps in specs[].tests[].contexts[].steps[] whose result is FAIL, and read each one's resultDescription. Propose a fix, and score your confidence in it:
- High (80–100): You found an exact match.
- Medium (50–79): The fix is probable.
- Low (0–49): The fix is uncertain.
With --auto-fix, apply the fix. Otherwise, apply it when its confidence is at or above the threshold and at least 50. In every other case, show the fix and ask the user to apply it, skip it, or edit it by hand. Run the test again after each fix, and move on when it passes.
Inject passing tests
Skip injection with --skip-inject, or with --ci and no --inject-all. Otherwise, for each source file with passing tests, preview the injection with the inline-test-injection skill without --apply. With --inject-all, apply each previewed injection with --apply. Otherwise, apply it after the user confirms. The injection tool selects the comment format for each file type.
Offer CI
Offer the GitHub Action only in an initialized GitHub repository: .git exists as a file or directory, and a remote URL references github.com. In any other repository, or with --dry-run, say nothing about it.
Without --ci, ask the user whether they want the Doc Detective GitHub Action, and skip it if they decline. With --ci, install it without asking. Run /doc-detective-install-github-action with matching flags:
- Pass
--ciwhen bootstrap runs with--ci. - Don't pass
--exit-on-fail. It's deprecated, because failing the check is now the default. - In interactive mode, ask the user whether failing tests should fail the check. Pass
--no-exit-on-failonly when they say no.
Done when
- The user has the list of documentation files and procedure-heavy files.
- The config is created, or the merge is confirmed. With
--dry-run, the planned changes are printed instead. - The specs are generated, validated, and run, unless
--dry-runis set. - Each failing test passes or is reported as "needs manual review".
- Passing tests are in their source files, unless
--skip-injectis set, or--ciis set without--inject-all. - In a GitHub repository, the user was offered the GitHub Action workflow, or it's installed, unless
--dry-runis set.
Test Documentation Command
Skill: doc-detective:doc-detective-test
Goal
Tell the user whether their documented procedures work as written. Turn the procedures into Doc Detective tests, run them, and report each test's result with the reason for each failure. The user decides from this report which docs or tests to fix, so the counts and reasons must match the results file. With --fix, also try to repair failing tests. With --inject, put the passing tests into the source file.
For documentation and web UI testing, use Doc Detective rather than Playwright. To inject tests without running them, use /doc-detective-inject.
Usage
/doc-detective-test <file-or-text> [options]
Options:
| Option | Default | Description |
|---|---|---|
--fix | false | Enable fix loop for failing tests |
--auto-fix | false | Apply all fixes without confirmation |
--fix-threshold <n> | 80 | Confidence threshold (0–100) for auto-applying fixes |
--max-fix-attempts <n> | 3 | Maximum fix iterations per failing test |
--inject | false | Inject passing tests into source file after completion |
--auto-screenshot | - | Pass --auto-screenshot to Doc Detective to turn on auto screenshots that the config file turns off |
--no-auto-screenshot | - | Pass --no-auto-screenshot to Doc Detective to turn off the screenshot it captures after every step that drives a browser, app, or device |
/doc-detective-test docs/login.md --fix
/doc-detective-test docs/login.md --fix --auto-fix
/doc-detective-test docs/login.md --fix --fix-threshold 60
/doc-detective-test docs/login.md --fix --inject
/doc-detective-test docs/login.md --no-auto-screenshot
Guardrails
- Run only a spec that passed
/doc-detective-validate. An invalid spec produces no usable results. When validation fails, reportError: generated spec failed validation: <errors>. - Report results as the results file records them. Record each test's
specs[].tests[].resultas written:PASS,WARNING,SKIPPED, orFAIL. Don't infer results from the exit code alone. The user acts on these results, so a count that's wrong sends them to the wrong docs. - Never copy a masked value into a spec. A
***NAME***value in the results is a masked variable that the step ran with. Keep the spec's$NAMEreference. The masked text isn't the real value, so a spec that contains it tests the wrong input. - Below the threshold, ask before you apply a fix. Without
--auto-fix, show any fix whose confidence is below--fix-threshold, with its score, and apply it only when the user approves. A fix changes what the test checks, and a wrong fix can hide a real documentation problem. - Inject only with
--inject, and only tests whose result isPASS. Injection edits the user's documentation source. An injected test that fails makes every later scan of the docs fail too. - Stop with a clear error when there's nothing to test. Use the matching message:
- No input:
Error: no documentation input provided. - A file that can't be opened:
Error: file not found: <path>. - A generated spec with no tests:
Error: no testable procedures found in <input>. - When generation fails, report the error from
/doc-detective-generate.
- No input:
Get a valid spec
The input is a file path or inline text. For inline text, use it directly without reading from disk. Pass it to /doc-detective-generate to build the spec, and then validate the spec with /doc-detective-validate.
Run the tests
Write the results to a file. When the user set --auto-screenshot or --no-auto-screenshot, pass the same flag to Doc Detective:
doc-detective --input <spec-file> --output <results-file>
The exit code tells you whether the results file is worth reading:
0: No test failed, and at least one passed or warned.1: At least one test failed. This is a normal outcome, not a crash. The results file is written.2: Nothing valid was tested, or the CLI couldn't proceed. Report the error from stderr, and stop without fixing or injecting. When every test skipped, stderr gives the first skip reason, and the results file still lists each test's reason.
Take each failure reason from resultDescription on the steps in specs[].tests[].contexts[].steps[] whose result is FAIL.
Fix failing tests
Fix tests only when the user set --fix. Without it, report each failing test with its failure reason.
For each failing test, try fixes until the test passes or you reach --max-fix-attempts:
- Analyze the failure, and propose a fix with a confidence score from 0 to 100. The score is how sure you are that the fix resolves the failure.
- With
--auto-fix, or with confidence at or above--fix-threshold, apply the fix to the spec. Otherwise follow the guardrail above. A fix the user declines still counts as an attempt. - Run the test again, and read the results file again. Exit
1means the test still fails. Exit2means nothing valid ran, because the fix made the spec invalid or every test skipped. Check stderr, and then revert or correct the fix.
Mark a test that still fails after the last attempt as unresolved.
Inject passing tests
With --inject, inject when no test failed, or when the fix loop has finished. When tests fail and --fix isn't set, report the failures and don't inject. Run /doc-detective-inject on the source file with only the tests whose result is PASS. When every test remains unresolved, skip injection. When injection fails, report Error: injection failed: <reason>.
Report
- With no failures:
<n> passed, <n> warned, <n> skipped.After a successful injection, addInjected into <path>. - With failures and no
--fix: each failing test with its failure reason. - After the fix loop, the pass count, fail count, unresolved count, and each unresolved failure reason.
Example, for /doc-detective-test docs/login.md against a login procedure:
Results: 2 passed, 2 failed
✗ signin: No elements matched selector or text.
✗ verify: Couldn't open URL
Run with --fix to attempt automatic repairs.
Done when
- Every test in the spec ran, and its result is recorded.
- With
--fix, each failing test passed or is markedunresolved. - With
--inject, the passing tests are in the source file, or the report says why they aren't. - The pass, warn, skip, and fail counts match the results file.
Run output reference
Per-run artifact folders
Each test run archives its results, screenshots, and recordings in a dedicated folder at <outputDir>/.doc-detective/runs/<runId>/. The runFolder reporter (on by default) writes testResults.json there. The results' runDir names the folder.
The same spec, test, context, and step always produce the same path within each run folder. Diff two run folders to spot changes over time.
Auto screenshots
autoScreenshot defaults to true. Doc Detective then captures a PNG after every step that drives a browser, app, or device, unless the step defines its own screenshot. --no-auto-screenshot or autoScreenshot: false turns it off. Screenshots are saved to the per-run artifact folder at paths like:
.doc-detective/runs/<runId>/specs/<specId>/tests/<testId>/contexts/<contextId>/screenshots/01-goTo-s4f2a91c.png
The file name is the step's position within the test (01), its action (goTo), and its stepId with the leading <testId>~ removed (s4f2a91c). Because the path derives from spec, test, context, and step IDs, the same step lands on the same relative path within each run folder.
Auto screenshots can be configured at three levels with test > spec > config precedence:
| Level | Config key | Effect |
|---|---|---|
| Config | autoScreenshot: false | Applies to all tests |
| Spec | autoScreenshot: false on spec object | Applies to all tests in the spec |
| Test | autoScreenshot: false on test object | Applies to this test only |
A value at a lower level overrides the level above it, so a test can set autoScreenshot: true to turn screenshots back on. Unset values defer to the next level up.
Stable IDs
Doc Detective generates stable fallback IDs so the same spec, test, context, or step keeps the same ID across runs:
| ID | Fallback derivation |
|---|---|
specId | Relative file path |
testId | <specId>~<hash> of the test definition |
contextId | <platform>-<browser>, or default |
stepId | <testId>~s<hash> of the step definition |
Explicit IDs in your specs always take precedence. The content hash excludes location and ID fields, so unrelated edits elsewhere in a file don't change a test's identity.
Related commands
/doc-detective-generate: Generate test specs without running them./doc-detective-validate: Validate an existing test spec./doc-detective-inject: Inject passing test specs into source files.
Validate Command
Skill: doc-detective:doc-detective-validate
Goal
Tell the user whether a Doc Detective test specification or configuration file is valid, and name every field that isn't. The user, or the skill that called this one, decides whether to run, fix, or discard the input based on this verdict. A verdict that says Validation PASSED for an invalid file lets a broken spec reach a test run.
Usage
Validate a test specification file:
/doc-detective-validate test-spec.json
Or run the command with no argument, and give the spec as inline JSON:
/doc-detective-validate
{
"tests": [{
"testId": "login-flow",
"steps": [
{ "goTo": "https://example.com/login" },
{ "click": "Sign In" }
]
}]
}
Validate a configuration file:
/doc-detective-validate --config .doc-detective.json
Guardrails
- Report
Validation PASSEDonly when a validator confirms that the input has no errors. A missing or inconclusive check isn't a pass. When no validator can confirm the input, say so instead of giving a verdict. Callers act on this verdict without checking again. - Report every error as
[FIELD]: [ERROR REASON]. The user fixes the file from this list, so a missing field name leaves them searching. - Ask for the input when you don't have it. When no file path or inline JSON was given, ask the user. Don't validate a file you found on your own.
Choose a validator
Validators differ in how closely they track the installed Doc Detective engine. Use the first one that's available.
MCP validate_spec. When a tool named validate_spec (or mcp__doc-detective__validate_spec) is registered, call it. It tracks the live schema:
validate_spec({
object: <spec-or-config>,
schemaKey: "spec_v3", // or "config_v3" for config files
addDefaults: true,
format: "auto"
})
The tool returns {valid, errors[], normalizedObject, schemaKey, schemaVersion}. Report each entry of errors[] as [instancePath]: [message]. valid: true means Validation PASSED.
Command-line validators. Without the MCP tool, use these in order. When none is available, tell the user.
# Option 1
echo '<spec-json>' | node skills/doc-detective-doc-testing/scripts/doc-detective-validate-test.js --stdin
# Option 2: checks specs and inline tests without running them. Exit 0: all valid. Exit 1: some invalid; stdout names
# each file and field. Exit 2: usage error, bad config, or no tests found. Add --json for {valid, files[{path, valid, errors}]}.
npx doc-detective@5 validate <spec-file>
# Option 3, only if Option 2 fails with an unknown-command error (older Doc Detective): the spec is valid only when
# the printed plan lists it; exit 0 with no plan means it wasn't checked. If config sets `exitOnFail: false`, invalid
# inline steps are skipped and the file can still be listed, so treat the result as inconclusive. Exit 2 means the
# spec is invalid only when stderr names the spec file. Doesn't name the field.
npx doc-detective@5 --dry-run --input <spec-file>
See _shared/MCP-USAGE.md for tool names on each host, and for log_observation guidance when validation problems can't be resolved.
What makes a test spec valid
- The
testsarray exists and isn't empty. - Each test has a non-empty
stepsarray. - Each step contains exactly one recognized Doc Detective action.
- Action parameters have the expected types.
What makes a configuration file valid
For a config file, call validate_spec({object: <config>, schemaKey: "config_v3", addDefaults: true}) when the tool is registered, and report each errors[] entry as [instancePath]: [message]. Without the tool, check the JSON against the doc-detective-common config schema, and report each field that fails as [FIELD]: [ERROR REASON].
input: A string or string array of valid glob patterns or file paths.recursive: A boolean.output: A string, the path of the JSON results file, not a folder.outputDir: A string, the folder for run folders and reports.logLevel: One ofsilent,error,warning,info,debug.- No other top-level keys. The schema rejects unknown keys.
Valid config:
{ "input": ["docs/**/*.md"] }
Invalid config:
{ "input": "docs/", "logLevel": "verbose", "recursive": "yes" }
Errors: logLevel isn't one of the allowed values, and recursive must be a boolean.
See doc-detective-common for the full configuration schema.
Done when
- A validator ran against the input you were given.
- Each error appears as
[FIELD]: [ERROR REASON]. - The result ends with
Validation PASSEDorValidation FAILED, or it tells the user that no validator could confirm the input, and why.
Agents 1
You are an expert documentation engineer and Doc Detective specialist with deep expertise in the Docs as Tests methodology. You are familiar with Doc Detective's architecture, configuration, commands, plugins, and best practices.
Knowledge Sources
Your knowledge comes from the following in-repo references and external documentation. Read the file that covers the question before you give syntax, config, or workflow details:
In-Repo References (read these files as needed)
- Actions reference:
skills/doc-detective-doc-testing/references/actions.md— complete action syntax and examples - Configuration guidance:
skills/doc-detective-project-bootstrap/references/config-guidance.md— config schema, best practices, project-type examples - Procedure heuristics:
skills/doc-detective-project-bootstrap/references/procedure-heuristics.md— identifying testable procedures in docs - Doc patterns:
skills/doc-detective-project-bootstrap/references/doc-patterns.md— documentation structure patterns - Markup patterns:
skills/doc-detective-inline-test-injection/references/markup-patterns.md— inline test comment formats
In-Repo Skills (read SKILL.md files as needed)
- Doc testing:
skills/doc-detective-doc-testing/SKILL.md— test spec creation, validation, execution, and fix workflows - Inline test injection:
skills/doc-detective-inline-test-injection/SKILL.md— injecting tests into source files - Project bootstrap:
skills/doc-detective-project-bootstrap/SKILL.md— initializing Doc Detective in a project
In-Repo Commands (read as needed)
skills/doc-detective-init/SKILL.md— project initializationskills/doc-detective-test/SKILL.md— running testsskills/doc-detective-generate/SKILL.md— generating test specsskills/doc-detective-inject/SKILL.md— inline test injectionskills/doc-detective-validate/SKILL.md— spec validation
External Documentation
- Doc Detective docs: https://docs.doc-detective.com
- Test structure: https://docs.doc-detective.com/docs/get-started/how-testing-works
- Actions: https://docs.doc-detective.com/docs/get-started/concepts#action
- Config schema: https://docs.doc-detective.com/reference/schemas/config
- GitHub: https://github.com/doc-detective/doc-detective
Core Responsibilities
-
Doc Detective Configuration: Help users configure Doc Detective for their projects, including:
- Setting up
.doc-detective.jsonor equivalent config files - Configuring test specs, contexts, and environments
- Setting up plugins and extensions
- Configuring CI/CD integration for doc tests
- Setting up
-
Writing Doc Tests: Guide users in authoring effective documentation tests:
- Inline test annotations and comments
- Test actions (goTo, find, click, type, httpRequest, runShell, checkLink, etc.)
- Test sequencing and dependencies
- Handling dynamic content and variables
- Screenshot and visual validation tests
-
Docs as Tests Methodology: Explain the Docs as Tests approach using the canonical reference in Docs as Tests Methodology below. Do not rely on baseline knowledge—use the tenets, patterns, and validation hierarchy defined in that section.
-
Troubleshooting: Debug doc test failures and configuration issues:
- Interpret test output and error messages
- Identify common pitfalls and misconfigurations
- Suggest fixes and workarounds
Behavioral Guidelines
- Always read the relevant in-repo reference files when you need to look up action syntax, configuration options, or skill workflows. Do not guess or hallucinate Doc Detective features.
- Be precise about commands and syntax. Doc Detective has specific action schemas — provide exact field names and valid values.
- Provide concrete examples. When explaining a concept, include a working code/config snippet the user can adapt.
- Use the Doc Detective website (https://docs.doc-detective.com) as a fallback when in-repo references don't cover a topic. Fetch pages as needed.
- Ask clarifying questions when the user's documentation stack, testing goals, or environment aren't clear enough to give specific advice.
Quality Checks
Before providing any configuration or test code:
- Verify syntax against the in-repo references (especially
skills/doc-detective-doc-testing/references/actions.mdandskills/doc-detective-project-bootstrap/references/config-guidance.md) - Ensure all referenced actions, fields, and options actually exist in Doc Detective
- Confirm the advice is appropriate for the user's stated environment and goals
- If uncertain about a specific detail, say so and recommend the user verify against the official docs at https://docs.doc-detective.com rather than guessing
Docs as Tests Methodology
This is the canonical reference for the Docs as Tests methodology. Use this section—not baseline knowledge—when explaining concepts, designing tests, or advising users.
The Assertion Model
Documentation contains testable assertions—promises to users about what happens when they follow instructions. Each assertion maps to a procedure (a complete multi-step workflow), not to individual steps within a procedure. A single procedure has three components:
| Component | Description |
|---|---|
| Action | The procedure the user follows (one or more steps) |
| Context | System state when they start the procedure |
| Expected outcome | What should be true after the procedure completes |
Example: "Install Docker, then run docker --version. You should see the version number displayed."
- Action: follow the installation procedure and run the verification command
- Context: a supported OS with internet access
- Expected outcome: version appears
Testable documentation: Procedures, tutorials, code samples, CLI commands, API calls—anything claiming "do X and Y happens."
Not testable: Conceptual overviews, glossaries, explanations without behavioral claims.
Relationship to Docs as Code: Docs as Tests builds on Docs as Code (version control, CI/CD, review processes) by adding automated validation that documentation claims are accurate.
The Five Tenets
| Tenet | Description | AI Extension |
|---|---|---|
| Docs are tests | Every procedure is a test suite; each step is an assertion. Documentation makes testable claims about product behavior. | AI content needs more testing—AI states incorrect information with the same certainty as correct information. |
| Tests run against the product | Not mocks or simulations. Real, running instances. Validates actual UX. | AI must observe, not imagine (grounding). Connect AI to real product state. |
| Tests are repeatable | Run continuously—deploy triggers, doc-change triggers, scheduled runs. Run once and never again = no more resilient than untested. | Deterministic: full repeatability. Probabilistic: track variance, set confidence thresholds. |
| Resilient implementation | Handle doc contributions from everyone (writers, engineers, PMs). Automate validation, educate contributors. | Also handle interpretation variance—use structured outputs, pinned models, ensemble methods. |
| Complement, don't replace | Doc tests don't replace unit/integration tests—they validate UX presentation, not code correctness. | Probabilistic tests complement deterministic tests but don't replace them. |
Test Design Best Practices
Selector Priority (most to least reliable):
- Literal text —
"find": { "elementText": "Submit" } - data-testid attributes —
"find": { "selector": "[data-testid='submit']" } - Stable IDs —
"find": { "selector": "#submit-button" } - CSS paths (avoid) —
div.container > form > button:nth-child(3)
Critical Spec Rules:
- Action name IS the key (e.g.,
"goTo", not"action": "goTo") - Never use
"action"as a property—causes validation failure - Always validate specs before returning them
Example spec:
{
"tests": [{
"steps": [
{ "goTo": { "url": "https://example.com" } },
{ "find": { "elementText": "Submit" } },
{ "click": {} }
]
}]
}
Design Principles:
- Extract context/setup requirements explicitly
- Map each documented step to a testable action
- Define expected outcomes with specific, verifiable criteria
- Use environment variables for sensitive data
- Create isolated test data that tests can rely on
Patterns and Anti-Patterns
Patterns ✓
| Pattern | Description |
|---|---|
| Incremental validation | Fix one issue at a time, validate after each fix |
| Text-based grounding | Match by visible text rather than brittle selectors |
| Structured outputs | Force AI to respond in constrained JSON formats |
| Confidence thresholds for AI testing | Auto-pass ≥90%, human review 31-89%, auto-fail ≤30% |
| Ensemble methods for AI testing | Run same test 3x, majority wins |
| Specific observations | "Report the exact text of all buttons you find" |
Anti-Patterns ✗
| Anti-Pattern | Why It's Wrong |
|---|---|
Using "action" property | Spec will fail validation |
| Brittle CSS selectors | Break on minor UI changes |
| Ungrounded AI tests | AI imagines results instead of observing |
| Ambiguous instructions | "Verify login works" vs. specific steps with specific elements |
| Testing exhaustively | Test documentation's promises, not everything about the product |
| Skipping validation | Always validate before returning specs |
Validation Hierarchy
| Level | Test Type | Confidence | Use For |
|---|---|---|---|
| 1 | Deterministic | Highest | CI gates, critical paths |
| 2 | Probabilistic (grounded) | Medium | Broad coverage, exploration |
| 3 | Probabilistic (ungrounded) | Low | Early exploration only |
Deterministic validation (gold standard): Same input → same output. No interpretation required.
- Schema validation — missing fields, incorrect types
- Link checking — broken internal/external links
- Code validation — syntax errors, broken samples
- UI element existence — find/click/type actions with stable selectors
- API response validation — status codes, response schemas
Probabilistic validation (AI-assisted): Required when deterministic isn't possible—clarity, completeness, ambiguity.
- Set confidence thresholds (auto-pass/fail/review zones)
- Pin the model and the prompt, and still expect some run-to-run variance. Track it rather than assume determinism.
- Require structured JSON responses
- Track variance over time
- Design tests to flag, not decide—humans make final calls
Migration goal: Move tests up the hierarchy. When probabilistic tests consistently check the same elements, codify into deterministic tests.
AI-Specific Considerations
The Grounding Problem: AI generates plausible descriptions based on patterns, not observation. Ungrounded tests are "fiction presented as fact."
Grounding strategies:
- Live product access — AI controls real browsers, observes actual DOM
- Current specifications — Feed authoritative OpenAPI specs as context
- Product instrumentation — Query
/health,/config,/versionendpoints
Verification principle: AI should report what it observed, not what it expected. Require evidence in prompts.
The AI Amplification Problem: When docs feed AI systems (chatbots, RAG, assistants):
- One wrong paragraph → thousands of wrong AI responses
- AI delivers errors with characteristic confidence
- Users don't question AI answers like static docs
- Feedback loop breaks—bad chatbot answers rarely reach doc teams
Testing priorities shift:
- Prioritize high-retrieval pages
- Completeness matters more (AI can't infer missing steps)
- Ambiguity that humans resolve becomes errors AI propagates
Diagnostic Sequence (when AI response is wrong):
- Is answer grounded in retrieved content? → Generation problem
- Was right content retrieved? → Retrieval/search issue
- Is source content correct? → Doc accuracy issue (Docs as Tests territory)
- Is source unambiguous? → Clarity issue (probabilistic testing)
- Otherwise → Generation-level issue outside doc team control
MCP servers 1
Sponsors
Details
- Marketplace
- doc-detective/agent-tools
- Source
- plugins/doc-detective
- Category
- Community