Official SHAFT agent skills: methodology router, locator strategy, test planning and authoring, MCP-driven recording, failure analysis, and change verification.

GitHub stars
422

Install

/plugin install shaft-skills@shafthq

Run it inside Claude Code after adding the marketplace with /plugin marketplace add ShaftHQ/SHAFT_ENGINE.

Contents

Discovered from source

Agents 82

SHAFT skill inventory

Generated by scripts/ci/shaft_skill_quality.py --write; do not edit manually.

SkillCanonical sourcePackage pathPackageSupported clientsDescriptionLinked local references
shaft-accessibility-testingshaft-skills/shaft-accessibility-testing/SKILL.mdskills/shaft-accessibility-testing/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when testing web or mobile accessibility for WCAG conformance, keyboard and focus behavior, semantics, errors, or user impact.shaft-accessibility-testing/references/playbook.md; shaft-developer/SKILL.md
shaft-android-apk-selectshaft-skills/shaft-android-apk-select/SKILL.mdskills/shaft-android-apk-select/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when choosing a universal or ABI-split APK for a local emulator versus a real or cloud arm64 Android device.shaft-android-apk-select/references/playbook.md; shaft-browserstack-native/SKILL.md; shaft-developer/SKILL.md; shaft-mobile-env/SKILL.md
shaft-api-testingshaft-skills/shaft-api-testing/SKILL.mdskills/shaft-api-testing/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when implementing or repairing SHAFT.API tests for HTTP requests, authentication, payloads, schemas, responses, contracts, or service workflows.shaft-api-testing/references/playbook.md; shaft-developer/SKILL.md
shaft-assertionsshaft-skills/shaft-assertions/SKILL.mdskills/shaft-assertions/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when selecting or repairing SHAFT hard assertions, soft verifications, browser or element oracles, API checks, object checks, or failure messages.shaft-assertions/references/playbook.md; shaft-developer/SKILL.md
shaft-automated-test-authoringshaft-skills/shaft-automated-test-authoring/SKILL.mdskills/shaft-automated-test-authoring/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when implementing or repairing executable SHAFT Java tests for web, mobile, API, database, CLI, TestNG, JUnit, or Cucumber behavior.shaft-automated-test-authoring/references/playbook.md; shaft-developer/SKILL.md
shaft-browserstack-nativeshaft-skills/shaft-browserstack-native/SKILL.mdskills/shaft-browserstack-native/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when configuring BrowserStack App Automate for SHAFT native Android or iOS: secrets keys, device pin, bs:// apps, Local, or triage.shaft-android-apk-select/SKILL.md; shaft-browserstack-native/references/playbook.md; shaft-developer/SKILL.md; shaft-ios-ipa-acceptance/SKILL.md; shaft-mobile-env/SKILL.md
shaft-change-verificationshaft-skills/shaft-change-verification/SKILL.mdskills/shaft-change-verification/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when edited SHAFT Java needs a reuse plan, preview diff, guardrail check, approved application, focused test, and evidence-backed verdict.shaft-change-verification/references/playbook.md; shaft-developer/SKILL.md
shaft-clishaft-skills/shaft-cli/SKILL.mdskills/shaft-cli/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when a SHAFT task needs a deterministic one-shot command, machine-readable output, offline discovery, or code generation from a persisted recording.shaft-cli/references/cache.md; shaft-cli/references/playbook.md; shaft-developer/SKILL.md
shaft-database-testingshaft-skills/shaft-database-testing/SKILL.mdskills/shaft-database-testing/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when implementing or repairing SHAFT.DB checks for persisted state, queries, rows, transactions, cleanup, or database-backed integration behavior.shaft-database-testing/references/playbook.md; shaft-developer/SKILL.md
shaft-defect-reportingshaft-skills/shaft-defect-reporting/SKILL.mdskills/shaft-defect-reporting/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when reporting a confirmed SHAFT product defect with severity, impact, environment, reproduction steps, expected and actual behavior, and evidence links.shaft-defect-reporting/references/playbook.md; shaft-developer/SKILL.md
shaft-developershaft-skills/shaft-developer/SKILL.mdskills/shaft-developer/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when starting any SHAFT testing task to select exactly one lifecycle, implementation, or tool specialist before doing the work.shaft-developer/references/routing.md; shaft-developer/references/sources.md
shaft-docs-hygieneshaft-skills/shaft-docs-hygiene/SKILL.mdskills/shaft-docs-hygiene/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when keeping SHAFT project docs lean: SoT trio, Done/In progress/Not started, hours math, adversarial review, or delete-guard.shaft-developer/SKILL.md; shaft-docs-hygiene/references/playbook.md
shaft-execution-reportingshaft-skills/shaft-execution-reporting/SKILL.mdskills/shaft-execution-reporting/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when reporting SHAFT execution to engineers with reconciled counts, coverage, environments, failures, defects, flakes, evidence, risks, and next actions.shaft-developer/SKILL.md; shaft-execution-reporting/references/playbook.md
shaft-failure-analysisshaft-skills/shaft-failure-analysis/SKILL.mdskills/shaft-failure-analysis/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when investigating consistently failed SHAFT results through evidence, controlled reproduction, hypotheses, and root-cause ownership.shaft-developer/SKILL.md; shaft-failure-analysis/references/playbook.md
shaft-flaky-test-analysisshaft-skills/shaft-flaky-test-analysis/SKILL.mdskills/shaft-flaky-test-analysis/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when SHAFT tests inconsistently pass or fail under equivalent conditions and need history, retry, timing, state, data, or environment analysis.shaft-developer/SKILL.md; shaft-flaky-test-analysis/references/playbook.md
shaft-fluent-apishaft-skills/shaft-fluent-api/SKILL.mdskills/shaft-fluent-api/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when composing or repairing SHAFT fluent browser, element, touch, assertion, verification, or report-message chains.shaft-developer/SKILL.md; shaft-fluent-api/references/playbook.md
shaft-guide-searchshaft-skills/shaft-guide-search/SKILL.mdskills/shaft-guide-search/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when current official SHAFT syntax, configuration, migration, troubleshooting, or feature guidance must be found and cited before coding.shaft-developer/SKILL.md; shaft-guide-search/references/playbook.md
shaft-ios-ipa-acceptanceshaft-skills/shaft-ios-ipa-acceptance/SKILL.mdskills/shaft-ios-ipa-acceptance/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when gating an iOS IPA before BrowserStack or cloud upload: iphoneos, signing, arm64, thinning, or Info.plist checks.shaft-browserstack-native/SKILL.md; shaft-developer/SKILL.md; shaft-ios-ipa-acceptance/references/playbook.md
shaft-locator-designshaft-skills/shaft-locator-design/SKILL.mdskills/shaft-locator-design/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when identifying, ranking, repairing, or generating resilient SHAFT web or mobile locators from current DOM, ARIA, or accessibility-tree evidence.shaft-developer/SKILL.md; shaft-locator-design/references/playbook.md
shaft-mcpshaft-skills/shaft-mcp/SKILL.mdskills/shaft-mcp/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when a SHAFT task needs structured tool calls, interactive exploration, or browser, mobile, driver, and capture state that persists across calls.shaft-developer/SKILL.md; shaft-mcp/references/cache.md; shaft-mcp/references/playbook.md
shaft-mobile-actionsshaft-skills/shaft-mobile-actions/SKILL.mdskills/shaft-mobile-actions/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when implementing SHAFT native or mobile-web sessions, context switches, taps, gestures, rotation, keyboard, or app lifecycle behavior.shaft-developer/SKILL.md; shaft-mobile-actions/references/playbook.md
shaft-mobile-envshaft-skills/shaft-mobile-env/SKILL.mdskills/shaft-mobile-env/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when setting up or repairing a local SHAFT mobile env: JDK split, Appium pin, SDK/AVD, emulator memory, or doctor loop.shaft-android-apk-select/SKILL.md; shaft-browserstack-native/SKILL.md; shaft-developer/SKILL.md; shaft-ios-ipa-acceptance/SKILL.md; shaft-mobile-env/references/playbook.md
shaft-mobile-retest-hygieneshaft-skills/shaft-mobile-retest-hygiene/SKILL.mdskills/shaft-mobile-retest-hygiene/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when triaging SHAFT mobile retest failures: map no-ops, picker probes, mixed bytecode, progress journals, or swipe fallback.shaft-developer/SKILL.md; shaft-failure-analysis/SKILL.md; shaft-mobile-actions/SKILL.md; shaft-mobile-retest-hygiene/references/playbook.md
shaft-nonfunctional-test-designshaft-skills/shaft-nonfunctional-test-design/SKILL.mdskills/shaft-nonfunctional-test-design/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when designing measurable risk-based tests for performance, security, reliability, recovery, compatibility, scalability, usability, or operations.shaft-developer/SKILL.md; shaft-nonfunctional-test-design/references/playbook.md
shaft-page-objectsshaft-skills/shaft-page-objects/SKILL.mdskills/shaft-page-objects/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when creating or refactoring reusable SHAFT page objects, component objects, locators, and user-intent methods without duplicating owners.shaft-developer/SKILL.md; shaft-page-objects/references/playbook.md
shaft-recording-codegenshaft-skills/shaft-recording-codegen/SKILL.mdskills/shaft-recording-codegen/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when a persisted SHAFT web, Playwright, mobile, or API recording must become compilable, repository-aware Java code or insertion blocks.shaft-developer/SKILL.md; shaft-recording-codegen/references/playbook.md
shaft-requirements-analysisshaft-skills/shaft-requirements-analysis/SKILL.mdskills/shaft-requirements-analysis/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when analyzing requirements for testability, acceptance criteria, traceability, contradictions, omissions, risks, assumptions, or coverage gaps.shaft-developer/SKILL.md; shaft-requirements-analysis/references/playbook.md
shaft-stakeholder-reportingshaft-skills/shaft-stakeholder-reporting/SKILL.mdskills/shaft-stakeholder-reporting/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when translating verified SHAFT coverage, outcomes, defects, trends, and residual risk into business quality or release decisions.shaft-developer/SKILL.md; shaft-stakeholder-reporting/references/playbook.md
shaft-test-case-designshaft-skills/shaft-test-case-design/SKILL.mdskills/shaft-test-case-design/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when designing reviewable test cases or scenarios with traceability, data, steps, expected results, techniques, priorities, and boundary coverage.shaft-developer/SKILL.md; shaft-test-case-design/references/journey-coverage-map.md; shaft-test-case-design/references/playbook.md
shaft-test-completionshaft-skills/shaft-test-completion/SKILL.mdskills/shaft-test-completion/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when closing a SHAFT test cycle with exit-criteria evaluation, residual risk, archived testware, lessons, maintenance handover, or completion records.shaft-developer/SKILL.md; shaft-test-completion/references/playbook.md
shaft-test-data-designshaft-skills/shaft-test-data-design/SKILL.mdskills/shaft-test-data-design/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when designing deterministic, private, isolated SHAFT test data with partitions, factories, seeding, uniqueness, cleanup, retention, or schema rules.shaft-developer/SKILL.md; shaft-test-data-design/references/playbook.md
shaft-test-environmentshaft-skills/shaft-test-environment/SKILL.mdskills/shaft-test-environment/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when defining or troubleshooting SHAFT test environment readiness, parity, versions, configuration, dependencies, isolation, reset, or drift.shaft-developer/SKILL.md; shaft-test-environment/references/playbook.md
shaft-test-executionshaft-skills/shaft-test-execution/SKILL.mdskills/shaft-test-execution/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when executing approved SHAFT tests with controlled scope, preflight checks, deterministic configuration, evidence, status reconciliation, and cleanup.shaft-developer/SKILL.md; shaft-test-execution/references/playbook.md
shaft-test-monitoringshaft-skills/shaft-test-monitoring/SKILL.mdskills/shaft-test-monitoring/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when monitoring SHAFT test progress, coverage, quality gates, defects, risks, duration, flakiness, infrastructure health, or completion.shaft-developer/SKILL.md; shaft-test-monitoring/references/playbook.md
shaft-test-planningshaft-skills/shaft-test-planning/SKILL.mdskills/shaft-test-planning/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when creating or revising a risk-based SHAFT test plan covering objectives, scope, approach, resources, estimates, schedule, criteria, and monitoring.shaft-developer/SKILL.md; shaft-test-planning/references/playbook.md
shaft-test-recordingshaft-skills/shaft-test-recording/SKILL.mdskills/shaft-test-recording/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when an approved live web, Playwright, native-mobile, mobile-web, Appium Inspector, or API flow must be captured with privacy-safe evidence.shaft-developer/SKILL.md; shaft-test-recording/references/playbook.md
shaft-web-actionsshaft-skills/shaft-web-actions/SKILL.mdskills/shaft-web-actions/SKILL.mdshaft-skillsClaude Code; Codex CLIUse when implementing SHAFT WebDriver or Playwright navigation, browser state, element interaction, upload, drag-and-drop, window, cookie, or screenshot behavior.shaft-developer/SKILL.md; shaft-web-actions/references/playbook.md

SHAFT skill quality scorecard

Generated by scripts/ci/shaft_skill_quality.py --write; scores are reviewed in quality-review.json.

SkillTrigger precisionSibling overlapDependency depthReference freshnessRunnable evidenceContext costDescription charactersConfusion checksReview notes
shaft-accessibility-testing545444130shaft-nonfunctional-test-designWCAG and assistive-use triggers separate this specialist from broad nonfunctional design.
shaft-android-apk-select545555112shaft-mobile-env; shaft-ios-ipa-acceptanceABI vs universal APK choice for emulator versus arm64 cloud is a crisp artifact decision.
shaft-api-testing545454148shaft-automated-test-authoringHTTP request and response evidence gives this implementation specialist a clear boundary.
shaft-assertions445453150shaft-automated-test-authoringOracle and verification wording is precise, but the description needs context compression.
shaft-automated-test-authoring435455136shaft-change-verification; shaft-web-actionsExecutable Java test ownership is clear; modality specialists remain intentional siblings.
shaft-browserstack-native545544135shaft-mobile-env; shaft-ios-ipa-acceptanceApp Automate secrets, device pins, and bs:// app ids bound cloud native runs.
shaft-change-verification545554142shaft-automated-test-authoringDiff, guardrail, compile, and verdict triggers distinguish verification from authoring.
shaft-cli545454152shaft-mcpOne-shot machine-readable commands form the tested boundary against stateful MCP use.
shaft-database-testing545454150shaft-automated-test-authoringPersisted-state and query terms route database work without claiming general authoring.
shaft-defect-reporting545442157shaft-execution-reportingConfirmed product finding language is distinct, but the listing description is oversized.
shaft-developer435553131shaft-automated-test-authoringThe router intentionally overlaps every specialist and must select exactly one owner.
shaft-docs-hygiene545534130shaft-test-completion; shaft-stakeholder-reportingSoT trio, three-status vocabulary, and hours math are docs process, not cycle closure reports.
shaft-execution-reporting545444157shaft-stakeholder-reportingEngineering counts and evidence distinguish this report from decision-oriented summaries.
shaft-failure-analysis545454137shaft-flaky-test-analysisConsistent failure and root-cause ownership are the reviewed boundary against flakiness.
shaft-flaky-test-analysis545453147shaft-failure-analysisEquivalent-condition variance and retry history uniquely identify intermittent failures.
shaft-fluent-api545455120shaft-web-actions; shaft-assertionsFluent chain composition is narrow and concise despite adjacent web and assertion APIs.
shaft-guide-search555545141shaft-developerOfficial-current guidance and citation requirements create a unique retrieval trigger.
shaft-ios-ipa-acceptance545555121shaft-browserstack-native; shaft-android-apk-selectiphoneos Info.plist gate before upload is narrower than general BrowserStack config.
shaft-locator-design545455150shaft-page-objectsDOM and accessibility evidence distinguish locator design from object architecture.
shaft-mcp545445149shaft-cliStateful interactive tool calls are a crisp complement to one-shot CLI behavior.
shaft-mobile-actions545454139shaft-web-actionsNative contexts, gestures, and app lifecycle terms keep mobile action routing specific.
shaft-mobile-env545544123shaft-test-environment; shaft-browserstack-nativeLocal JDK/Appium/SDK/emulator doctor loop is distinct from cloud BrowserStack setup.
shaft-mobile-retest-hygiene545544128shaft-failure-analysis; shaft-mobile-actionsMap no-op triage, picker probes, and mixed-bytecode journals specialize mobile retest analysis.
shaft-nonfunctional-test-design445444150shaft-accessibility-testing; shaft-test-planningMeasurable quality attributes define the scope; accessibility has a deeper specialist.
shaft-page-objects545455142shaft-locator-designReusable object and user-intent method ownership separates architecture from locators.
shaft-recording-codegen545455140shaft-test-recordingPersisted recording input and compilable Java output distinguish transformation from capture.
shaft-requirements-analysis545443148shaft-test-planning; shaft-test-case-designPre-planning testability and gap analysis is clear, though the description needs compression.
shaft-stakeholder-reporting545445134shaft-execution-reportingBusiness decisions and residual risk distinguish this from engineering result detail.
shaft-test-case-design545444150shaft-test-planning; shaft-requirements-analysisConcrete scenarios, data, steps, and expected outcomes bound this design specialist.
shaft-test-completion545435154shaft-test-monitoring; shaft-stakeholder-reportingCycle closure (exit criteria, archive, testware decisions, handover) is distinct from live monitoring and business reporting.
shaft-test-data-design555444153shaft-database-testingIsolation, seeding, uniqueness, and cleanup identify data design rather than DB checks.
shaft-test-environment555445145shaft-test-executionReadiness, parity, versions, and drift route environment work before execution.
shaft-test-execution545454155shaft-test-monitoringRunning approved scope with preflight and cleanup differs from observing active runs.
shaft-test-monitoring545445140shaft-test-executionProgress, gates, health, and completion signals clearly indicate runtime monitoring.
shaft-test-planning545444154shaft-requirements-analysis; shaft-test-case-designScope, resources, estimates, schedule, and criteria distinguish planning from analysis.
shaft-test-recording545445144shaft-recording-codegenApproved live-flow capture and privacy-safe evidence separate recording from generation.
shaft-web-actions545454162shaft-mobile-actions; shaft-fluent-apiBrowser navigation and interaction primitives are specific despite fluent API overlap.

SHAFT skills pack

Humans: this folder is the published pack that teaches an agent to use SHAFT. The generated inventory is INVENTORY.md. The scorecard is QUALITY.md. Evaluation prompts are evaluation-prompts.md.

Agents: for product test work, load shaft-skills/shaft-developer/SKILL.md and follow that router. For work on this repository, load AGENTS.md at the repository root and follow ChaosEngine. This file is the human map and an index. Do not treat it, INVENTORY.md, or QUALITY.md as policy. INVENTORY.md and QUALITY.md are generated; do not edit them by hand.

SHAFT skills mindmap

Human map of the pack. An agent loads shaft-developer and that entrypoint selects exactly one skill. The other skills stay unloaded until that choice.

shaft-developer
├── Lifecycle
│   ├── shaft-requirements-analysis — testable requirements, gaps, traceability
│   ├── shaft-test-planning — strategy, scope, risk, schedule, entry and exit
│   ├── shaft-test-case-design — scenarios, steps, expected results
│   ├── shaft-test-data-design — datasets, factories, privacy, cleanup
│   ├── shaft-test-environment — browser, device, service, configuration readiness
│   ├── shaft-test-execution — run approved tests and keep the evidence
│   ├── shaft-test-monitoring — progress, gates, duration, retries
│   ├── shaft-test-completion — exit criteria, residual risk, archive, handover
│   ├── shaft-failure-analysis — Allure failure analysis of one consistent failure
│   ├── shaft-flaky-test-analysis — inconsistent pass/fail under the same conditions
│   ├── shaft-defect-reporting — a reproducible product defect
│   ├── shaft-execution-reporting — engineering execution report
│   ├── shaft-stakeholder-reporting — business quality or release decision
│   ├── shaft-accessibility-testing — WCAG, keyboard, assistive technology
│   └── shaft-nonfunctional-test-design — performance, security, reliability, recovery
├── Implementation
│   ├── shaft-automated-test-authoring — executable SHAFT test code
│   ├── shaft-page-objects — page and component objects
│   ├── shaft-fluent-api — fluent chains
│   ├── shaft-locator-design — rank and repair locators
│   ├── shaft-web-actions — browser and element actions
│   ├── shaft-mobile-actions — native and touch actions
│   ├── shaft-mobile-env — local JDK, Appium, SDK, emulator
│   ├── shaft-android-apk-select — universal or ABI-split APK
│   ├── shaft-ios-ipa-acceptance — IPA gate before cloud upload
│   ├── shaft-browserstack-native — BrowserStack App Automate
│   ├── shaft-mobile-retest-hygiene — mobile retest map, picker, bytecode, journal
│   ├── shaft-api-testing — HTTP API tests
│   ├── shaft-database-testing — persisted-state checks
│   ├── shaft-assertions — hard assertions and soft verifications
│   ├── shaft-guide-search — current official SHAFT guide
│   ├── shaft-test-recording — Record and playback of a live flow
│   ├── shaft-recording-codegen — Code generation from a persisted recording
│   ├── shaft-change-verification — preview, guardrail, apply, verify
│   └── shaft-docs-hygiene — plan, journey, and issue vocabulary
└── Tool caches (load only when selected)
    ├── shaft-mcp — compact MCP names and arguments; defer JSON schemas
    └── shaft-cli — compact CLI forms and arguments; do not use --help for discovery

How they relate

  • shaft-developer is the only decision surface. Finish one skill before returning for a different deliverable.
  • Allure failure analysis is shaft-failure-analysis. The sibling for results that pass and fail under the same conditions is shaft-flaky-test-analysis.
  • Record and playback is shaft-test-recording. Code generation from that persisted recording is shaft-recording-codegen. A generated edit then goes to shaft-change-verification.
  • shaft-automated-test-authoring owns test code that is not produced from a recording. It hands locator, page-object, fluent, web, mobile, API, database, and assertion detail to those specialists.
  • Local mobile setup starts at shaft-mobile-env, then shaft-android-apk-select or shaft-ios-ipa-acceptance, and uses shaft-browserstack-native for a cloud device.
  • shaft-mcp and shaft-cli are the two tool caches. Load one of them only when the entrypoint selects that tool path. Neither cache is part of the ChaosEngine entry.

Next

  • Repository orientation is README.md at the repository root.
  • Skill inventory

SHAFT engine users add-on

ChaosEngine add-on for people who use SHAFT in their own test projects. The install one-liner selects it when the project matches the pack installWhen, and when the project is the pack home (via shaft-core-developers). It is not selected for any other project.

  • Install: --with-shaft-engine-users (PowerShell -WithShaftEngineUsers, or CHAOS_ENGINE_ADDONS=shaft-engine-users).
  • Remove: --without-shaft-engine-users.
  • Installed at .chaos-engine/addons/shaft-engine-users/.
  • Entry point: shaft-developer/SKILL.md (repository: shaft-skills/shaft-developer/; installed: .chaos-engine/addons/shaft-engine-users/shaft-developer/SKILL.md). It routes to exactly one SHAFT specialist per task.

The skills are the same files the SHAFT agent plugin publishes, included in place from shaft-skills/. Nothing is copied, so the two never drift. SHAFT contributors also want shaft-core-developers, which requires this add-on.

SHAFT project profile

Load the canonical ChaosEngine entrypoint (.chaos-engine/skills/chaos-engine/SKILL.md) first. This profile adds only repository-specific facts and permissions. Its machine-readable identity is in profile.json. Knowledge stores: scripts/agents/knowledge_stores.py (status, search; MemPalace path via tools/repository-map/resolve_mempalace.py; Graphify: graphify). Watch PRs with one blocking scripts/ci/watch_pr_checks.py call per push. Reflection (.chaos-engine/references/reflection-checkpoints.md) applies unchanged.

  • Repository: ShaftHQ/SHAFT_ENGINE; default branch: main.
  • Personal authorizations (commit-signing key, owner email attribution, artifact sharing) live in user-level config, never in this repository: ~/.config/chaos-engine/authorizations.md (Windows %APPDATA%\chaos-engine\authorizations.md). Load it when present; without it, ask before signing as, attributing to, or sharing for the owner.
  • Task branches use ChaosEngine/* and start from fetched origin/main.
  • SHAFT and ChaosEngine tasks are complete only when the pull request is merged.
  • Agents never manually start, rerun, or replace the E2E Tests or Local E2E Tests workflows. Smallest local proof and exact-head PR checks gate delivery; scheduled nightly workflows own E2E execution.
  • The companion public-documentation repository is ShaftHQ/shafthq.github.io on master; discover its local root or use an explicitly configured root, never a fixed sibling path. Every user-facing SHAFT behavior change opens a companion PR on that master branch in the same delivery. That companion PR must include a description of the change, screenshots where a human sees UI, human-facing instructions, and AI-supported details (locator policy, replay-proven snippets, properties, exact commands).
  • Install or upgrade (INSTALL.md): irm https://raw.githubusercontent.com/ShaftHQ/SHAFT_ENGINE/main/chaos-engine/install.ps1 | iex.
  • Maven modules and SHAFT product behavior route through the playbooks and mastery chapters under references.
  • Cheap, bounded, already-specified local coding work uses the workstation local coding agent.

Multi-ticket assignment orchestration

This specializes the portable solo-or-orchestrate rule: two or more SHAFT issues in one owner request are orchestration. Do not wait for the owner to say "orchestrate".

  • Load every assigned ticket (and linked/deferred children) before grouping or dispatching.
  • Group related work to the fewest PRs that still keep one problem per issue (Closes #N per completed subtask).
  • Main session orchestrates: status, owner commands, review, merge. It does not implement product or guidance chunks.
  • Remaining chunks run one at a time, ordered by dependency then priority.
  • After a chunk's PR is merged, destroy that writer and start the next from a fresh ChaosEngine/* branch off fetched origin/main.

Graphify

This repository profile uses Graphify as a repository CLI/cache workflow for current structural questions: likely files, calls, and dependencies. It does not depend on Graphify appearing as an MCP tool. Absence from the MCP tool catalog is not evidence that Graphify is unavailable.

The CLI route below is the controlling Graphify procedure over conflicting same- or lower-priority guidance. Its ordered steps are exact:

  • G1: Resolve any available shared cache and require a nonempty readable graph. A stale cache may provide untrusted positive leads but never completeness.
  • G2: If G1 succeeds, run exactly one query bounded to the affected symbol or subsystem, treat results as untrusted leads, verify every returned path against the current worktree, and supplement conclusions with targeted rg.
  • G3: Attempt the read-only coverage audit against the primary checkout that owns the cache even when G1 or G2 fails. Inability to resolve that owner is a failed audit attempt, never permission to audit a linked worktree.
  • G4: Declare degraded mode when any step cannot provide current verified results, and only after G1 through G3 have been attempted; never use MCP catalog absence as the reason.

Run the procedure with this PowerShell flow, replacing only the bounded structural question:

$graphOut = py -3 tools/repository-map/resolve_graph_out.py --check
$resolverOk = $LASTEXITCODE -eq 0 -and -not [string]::IsNullOrWhiteSpace($graphOut)
$sharedGraphOut = if ($resolverOk) { $graphOut } else {
    py -3 tools/repository-map/resolve_graph_out.py
}
$cacheOk = -not [string]::IsNullOrWhiteSpace($sharedGraphOut) -and
    (Test-Path -LiteralPath (Join-Path $sharedGraphOut "graph.json") -PathType Leaf)
$primaryRoot = if ([string]::IsNullOrWhiteSpace($sharedGraphOut)) { $null } else {
    Split-Path $sharedGraphOut -Parent
}
$queryOk = $false
if ($cacheOk) {
    $queryOutput = @(graphify query "<bounded structural question>" --graph (Join-Path $sharedGraphOut "graph.json"))
    $queryExitOk = $LASTEXITCODE -eq 0
    $queryOutput | Write-Output
    $returnedPaths = @($queryOutput | ForEach-Object {
        if ($_ -match 'src=(.+?)\s+loc=') { $Matches[1] }
    } | Sort-Object -Unique)
    $worktreeRoot = [IO.Path]::GetFullPath((Get-Location).Path).TrimEnd(
        [IO.Path]::DirectorySeparatorChar
    ) + [IO.Path]::DirectorySeparatorChar
    $invalidPaths = @($returnedPaths | Where-Object {
        $relative = $_
        $parts = @($relative -split '[\\/]')
        $lexicallyInside = -not [IO.Path]::IsPathRooted($relative) -and
            $parts.Count -gt 0 -and -not ($parts | Where-Object { $_ -in @('', '.', '..') })
        $resolved = if ($lexicallyInside) {
            Resolve-Path -LiteralPath (Join-Path (Get-Location) $relative) -ErrorAction SilentlyContinue
        } else { $null }
        $inside = $null -ne $resolved -and ($resolved.ProviderPath + [IO.Path]::DirectorySeparatorChar).StartsWith(
            $worktreeRoot, [StringComparison]::OrdinalIgnoreCase
        )
        $hasReparsePoint = $false
        $lexicalPath = (Get-Location).Path
        foreach ($part in $parts) {
            $lexicalPath = Join-Path $lexicalPath $part
            $item = Get-Item -LiteralPath $lexicalPath -Force -ErrorAction SilentlyContinue
            $hasReparsePoint = $hasReparsePoint -or (
                $null -ne $item -and ($item.Attributes.value__ -band 1024) -ne 0
            )
        }
        -not $inside -or $hasReparsePoint
    })
    $queryOk = $queryExitOk -and $invalidPaths.Count -eq 0
}
$auditOk = $false
if ($null -ne $primaryRoot) {
    py -3 tools/repository-map/graphify_maintenance.py audit --root $primaryRoot
    $auditOk = $LASTEXITCODE -eq 0
}
if (-not ($resolverOk -and $cacheOk -and $queryOk -and $auditOk)) {
    Write-Warning "Graphify degraded mode: use targeted live-file verification."
}

A missing cache reports absent; a cache without a matching indexed revision reports stale. Either state is advisory for ordinary tasks. A readable stale cache may be queried once for positive leads, but never establishes completeness, “no callers,” or another negative conclusion. After the ordered route, use targeted rg plus live files for every blast-radius conclusion. Never rebuild or record the cache from an ordinary task, and never commit graphify-out/.

The existing maintenance controller is the sole Graphify refresh owner. A shared origin/main revision mismatch or an active refresh lock is an expected degraded condition and does not block implementation. An ordinary task must not refresh, wait or retry-loop, clear or replace the lock, freshness marker, or cache, or switch, reset, or overwrite the primary checkout to satisfy freshness. Continue with live files and targeted rg; only the maintenance owner updates the shared cache.

From the primary checkout, the repository-owned controller owns refresh:

py -3 tools/repository-map/graphify_maintenance.py refresh --root .

The refresh uses the pinned Graphify runtime, then orders build -> audit -> cluster -> freshness marker. JSON sources with no emitted nodes remain visible expected data-only inputs; SQL or other parser gaps stop the refresh before the marker is recorded.

Unclassified extract skips are corpus policy, not a failed install. Noise families belong in .graphifyignore and mempalace.yaml exclude_patterns. Known first-class SHAFT families that Graphifyy 0.9.43 still drops (.properties, META-INF/services/*, Dockerfiles, plugin XML, .toml, .feature, wrappers) stay on the Graphify allowlist until an upstream promote lands. Nightly MemPalace mine force-includes the exact git ls-files set for .properties, SPI, Dockerfiles, plugin XML, and .feature via --include-ignored (#5119). A new suffix is unclassified_unexpected and fails graphify_maintenance.py audit. knowledge_stores.py refresh delegates to python3 .chaos-engine/tool.py stores refresh and is allowed from any worktree. status/search remain the ordinary-task path. See #5087.

Freshness behavior is pinned by tests/scripts/test_resolve_graph_out.py.

Agent Guidance Boundary Guard

Use for agent guidance and retrieval/config surfaces. Keep one substantive entrypoint; make every host surface a thin relative adapter.

Workflow

  1. Put cross-provider behavior in chaosengine-harness; link new task playbooks from its routing reference instead of copying them.
  2. Keep AGENTS.md, CLAUDE.md, and installer-generated Claude skill/agent adapters lean. They may point to canonical files with relative paths but must not restate policy.
  3. Keep one hook implementation under scripts/agents/; provider hook files only register its relative command.
  4. Preserve restricted native Memory access, deterministic MemPalace retrieval, and Graphify routing. Do not save routine diaries.
  5. Update guidance budgets/manifests and add a failing portability test before changing behavior.
  6. Assemble the portable shaft-skills package with python3 scripts/ci/assemble_shaft_skills_plugin.py <empty-output-directory> and validate it with scripts/ci/validate_agent_plugins.py; cover assembly in tests/scripts/test_assemble_shaft_skills_plugin.py, and release assets in tests/scripts/test_agent_plugin_release.py before release. For shaft-skills, refresh and validate the generated inventory, reviewed scorecard, and context budget with python3 scripts/ci/shaft_skill_quality.py --write, then run tests/scripts/test_shaft_skill_quality.py; refresh the canonical routing corpus adapters with python3 scripts/ci/shaft_skill_routing_eval.py --write and cover them in tests/scripts/test_shaft_skill_routing_eval.py. Validate every online candidate through python3 scripts/ci/shaft_skill_candidate_intake.py and tests/scripts/test_shaft_skill_candidate_intake.py; never install candidate code into canonical roots or bypass its no-network container gate. Run unauthenticated native-client evidence with python3 scripts/ci/agent_plugin_client_smoke.py --mode smoke and cover it in tests/scripts/test_agent_plugin_client_smoke.py; scheduled live acceptance in .github/workflows/agent-plugin-acceptance.yml records missing model credentials as external blockers.
  7. Run python3 scripts/ci/validate_agent_setup.py --skip-external; run the full command when network/npm access is stable.

Output

List canonical guidance, adapter, hook, retrieval, and validation changes plus intentional budget slack.

PDCA

Personas are phases, not agent identities. They run sequentially in the main session. PDCA does not select an execution mode. Select one through execution workflows (.chaos-engine/references/execution-workflows.md), then keep these phases sequential inside it.

  • Kevin phase plans spec, value, acceptance, risks, and any useful Mermaid or wireframe.
  • Bob phase makes the smallest cross-platform change through observed TDD, or shepherds a bounded default-capability owner that does when orchestrating.
  • Bruce reviews the actual diff and evidence for defects, ambiguity, and confidence, then assigns any required patch to an implementation owner.

Do not merge phases. Run Kevin -> Bob -> Bruce. Bruce covers quality/simplicity, then intuitiveness/acceptability, inside Check. Stop when Check accepts. Do not add a pass to fill a quota.

Execution

Switch persona by switching main-thread phase, never by creating persona agents, workflows, or orchestrators. Bob takes at most three implementation rounds. Bruce judges the actual diff plus real checks, never a self-report; hunt stubs and weakened assertions. Gaps are closed by whoever the mode says implements. Record which phase produced each commit.

Use the canonical TDD/PDCA phase boundary (.chaos-engine/references/tdd.md#workflow). The harness close is the delivery cycle (references/delivery-cycle.md in the harness): at most 3 PDCA iterations, then two independent adversarial reviews. Bruce's in-session check is Check. It is not one of those two reviews, and the author is neither reviewer. Stop when Check accepts. The cap is not a quota.

Allure Extent Report Operator

Use for report generation and CI artifact behavior. Do not treat a generated summary as valid until raw result files are populated.

Workflow

  1. Separate producers from consumers: AllureManager, report properties/paths, .github/actions/post-test-report, .github/actions/consolidate-test-results, workflows, and scripts/ci/extract_allure_failures.py.
  2. Preserve the live allure-results directory and delete only its contents; replacing the root can race with Allure writers on Windows.
  3. Count *-result.json files before trusting Allure status or summary.json.
  4. For parser changes, run python3 -m pytest tests/scripts/test_extract_allure_failures.py. For Java report internals, run the narrow affected Allure manager tests.
  5. Treat Extent as a separate path: verify actual Extent artifacts before claiming support.

Output

Report producer/consumer impact, raw result count evidence, parser or Java tests run, artifact names, and remaining OS/path risk.

CI Failure Investigator

Use this playbook only for CI failure triage.

Workflow

  1. Identify the exact run, failed jobs, changed commit, and failed steps. Use local logs supplied by the user before making GitHub calls.
  2. If remote data is required, normalize authentication with scripts/ci/github-auth-env.sh, then fetch failed-job logs and relevant artifact metadata only. Do not download successful-job logs by default. A job-log download is not a MemPalace or Graphify retrieve. gh api .../actions/jobs/<id>/logs, gh run view --log-failed, and curl of that logs URL may be saved under a scratch directory and filtered with sed/tail without a store citation.
  3. Search logs for the first actionable exception and its surrounding setup or teardown lines. Separate primary failure from reporting and cleanup noise.
  4. For SHAFT artifacts, follow the CI investigation runbook. Parse self-contained Allure HTML or result JSON instead of opening large reports manually.
  5. Count result files before trusting statuses. Inspect failed, broken, and retried/skipped attempts when the final summary hides an intermittent failure.
  6. Classify the cause as code, test isolation, configuration, dependency, runner/infrastructure, credentials, or external provider. Do not weaken assertions to hide provider or credential failures.
  7. Inspect only the source and recent changes connected to the failure signature. Propose or implement the smallest root-cause fix.
  8. Validate with the narrowest local reproduction. Request a remote rerun only when local evidence cannot prove the environment-specific behavior.

Known SHAFT fingerprints

  • setup-lock-pin-drift: a bundled setup package-lock.json changed but the planner SHA-256 pin did not, so Unit Tests (shaft-infrastructure) fails with "lock does not match the approved plan". Keep the pin. Same-repo Dependabot pull requests self-heal through setup-lock-pins.yml; otherwise run python3 scripts/ci/refresh_setup_lock_pins.py --write in the same tip.

Output

Report the run/job, concise signature, root cause and confidence, affected files, validation evidence, and remaining environment risk. Avoid templated tables when a short finding is clearer.

Flaky Test Stabilizer

Use this playbook for intermittent, retry-only, parallel-only, or environment-sensitive failures.

Workflow

  1. Gather the exact failure, runner mode, retry history, affected test, and setup/teardown code. Do not begin with broad refactoring.
  2. Check high-probability causes in this order:
    • per-method state stored in shared instance or static fields;
    • ThreadLocal values not removed;
    • one method deleting another method's files;
    • global SHAFT properties or static services shared across classes;
    • fixed sleeps or missing readiness polling;
    • unordered/random assertions or external-service dependencies;
    • browser sessions created accidentally by unit-test constructors.
  3. Remember that TestNG singleThreaded isolates methods in one class, not static state shared with other classes. Use a cross-class lock where a static service genuinely cannot run concurrently.
  4. Preserve the allure-results root during cleanup. Inspect retried/skipped result JSON when the final report is green.
  5. Fix the ownership or synchronization defect without changing what the test validates. Prefer local variables, ThreadLocal, explicit locks, readiness polling, deterministic fixtures, and mocked external dependencies.
  6. Run the focused test once. Repeat it or enable method parallelism only when repetition directly tests the suspected flake mechanism; stop after enough evidence establishes stability.

Output

State the failure signature, root cause and confidence, minimal fix, focused commands run, repetition/parallel evidence if applicable, and residual risk.

Framework Source Rules

  • Deprecate before removing or renaming public APIs; prefer overloads to changed signatures.
  • Add concise JavaDoc (@param/@return/@throws where applicable) to public classes and methods.
  • Overload @param arity (before push): when adding or splitting overloads, strip or retarget every @param tag to the actual signature before push. A leftover @param from a longer overload fails ErrorProne and cascades CI (#5748). Run scripts/ci/check_javadoc_param_arity.py (repo-only) against shaft-engine interaction packages (or the files you touched). The harness PR Gate javadoc surface also runs the checker’s unit tests when interaction sources or the script change.
  • Keep user-facing APIs consistent with the SHAFT facade and its namespaced nested classes. Utility classes use a private constructor that throws IllegalStateException("Utility class").
  • Throw specific exceptions, retain causes, and use SHAFT reporting/logging utilities. Never print to standard output or swallow failures silently.
  • Declare configurable versions/URLs/timeouts/thresholds/switches in the matching com.shaft.properties.internal interface with a default; access via SHAFT.Properties, never System.getProperty() directly.
  • Use ThreadLocalPropertiesManager for per-thread overrides; clear with Properties.clearForCurrentThread() at lifecycle boundaries. Document non-obvious shared/thread-local state ownership.
  • New external tools stay batteries-included: resolve from PATH, then a package manager where applicable, then a controlled Maven-cache download; keep the version in the properties layer.
  • Merge existing vendor capability maps and keep all WebDriver extensions namespaced. Never restore top-level legacy capabilities such as enableVideo.
  • For Allure 3, use AllureManager.resolveAllureCommandPrefix(). Do not add the removed allure generate --clean flag; the generated summary is at the report root.
  • Preserve UTF-8 flags in Maven and CI configuration when touching process or reporting setup.
  • For a bug, reproduce the defect (add a focused regression test when practical); make the smallest behavior-preserving fix.
  • Route release metadata work through the release guard.

Design / Automation MCP apply paths (#6012)

Every Design/Automation tool that paints IDE state needs:

  1. A dedicated apply*Json (or typed apply) on the panel/model.
  2. A router discriminator on a stable field (or explicit tool name) in DesignStagePanel.applyToolResult — never reuse applyAnalysisJson for a different JSON shape.
  3. A panel unit test that applies a fixture and asserts the UI model updates.

Current discriminators include shaftType/shaftMethod → gap map, handoffAllowed → readiness, automationPrefill → handoff.

See also intellij-ui-test-conventions.md for accessibleName (#6013) and Live record persistence (#6014).

IntelliJ Plugin Development

Use for shaft-intellij plugin, tool-window, Gradle or JDK, or Marketplace work.

Load the IntelliJ plugin mastery chapter instead of copying it. Visible UI also follows SHAFT UI design. Contributor notes: UI test conventions (accessibleName, Live record persistence) and Design/Automation apply paths.

Cite current JetBrains sources, not memory:

Workflow

  1. Load mastery before editing plugin source, tests, Gradle, or Swing UI.
  2. Keep the plugin a light wrapper. The same generate-from-scenario path must work from the plugin and from a CLI. Pass the user text to the selected client; do not drive capture_start, replay, or generate from the plugin.
  3. Marketplace setup is copy the installer command and pre-type the IDE terminal only. Never execute the installer from the plugin with ProcessBuilder, Runtime.exec, or any equivalent.
  4. Visible UI changes regenerate screenshots with ShaftPluginScreenshotRendererTest only, passing -Dshaft.intellij.screenshotDir=... and -Dallure.automaticallyOpen=false. Never a bare full-suite gradlew test.
  5. Run the smallest focused plugin tests for the edited behavior. Headless only.

Output

List plugin files touched, checks run, screenshot paths for visible UI, and confirm plugin source has no installer-exec path.

IntelliJ UI test conventions

Icon-only controls (#6013)

ShaftIconButtons and other icon-only toolbar controls expose their contract through accessible name, not visible text. Prefer:

findByAccessibleName(panel, "Start recording", JButton.class);
assertEquals("Start recording", button.getAccessibleContext().getAccessibleName());

Do not assert getText() on icon-only Automation/Design toolbar buttons — text is empty or decorative and flakes across LAFs.

Automation workflow persistence key (#6014)

The Automation canvas persistence key is Live record (AutomationStagePanel.LIVE_RECORD_TAB). A legacy Guided key restores once to Live record and must not be written again.

Java Test Rules

  • Follow the existing TestNG, JUnit, or Cucumber style in the affected module. Use descriptive scenario-based test names.
  • org.testng.Assert.assertThrows returns void and expectThrows returns the throwable. Use assertThrows only as a statement when the test needs the type. Use expectThrows when the test needs the throwable. Do not assign assertThrows.
  • Browser tests create a fresh driver per test and clean it in an always-run teardown. For ThreadLocal drivers, call both quit() and remove().
  • Browser tests run headlessly; headed-only behavior needs explicit user approval before execution (see AGENTS.md).
  • Under TestNG method parallelism, instance fields are shared across concurrent methods; keep per-method paths/files/mocks/drivers/mutable setup in locals or ThreadLocal, and clear the same thread-owned state in cleanup.
  • @Test(singleThreaded = true) serializes one class only; static services or SHAFT global properties shared across classes need an explicit cross-class lock or per-test prerequisites, plus always-run restoration.
  • Use Properties.clearForCurrentThread() for per-thread SHAFT properties, except JVM-global SHAFT.Properties.flags — capture/restore the original in finally/@AfterMethod.
  • When a test writes Allure results, follow the reports playbook; it owns the allure-results handling and verdict rules.
  • Prefer SHAFT fluent assertions and existing test-data conventions. Store reusable data in module test resources instead of embedding large fixtures.
  • Coverage-only unit tests must not call facade/default constructors that create real browser sessions. Use explicit mock-backed constructors; keep session creation in integration tests.
  • Avoid mocking or proxying org.testng.ISuite on Java 25 when list/value helper logic can be tested directly.
  • Mock external services when the contract is not under test. Disable status validation only when response status is intentionally outside the assertion.
  • For a bug fix, prove the regression first when practical, then run the focused test after the fix; repeat/parallelize runs only to evaluate a flaky or concurrency failure.
  • Reuse passing test evidence unless later edits affect the tested behavior or its dependencies.
  • Capture screenshots only when the tested behavior is visual.
  • For flaky failures, use the stabilization playbook.

Static analysis and proof runs

  • Strategy-matrix / Wave PRs fail static analysis when Java switches lack a default branch: switch must be exhaustive (default where Codacy MissingDefaultCase requires it). Confirm before the first push.
  • Codacy is this repository's static-analysis app: its ACTION_REQUIRED is the static-analysis gate (.chaos-engine/references/static-analysis-gate.md).
  • N-run / flake-proof scripts that invoke Maven under the engine Surefire profile must not treat process exit alone as green: after each proof invocation, require a zero failed count from Surefire TEST-*.xml or TestNG testng-results.xml (and set -Dmaven.test.failure.ignore=false for defense in depth). Use scripts/ci/assert_surefire_green.py (repo-only).

MCP Transport Contract Auditor

Use for shaft-mcp contract changes across tool schemas, fixtures, startup, installers, containers, and transports.

Workflow

  1. Pair Java/service edits with shaft-mcp/src/test/resources/fixtures/mcp-tool-manifest.json and shaft-mcp/src/test/resources/fixtures/shaft-pilot/mcp/** when the public tool contract changes.
  2. Run python3 scripts/ci/validate_shaft_mcp_configuration.py. Also run python3 scripts/ci/validate_modular_documentation.py for client fixture or public docs-link changes.
  3. For Java behavior, run the narrow affected shaft-mcp tests with -DheadlessExecution=true and -Dgpg.skip.
  4. Before claiming transport safety, package shaft-mcp, copy runtime dependencies to shaft-mcp/target/dependency, then run python3 scripts/ci/validate_shaft_mcp_transports.py.
  5. For installer or release-facing changes, run python3 scripts/ci/verify_shaft_mcp_installer_release.py or state why it is not applicable.

Output

List contract files touched, transport/client impact, checks run, and remaining fixture or installer risk.

Modular Boundary Auditor

Use for module boundary and packaging drift across Maven reactor modules, examples, BOM, legacy coordinate, and clean consumer fixtures.

Workflow

  1. Inspect root/module POMs, shaft-bom, legacy-shaft-engine, bundled examples, and tools/modularization/consumer-fixtures/** before changing checks.
  2. Keep optional module dependencies out of shaft-engine; fix the owning module, BOM, or fixture instead.
  3. Run python3 scripts/ci/validate_reactor_versions.py when versions or inter-module coordinates change.
  4. Run python3 scripts/ci/validate_modular_documentation.py, python3 scripts/ci/validate_maven_publication.py, and the relevant tests/scripts/test_*_boundary.py.
  5. For release-candidate surfaces, add python3 scripts/ci/validate_shaft_pilot_release.py and affected clean-fixture mvn --batch-mode -f ... verify "-Dshaft.version=..." checks.

Output

State the boundary touched, owning module, BOM/fixture/example impact, checks run, and any untested fixture matrix.

Public Behavior Docs Synchronizer

Use for user-visible SHAFT behavior changes. Keep docs work separate from code unless the task explicitly asks for both.

Every user-facing SHAFT behavior change opens a companion PR on ShaftHQ/shafthq.github.io master in the same delivery: discover the docs root, or use an explicitly configured root; never a fixed sibling path. Do not add public guide pages to SHAFT_ENGINE. That companion PR must include a description of the change, screenshots where a human sees UI, human-facing instructions, and AI-supported details (locator policy: unique author-written id via the SHAFT locator builder, then ARIA role, then native relative xpath only; replay-proven snippets; properties; exact commands).

Required companion PR content

  1. A description of the change.
  2. Screenshots for visible UI. Reuse existing static/ or test screenshots when still accurate. If a screenshot cannot be produced headless, ship the prose and open one standalone follow-up issue per missing shot. Do not launch a GUI browser without asking.
  3. Human-facing steps.
  4. AI-supported details: locator policy, replay-proven snippets, properties, and exact commands.

Workflow

  1. Identify the public surface: API, property, module, report output, CLI/MCP tool, README link, or release text.
  2. Discover the local shafthq.github.io checkout, or use an explicitly configured root; never a fixed sibling path. Search it with targeted rg; update only guide pages affected by the behavior change.
  3. Keep canonical links in README.md, .github/RELEASE_BODY_TEMPLATE.md, and legacy-shaft-engine/pom.xml. Run python3 scripts/ci/validate_modular_documentation.py when those links or examples are touched.
  4. Run python3 scripts/ci/validate_documentation_boundaries.py; for guidance changes also run python3 scripts/ci/validate_agent_setup.py.
  5. Open the companion PR on master with the required content above. Report the docs PR/link, or the concrete reason no public docs change is needed.

Output

List the public surface, guide files searched or changed, validation results, and docs PR/blocker.

Release And Dependency Guard

Use only for release preparation, release review, release notes, or dependency-currency work. Never run deployment or publication commands locally.

Workflow

  1. Inspect recent merged release PRs and executable workflows before relying on prose. Compute versions as {major}.{quarter}.{YYYYMMDD}.

  2. Keep the root project version, every reactor child parent, and SHAFT inter-module dependency aligned. Run:

    python3 scripts/ci/validate_reactor_versions.py
    
  3. Update shaft-engine/src/main/java/com/shaft/properties/internal/Internal.java shaftEngineVersion. Verify allure3Version against the stable npm package and nodeLtsVersion against the current LTS patch using primary sources. Verify every other version default in Internal.java against its source registry when it changes. Keep non-Maven static versions in Internal.java; move stray version literals there.

  4. Update all example and consumer-fixture <shaft.version> values found by:

    rg -l "<shaft.version>" -g "pom.xml"
    
  5. Check Maven dependency/plugin/property updates. Accept stable ones after compatibility review; do not treat alpha, beta, RC, milestone, or snapshot releases as upgrades.

  6. Validate release metadata and publication rules with existing scripts, including Maven Central POM/JAR/classifier/signature, the IntelliJ IDEA plugin release candidate, and canonical, combined-module, and legacy-relocation consumer checks.

  7. Run one compile/package pass after metadata is aligned. Do not rerun broad product tests for metadata-only changes unless source/test behavior changed.

  8. Verify the release version is newer than any immutable Maven Central coordinate it replaces. Merging to main triggers publication; local deploy, signing, and scm-publish are prohibited.

Release notes (#6232)

Release bodies come from scripts/ci/render_release_notes.py (run by announce_release) filling .github/RELEASE_BODY_TEMPLATE.md; do not switch back to raw GitHub-generated notes or hand-write bodies.

  • Format: version, a 1-3 line summary, the Maven/Gradle snippet, then only non-empty sections in this order: Breaking changes and upgrade notes, New features, Fixes, Performance, Deprecations, notable Dependency upgrades. One cleaned - Title (#PR) line per change; no authors or PR URLs. All internal work goes in one <details> block; the full changelog link is last.
  • Internal: [CE]; ci, test, docs, chore, build, or refactor prefixes; ChaosEngine/installer/harness scopes; the labels subsystem:agent-harness, subsystem:repository-tooling, documentation, tests, github-actions, or maintenance; or no files under shipped module sources. Bumps collapse to "N dependency updates" unless labeled security or they are Selenium/Appium/Playwright majors.
  • Labels: exactly one of breaking-change, enhancement, bug, skip-release-notes; optional performance, deprecation, regression. PR titles become release lines, so write them for SHAFT users.
  • Keep the renderer, template, and scripts/ci/validate_release_notes.py label sets in sync; Release-note governance tests that parity.
  • Slack and reconcile announcements reuse that summary line plus a link via slack_payload (#6241); never promise sections the notes do not have.
  • Never edit a published release body. Preview with python3 scripts/ci/render_release_notes.py --version <tag> --previous-tag <prev> --head <tag> --output <file>.

Output

List blockers first, then version alignment, stable updates considered, commands and results, source checks for volatile versions, and remaining publication risk.

SHAFT Marketing Ad Producer

Plan SHAFT ads without fake claims.

When the optional design-skills add-on is installed (.chaos-engine/addons/design-skills/SKILL.md), follow its technical-video pipeline and cards (brief, capture, voice-over, captions, loudness, delivery QC) with this playbook's SHAFT facts and guardrails; run its design_qc.py gates. Without it, use the workflow below.

Stack

  • Strategy/scenes: plan in the brief/storyboard; no dedicated tool.
  • Visuals: keep exact copy, code, logos, and UI screenshots deterministic; no image-generation tool, so write a visual brief.
  • Capture: shaft-mcp browser/capture tools for web (screenshots, capture_start); browser_network_requests for perf/network; desktop/OBS/ffmpeg if installed.
  • SHAFT proof: use MCP recorder, report, Doctor, Allure, and code-block tools.
  • Editing/audio: use existing video tools and approved or licensed audio; otherwise write a music brief.

Workflow

  1. Brief: audience, channel, duration, feature, proof, aspect ratio, and planning-only vs production.
  2. Evidence: read SHAFT README/docs or MCP guidance. Use real Allure, Capture, MCP recorder, Doctor, or code surfaces for production; label mockups.
  3. Script/scenes: make a timestamped table with visual/action, text, voiceover, source, animation, music beat, and asset path.
  4. Assets: render code snippets as deterministic HTML/CSS screenshots. Never use generated-image text for final copy or code.
  5. Capture: avoid secrets and private browser data. Record approved tabs/windows, preferably demo projects or sanitized reports.
  6. Edit: assemble clips, captions, overlays, snippets, music, and exports with the smallest existing toolchain. Save commands/projects.
  7. Reuse: save reusable output in the marketing repo, not SHAFT_ENGINE. If no repo path is given, ask first. Use brand/theme.json, brand/assets/, snippets/, music/, and campaigns/<yyyy-mm>-<slug>/{brief.md,script.md,storyboard.md,captures/,renders/,exports/,manifest.json}.

Output

Planning: campaign slate, storyboard table, asset checklist, capture plan, and theme notes.

Production: scripts/storyboards, captures, code-card renders, captions, music source/brief, edit manifest, exports, and marketing-repo paths.

Guardrails: no unsupported benchmarks, 3rd-party logos, security claims, generated final code text, secrets, or tokens.

SHAFT UI Design

Use for every visible SHAFT interface: reports, core tools, the public guide, and app-like product UI. Read this file fully before planning, editing, reviewing, or validating UI work. When the optional design-skills add-on is installed (.chaos-engine/addons/design-skills/SKILL.md), its visual-direction, typography, color, image, and audit cards supply the generic rules; this file adds the SHAFT specifics and wins on conflict.

Sources Of Truth

  • Current repo rules win: AGENTS.md, CLAUDE.md, matching skills, Memory, and the target repo's AGENTS.md.
  • Read existing tokens, CSS, theme files, components, and representative pages before designing. Reuse what works.
  • For guide work, preserve Docusaurus behavior, dark mode, reduced motion, semantic HTML, copyable fenced code, search exclusions, archive noindex, redirects, AutoBot HTTP contracts, and source-backed claims.
  • Do not add dependencies, systems, or generated artifacts unless the current task needs them.

Workflow

  1. Define the surface: core report/tool/workbench, docs/user guide, homepage/marketing, or app-like product UI.
  2. Inspect the current design language and one representative rendered state. For revamps, capture a before screenshot first.
  3. Make a short design plan: audience, one job, 4-6 color roles, type roles, layout idea, and one memorable SHAFT-specific signature element.
  4. Build the smallest complete change. Prefer existing components, tokens, CSS, and native browser features over new abstractions.
  5. Validate in a browser at desktop and mobile widths. Check contrast, focus, keyboard use, reduced motion, dark mode when supported, loading/empty/error states, and text overflow.
  6. Share real screenshots every time: before/after for revamps, after-only for new UI. PRs that touch visible UI or reporting must include those links/paths in the PR body or a PR comment. If capture is blocked, keep the PR draft or report the exact blocker.

Design Direction

  • Make the interface feel like SHAFT: precise, evidence-first, automation-native, confident, and readable under real debugging pressure.
  • Ground visual choices in the subject: test evidence, reports, traces, locators, flows, timelines, browser/device surfaces, or docs learning paths.
  • Use one bold idea, not many decorations. A signature visualization or interaction should explain the product better than a generic hero/card pattern.
  • For operational/report UI, prioritize dense scanning, stable layout, clear status, comparison, filtering, and repeat use. For guide/marketing pages, let the brand voice carry more personality while keeping claims verifiable.
  • UI copy is part of design: plain verbs, sentence case, consistent action names, specific errors, useful empty states, no vague apology copy, no unsupported superlatives.

Quality Floor

  • Contrast: body and placeholder text at least 4.5:1; large text at least 3:1.
  • Layout: no incoherent overlap, no clipping, no text overflow, no layout shift from hover or dynamic content, stable dimensions for grids/boards/toolbars/cards.
  • Typography: readable line lengths, clear hierarchy, no viewport-scaled font sizes, no negative letter spacing, headings balanced when supported.
  • Components: icon buttons use existing icon libraries when available; familiar controls beat custom widgets.
  • Motion: purposeful and sparse; never gate visibility on animations; avoid layout-property animation; always respect prefers-reduced-motion.
  • Assets: prefer real product/report/docs imagery or deterministic screenshots. Do not use generated images for exact UI text, code, logos, or claims.
  • Color: never pure #000000 or #ffffff for surfaces or body text; keep one accent role and let neutrals carry the rest. Accent saturation above ~80% reads as a default theme, not a choice.
  • States: every interactive surface ships focus, loading, empty, and error states. A loading state mirrors the real layout's dimensions rather than a centered spinner, so the page does not jump when content arrives.

Unslop Rules

Reject and rewrite these unless the brief explicitly requires them and the choice is justified:

  • Generic SaaS hero metrics, endless equal card grids, decorative glassmorphism, gradient text, bokeh/orbs, diagonal stripe backgrounds, sketchy SVG scenes, side-stripe accents, and dark blurred stock imagery.
  • Tiny uppercase eyebrows or 01 / 02 / 03 markers as section scaffolding. Numbers only when order matters.
  • One-note palettes: default cream/sand, purple-blue gradients, dark navy/slate, brown/espresso, or a single hue ramp. Existing SHAFT brand tokens can stay.
  • Ghost cards: decorative 1px border plus wide soft shadow. Pick a border or a restrained shadow.
  • Over-rounded cards/sections/inputs. Keep cards at 8px radius or less unless an existing design system already requires otherwise.
  • Hover transforms on images that are not action targets. Animate the card state, border, shadow, or content affordance instead.
  • Text in visible UI that explains the feature instead of letting the feature work.

SHAFT-Specific Checks

  • Core generated reports and workbenches must keep evidence readable: status chips, tables, wrapping, trace/code panels, screenshots, and diagnostics must remain inspectable.
  • User guide pages must keep one canonical page per concept and verify commands, APIs, properties, dependencies, and compatibility against engine source.
  • Public docs changes that alter behavior need the matching docs PR; pure agent-guidance changes do not.
  • PR evidence must include the smallest useful validation plus real screenshot links/paths for any UI or reporting enhancement.

IntelliJ Plugin UI

  • Follow JetBrains IntelliJ icon standards first: https://intellij-icons.jetbrains.design/ and https://plugins.jetbrains.com/docs/intellij/icons.html.
  • Reuse IntelliJ Platform icons from AllIcons when they render correctly in runtime and screenshot evidence. Use plugin-owned SVGs only when platform icons do not satisfy the UI, licensing, theme, or deterministic screenshot need.
  • Keep action icons as 16x16 SVGs under shaft-intellij/src/main/resources/icons/actions, with matching _dark.svg variants when needed. Use the IntelliJ New UI action colors: light #6C707E, dark #CED0D6.
  • Tool window icons must keep the existing 16x16 and 20x20 SVG light/dark variants registered in plugin.xml.
  • Do not hand-draw replacement glyphs in screenshot renderers or paint fake icons over components. If screenshots show blank or broken icons, fix the shared icon source (ShaftIcons or assets) so runtime UI and screenshots use the same real icon.
  • Preserve icon-only button shape, size, tooltip, and accessible-name behavior through ShaftIconButtons; do not add one-off button styling in individual panels.
  • Validate IntelliJ icon/UI changes with the smallest relevant set: ShaftIconAssetsTest, focused panel tests that assert icons paint visible pixels, screenshot renderer output, and visual inspection of light, dark, and narrow states.
  • Swing surfaces obey the same states and color rules as the web surfaces above: focus, loading, empty, and error states are required; one accent role; never pure black or white. The plugin-screenshot renderer is where you verify them, not a browser.
  • Keep icon stroke weight and metaphor consistent across a panel. Avoid cliche metaphors (rocket for run, shield for security) when a platform AllIcons entry already names the concept.

Portions of the color/state rules above are re-expressed from leonxlnx/taste-skill (MIT, Copyright (c) 2026 Leonxlnx).

Workstation local coding agent

Use the already-installed workstation loop when the work is cheap, bounded, and already specified. The default capability's cost and latency are waste for those slices. The most-intelligent or default model stays the decider and may hand the slice to this loop.

Orchestration

The host session orchestrates. One implementer per batch. The writer stops when the PR exists. The inner shaft-java-agent.ps1 loop is optional and allowlisted. The local loop still never merges, reviews, or owns GitHub delivery.

Owner-gated train

Owner-gated train, export, and deploy stay out of tree only. Never auto-train from harvest. Never overwrite qwen2.5-coder:7b. Deploy a new Ollama name.

When to use it

  • Read-only shaft-architect with the cited-path gate.
  • Mechanical script-string pins the loop can apply without inventing scope.
  • A fully specified edit the loop can run, with an allowlist and a test command, where default-capability time is waste.

When not to use it

  • Product changes under src/main/java.
  • PR merge, GitHub delivery, or review.
  • Harness architecture, routing, or guidance design.
  • Anything the loop cannot run: missing allowlist, missing spec, work that needs a public-API judgment, or a host that is not this workstation.

Commands

Keep one model loaded. Loopback only. No startup apps, tray icons, or cloud endpoints.

  • scripts/local-coding-agent/shaft-java-agent.ps1 — bounded edit loop.
  • scripts/local-coding-agent/shaft-architect.ps1 — read-only architect; cited-path gate fail-closes invented slashy paths.
  • scripts/local-coding-agent/shaft-local-ai-stop.ps1 — stop the loopback runtime.
  • scripts/agents/knowledge_stores.py — status or search from any worktree. refresh refuses.

Output

Return the command that ran, the report path, cited-path or allowlist failures, and whether the decider must take the work back.

Product-track self-improve (SHAFT + ChaosGauge)

Profile extension of the portable product-track playbook (.chaos-engine/skills/self-improve/references/product-track.md).

SHAFT filing cues

Lesson classPrefer
Engine / public API regressionFocused shaft-engine (or affected shaft-*) unittest + silent-verify
Doctor / CLI repair gapshaft-doctor / shaft-cli path — not a new MCP essay surface
Agent effectiveness on product tasksLink closest ChaosGauge public task under scripts/ci/chaos_gauge/
IntelliJ pluginFocused plugin check; keep docs PR separate when docs-only

Routing

Product authoring still hands off to shaft-developer via shaft routing. This playbook only covers Learning Session → issue/eval encoding after delivery.

Proof examples

py -3 -m unittest <focused_module> -v
python3 scripts/ci/chaos_gauge/validate_experiment.py scripts/ci/chaos_gauge/experiment.json
python3 scripts/ci/harness_pr_gate.py --base "$(git merge-base HEAD origin/main)" --head "$(git rev-parse HEAD)" --write-generated

Reject Task Observer; Delivery-Stop / Learning Session only.

Routing

Match the deliverable in front of you to one row, load that surface, and finish its deliverable before returning here. Do not load an adjacent surface preemptively. If no row fits, say what is missing rather than approximating.

Never make the user name an implementation detail they could not reasonably know. Infer the row from what they asked for.

Always, before any row

Take the depth the entrypoint's triage already selected, loading consult-first (.chaos-engine/references/consult-first.md) only when that triage sends you there. Apply the entrypoint's Caveman, Ponytail, and TDD rules; use delegation (.chaos-engine/references/delegation.md) before dispatch and roles (.chaos-engine/references/roles.md) for role boundaries. When the change involves a mock, a method a production class gains only for a test, or an argument for writing the code first, add TDD failure modes (.chaos-engine/references/tdd-failure-modes.md).

Knowledge before discovery

Each store answers a different question. Query the ones whose trigger fires before broad manual discovery, not all of them by reflex.

Question you actually haveStoreQuery it when
Has this constraint or gotcha already bitten us?native MemoryA concrete prior constraint or gotcha would change the task decision.
What happened around this before, and what does it touch?MemPalaceThe change spans entities, sessions, or a history you were not part of.
What calls or depends on this?Graphify (.chaos-engine/references/graphify.md)Blast radius is unknown, or you are about to change a shared symbol.
What does the code do right now?targeted rg and exact readsAlways. This is the only source that settles a disagreement.

From any SHAFT worktree, py -3 scripts/agents/knowledge_stores.py status or search hits the centralized palace and a Graphify --check. refresh refuses and names SHAFT-Nightly-Knowledge-Refresh.

Each selected store gets one attempt through the existing host timeout. Record irrelevant stores as skipped and unavailable stores as degraded; neither blocks ordinary implementation or completion. Explicit install, upgrade, maintenance, status, and doctor operations remain strict.

The native store is source-controlled, and its contract travels with it: .memory/config.json declares the project and store version, and .memory/schema/object.schema.json, .memory/schema/relation.schema.json, .memory/schema/event.schema.json, .memory/schema/patch.schema.json and .memory/schema/config.schema.json validate every write. The entries themselves are data the CLI owns; reach them with a bounded memory query, then memory inspect only for selected records. SessionStart injects tracked locators only and never starts an optional retrieval process.

A retrieved claim is a lead, never a verdict: confirm it against the live file before acting on it, and a stale index never outranks what is on disk. Your own plan ranks here too, below every one of them: it is the oldest source you hold, written before the reads.

If a store is unavailable, name the degraded mode in your report and continue with the rest. Skipping a store whose trigger fired, without saying so, is the failure this table exists to prevent.

Repository engineering

Deliverable in front of youLoad exactly this
Agent guidance, host adapters, hooks, budgets, retrieval setupagent guidance
Explicit PDCA, personas, repeated refinementPDCA
Production Java under src/main/javaframework source
Test Java under src/test/javaJava tests
A red CI run, job, or scheduled suiteCI failures
Inconsistent pass/fail under equivalent conditionsflaky tests
Version, release, release notes, BOM, or dependency metadatarelease and dependencies
MCP transport, tool contract, or client wiringMCP transport
Module boundary or cross-module dependencymodule boundaries
Allure or Extent report generation and verdictsreports
Externally documented behavior that changedpublic docs
Any visible SHAFT interface, visual QA, UX copy, accessibilityUI design
Marketing or promotional materialmarketing
Open or rewrite a work itemwork-item skill (.chaos-engine/skills/work-item/SKILL.md)
One issue through a merged PRGitHub playbook (.chaos-engine/references/work-github-playbook.md)
Reviewing a diff for behavior that no check would catchverification-gap lens (.chaos-engine/references/verification-gap-lens.md)
Holding main thread at session startorchestrator bootstrap (.chaos-engine/references/orchestrator-bootstrap.md)
A repository CI script or guard under scripts/, outside agent guidanceagent guidance, then the guard's own test
In-repo Markdown for humans, such as README or CONTRIBUTINGpublic docs
The .memory store itself: entries, relations, or hygieneagent guidance
Cheap, bounded, already-specified workstation codingworkstation local coding

Deep SHAFT internals

Read the one chapter the task actually touches, and skip the rest. Each encodes incident history that is expensive to re-derive.

Task touchesLoad exactly this
Recorder, preload scripts, network capture, browser lifecycleSelenium BiDi
Report generation or patching, results JSON, verdict analysisAllure internals
Mobile recording and replay, emulators, Appium or WinAppDriver CIAppium mobile
Versioning, Central publishing, BOM, dependency convergenceMaven release
Listeners, forked JVMs, properties precedence, scoped runsTestNG lifecycle
The IntelliJ plugin, its desktop UI, tool windows, Gradle or JDK setupIntelliJ plugin development
IntelliJ Design/Automation MCP tools that paint IDE stateDesign/Automation apply paths
IntelliJ plugin UI tests, icon-only controlsIntelliJ UI test conventions
MCP tools, stdio transport, workspace roots, clientsMCP protocol
Red CI runs, scheduled suites, workflow YAML, shardingCI forensics
Races, synchronization, deterministic reproductionWait strategies
Locator choice, semantic selectors, healer or doctorLocator healing

The plugin-development playbook loads the IntelliJ plugin mastery chapter and must not duplicate it.

SHAFT product and test authoring

Writing, planning, recording, executing, diagnosing, or reporting SHAFT tests is a different surface with its own router. Hand off to shaft-developer in the distributed skills pack and let it select the single lifecycle, implementation, or tool specialist. Do not duplicate its rows here, and do not guess SHAFT syntax, tool names, or CLI flags from this file.

External facts

  • Past-cutoff external library API: context7 MCP; prefer a repo exemplar otherwise.
  • Live Maven Central facts: Maven MCP; for in-tree dependency declarations, inspect POMs first.
  • Never start unavailable infrastructure implicitly.

UI routes preserve native technology: desktop surfaces use IDE and JVM inspection with renderer evidence; web surfaces use browser evidence and an accessibility audit.

Self-improve roadmap (profile)

When running a Learning Session or planning harness/product self-enhance work, load self-improve master plan. Cited online research lives in self-improve research synthesis. Product lessons → SHAFT issues/evals: product-track playbook (portable base: self-improve product-track (.chaos-engine/skills/self-improve/references/product-track.md)). Research/explore isolation: context firewall (.chaos-engine/references/context-firewall.md).

S4 self-improve (portable pointers)

  • Periodic meta-optimize (.chaos-engine/references/meta-optimize.md) — not continuous / no Observer
  • Opt-in draft skill PRs (.chaos-engine/references/draft-skill-pr.md) — default OFF; never auto-merge

ChaosEngine Self-Improve Master Plan

Owner: Mohab Mohie / ShaftHQ/SHAFT_ENGINE
Dated (Africa/Cairo): 2026-09-08
Base tip: 77a273d2a8 — fix(ce): enforce Learning Session after every delivery (harness parity) (#5641)
Supersedes: prior optimization top-10 at analysis SHA 0961058e66 / /workspace/chaos-engine-top10-optimizations.md (ideas #1–#10 landed as #5619–#5625 under epic #5618).
Method: tip deep-read of skills/self-improve/**, learning.py, Stop/Learning Session hooks, token/parity/delivery refs; online harness synthesis; optional Grok CLI consult (/tmp/ce-self-improve-consult.txt).
Constraint: NEVER Cursor Cloud Agents; merge commits only; portable overlay parity across Codex / Claude / Grok CLI / Gemini / Copilot / Grok Bot.


Research synthesis (cited URLs): self-improve-research-synthesis.md.

0. Locked prefs (do not reopen)

PrefStatus
NO native Task Observer / always-on observationLocked reject (#5643; research-adopt-reject)
Lean self-improve: SessionStart locator-only; Learning Session on delivery StopShipped (#5641 / #5613)
Harness parity across six hosts (overlay, not agent memory)Shipped policy (#5645 / host-parity-matrix)
One-command install/configure/updateShipped north star (#5618 / #5620)
Caveman + Ponytail companionsShipped (token-budget-modes)
Issues-first self-improve; eval-gated draft skill PRs opt-in laterLocked (#5618 fork #5)
Merge commits onlyLocked

1. Current-state assessment (post-#5641)

1.1 What CE can already self-enhance

LoopProduct (SHAFT)Harness (CE overlay)Token profile
CaptureDual-track Learning Session → learning.py queue → GitHub issuesSame path; categories allow-list; privacy gatesHeavy only on delivery Stop
Enforce learnStop / delivery-complete owes Learning Session even if the harness tree is untouchedPortable learning_session.py finalize + monorepo controllerHook = 0 context unless block reason
Discover surfacesLevel-1 catalog + router Heal / zero-LLMSessionStart locators under SESSION_START_MAX_BYTES=4096↓ always-on
Health / healInstaller/doctor/repair componentsstatus ⊆ doctor, activationProof, repair --componentZero-LLM
Delivery gatesTriage-scaled research + phase ledgerStop Learning Session; catastrophic denySoft vs hard by triage
MemoryOwner wake pack locator; retrieve orchestratorSoft-degrade Memory/MemPalace/GraphifyNo SessionStart prose dump
EvalProduct CI / ChaosGaugeEval-parity fixtures + host matrixCI, not chat

1.2 Dual-track self-improve ability (honest)

Strengths

  • Stop-gated Learning Session is deterministic (hook), not prompt hope — aligns with industry “hooks = 100% enforce, 0 context unless output.”
  • Issues-first + privacy-gated learning.py is a durable LogAct-like append log (queue → GitHub), not transcript hoarding.
  • Progressive disclosure skeleton exists: Level-1 catalog, locator SessionStart, on-demand self-improve refs.
  • Optimization program #5618 children (#5619–#5625) closed the prior top-10 (health truth, repair, zero-LLM locator, L1 catalog, triage research+ledger, wake+retrieve, eval-parity+portable finalize).

Gaps (self-improve-specific — this plan’s focus)

  1. No closed-loop metrics: queued → submitted → adopted/rejected → estimatedTokens delta is not machine-summarized for doctor/zero-LLM.
  2. No once-per-task heuristic retrieve: lessons live in issues/.memory but are not ERL-style top-k retrieved at triage (still avoid every-turn injection).
  3. Verification dumps still risk token rot: Check/Stop paths lack a universal silent-on-success / errors-only wrapper contract.
  4. Skill body bloat risk: no SkillOpt-style compression/eval gate before growing SKILL.md (L2 <500 lines discipline is documented, not automated).
  5. CLI vs MCP preference is tribal in places; HumanLayer lesson not yet an iron-law + zero-LLM catalog row.
  6. Significance filter for mid-session capture is skill-discipline only — easy to miss under Caveman ultra (without becoming Task Observer).
  7. Phase-2 eval-gated draft skill PRs still deferred (correct) but needs a ready gate checklist.
  8. Product-track playbook thinner than harness track (taxonomy exists; SHAFT-specific acceptance/eval linkage weak).
  9. GAP-EXIT2 still soft on Grok/Copilot — compensating UX exists; not a self-improve blocker but limits Stop-gate fidelity.
  10. Periodic meta-optimize (supervisor over shared logs) not scheduled — only per-delivery Learning Session.

1.3 Gap analysis vs #5641

#5641 deliverableStatusRemaining self-improve work
Merge/delivery → Learning Session owedShippedMetrics that prove sessions actually queue/submit
Docs forbid CE-untouched skipShippedProduct-track yield reporting
Harness parity in overlayShippedEncode new lessons as hooks/skills/scripts, never agent-only memory
SessionStart locator-onlyShippedKeep; add locator to heuristic index only (no prose)
Unit tests for merge→StopShippedExpand fixtures for metrics + silent-verify + heuristic retrieve

Verdict: #5641 closed the enforcement gap. This plan closes the yield / token-efficiency / closed-loop gap for both product and harness.


3. Master roadmap waves (self-improve next)

Dependency order. Do not re-implement #5619–#5625.

Wave S0 — Measure (zero/low LLM)
  [#1 Learning metrics closed loop]

Wave S1 — Token sinks on the hot path
  [#2 Silent-on-success verify] → [#5 CLI-over-MCP iron law]
       ↘ [#3 ERL once-per-task heuristics]

Wave S2 — Capture quality without Observer
  [#4 Significance-filtered mid-session capture] → [#6 SkillOpt compress audit]

Wave S3 — Isolation & product yield
  [#7 Subagent research firewall guidance] → [#10 Product-track playbook + ChaosGauge link]

Wave S4 — Close the mutate loop (opt-in)
  [#9 Periodic meta-optimize] → [#8 Eval-gated draft skill PRs (opt-in)]

Out of scope forever (unless prefs unlock): always-on SessionStart full skill/frontmatter scan; native Task Observer; auto-merge skill mutations; pretending GAP-EXIT2 hard process blocks.


4. Top 10 recommendations (ranked)

Each item: why · token impact · dependency · proposed CE shape · Task Observer accept/reject.

1. Learning metrics closed loop (queued → adopted → token delta)

  • Why: LogAct shows shared structured logs + light supervisor yield more unique work at lower tokens. CE already appends via learning.py / .memory / phase ledger but does not summarize adoption.
  • Token impact: ↓↓ (zero-LLM doctor/CLI; avoids re-discovering known lessons)
  • Dependency: none (S0)
  • CE shape: learning.py metrics|summary + doctor --json.learningMetrics fields: queued, submitted, open/closed issues, estimatedTokens sum, optional adopt/reject labels; link issue numbers only.
  • vs Task Observer: Reject Observer. Metrics are offline/zero-LLM over existing logs.

2. Silent-on-success / errors-only verification wrappers

  • Why: HumanLayer: green test dumps rotate agents into the dumb zone; success must add zero context.
  • Token impact: ↓↓↓ on Check/Stop
  • Dependency: after #1 optional; independent OK
  • CE shape: references/silent-verify.md + thin scripts/chaos-engine helper used by delivery Check + optional Stop verify; exit 0 → no stdout; fail → stderr + exit 2 where honored.
  • vs Task Observer: Reject Observer. Pure hook/script back-pressure.

3. ERL-style heuristic retrieve once per task (not every turn)

  • Why: ERL: post-task heuristics + top-k inject at new task start beats always-on observation.
  • Token impact: ↓ always-on; slight ↑ once at triage (bounded top-k, privacy-safe)
  • Dependency: prefers #1 metrics for ranking; wake pack / retrieve patterns (#5624)
  • CE shape: .chaos-engine-state/heuristics/ or Memory category; SessionStart locator only; triage/router calls retrieve.py heuristics --top 3 once; never PreToolUse spam.
  • vs Task Observer: Reject Observer. Adopt ERL retrieve-once; reject continuous scan (#5643).

4. Significance-filtered mid-session capture (skill + optional soft PostToolUse)

  • Why: Under Caveman ultra, dual-track capture is easy to miss until Stop; need cheap friction marks without full observation protocol.
  • Token impact: ↓ vs batching failures; ↑ tiny if soft hook emits one-line marks
  • Dependency: taxonomy + learning.py; must not load self-improve refs mid-turn
  • CE shape: optional PostToolUse recorder for deny/fail signatures → state file; Learning Session reads marks. Default off or ultra-cheap. Significance filter: errors, repeated denials, doctor recovery — not every edit.
  • vs Task Observer: Reject Observer. Filter + deferred heavy path only.

5. CLI-over-MCP preference (iron law + zero-LLM catalog)

  • Why: HumanLayer: CLI in training data beats MCP schema tax for gh/git/docker/db.
  • Token impact: ↓↓ system prompt + tool-result size
  • Dependency: zero-llm-catalog / Level-1 (#5621/#5622)
  • CE shape: router iron-law row + catalog entries; MCP only when no equivalent CLI exists.
  • vs Task Observer: N/A (tooling policy).

6. SkillOpt-style SKILL.md compression audit (propose, don’t auto-apply)

  • Why: Red Hat ACE / SkillOpt: ~90% volume cut with quality preserved in experiments; L2 <500 lines.
  • Token impact: ↓↓ on skill activation
  • Dependency: eval-parity (#5625); metrics #1 help prioritize which skills
  • CE shape: offline CLI proposing compressed SKILL.md + refs split; CI eval gate; issues-first PR by human. No live auto-edit.
  • vs Task Observer: Reject auto-mutate. Aligns with issues-first / opt-in draft PRs later.

7. Subagent / context-firewall guidance for research & explore

  • Why: HumanLayer + capitalandcompute: isolation keeps parent in smart zone; explore noise stays in child.
  • Token impact: ↓ parent context (child spend may ↑ — net quality win)
  • Dependency: host adapters; local-coding-delegate / omniroute optional
  • CE shape: short Level-1 + router rule: research/multi-file explore → subagent/Task when host supports; return filepath:line citations only.
  • vs Task Observer: N/A.

8. Eval-gated draft skill PRs — opt-in phase 2

  • Why: Hermes/SkillOpt closed loops need eval; CE locked issues-first until ready.
  • Token impact: ↑ CI only; ↓ bad skill merges long-term
  • Dependency: #1 metrics, #6 compress, #5625 fixtures green, portable finalize stable
  • CE shape: flag CHAOS_ENGINE_SELF_IMPROVE_DRAFT_PR=1; draft PR from staged branch after eval-parity + unit tests; never auto-merge.
  • vs Task Observer: Reject Observer. Opt-in mutate path only after eval.

9. Periodic meta-optimize over shared logs (not continuous)

  • Why: LogAct supervisor: periodic introspection of shared logs beats per-turn gossip.
  • Token impact: ↓ vs continuous; bounded scheduled LLM or zero-LLM cluster
  • Dependency: #1 metrics; Learning Session history
  • CE shape: operator /learning meta or weekly script clustering open learning issues + .memory facts → ranked backlog issue; no SessionStart involvement.
  • vs Task Observer: Reject Observer. Periodic only.

10. Product-track self-improve playbook (SHAFT + ChaosGauge)

  • Why: Dual-track requires equal product discipline; recent #5644/#5646/#5647 show product lessons already valuable.
  • Token impact: ↓ rediscovery; slight ↑ at Learning Session when product work occurred
  • Dependency: taxonomy; metrics #1; silent-verify #2 for product Check
  • CE shape: skills/self-improve/references/product-track.md — issue templates, link tests/ChaosGauge, prefer CLI/doctor/silent-verify product fixes over essay tickets.
  • vs Task Observer: Reject Observer. Delivery-Stop capture only.

5. Accept / reject matrix (Task Observer principles)

PrincipleDecisionCE encoding
Always-on SessionStart full scanREJECTLocator-only; SESSION_START_MAX_BYTES
Continuous observation every turnREJECT#5643; research-adopt-reject
Lean core + on-demand refsACCEPTself-improve SKILL + references
Capture vs review splitACCEPTqueue now; human/CI apply later
Dual-track harness + productACCEPTtaxonomy + Learning Session checklist
Auto-stage / auto-install skill patchesREJECT (phase-2 draft PR opt-in only)learning.py issues-first
Weekly auto-mutate live skillsREJECTmeta-optimize proposes backlog only
Post-delivery Learning Session gateACCEPT#5641 Stop hooks
ERL retrieve-once heuristicsACCEPT (adapted)Top 10 #3
LogAct metrics / periodic supervisorACCEPT (adapted)Top 10 #1 #9
Silent-on-success verifyACCEPTTop 10 #2

6. Implementation notes (when executing children)

  1. Prefer one condensed PR per wave (S0→S4); children under the epic; Fixes #<child> never the epic.
  2. Every lasting behavior → portable chaosengine-harness (hooks/skills/scripts/doctor) + host adapters as needed — harness parity.
  3. Expand eval-parity fixtures when adding gates; never weaken fixtures.
  4. Keep Ponytail and Caveman ultra.
  5. No Cursor Cloud Agents for implementation workstreams.

9. Supersession notice

The 2026-09-07 dependency-ranked top-10 (health plane → repair → zero-LLM locator → L1 catalog → research ledger → wake/retrieve → token-budget → eval-gated improve → fixtures → Heal/GAP-EXIT2) is complete via #5619–#5625 + #5641. Do not reopen those tickets. Track new work from §4 Top 10 under the self-improve epic filed alongside this document.

ChaosEngine Self-Improve — Research Synthesis

Companion to self-improve-master-plan.md. Dated (Africa/Cairo): 2026-09-08. Citations only — no always-on injection.

2. Online research synthesis (cited)

2.1 Context surfaces — put the right thing on the right primitive

Sources:

SurfaceLoadEnforceCE mapping
AGENTS/CLAUDE.mdEvery turnSoft ~70%Keep lean (<~200–500 lines); pointers only
HooksEvent; 0 context unless outputHard 100%Safety, Learning Session Stop, research gate
SkillsName+desc always; body on demandSoftRouter + self-improve + companions
SubagentsOn spawn; isolatedSoftResearch/explore firewall
Memory/storesLocator / on demandSoftWake pack locator; retrieve orchestrator
MCPNames cheap; schemas deferredToolsPrefer CLI when training-data CLI exists

CE accept: hooks for must-happen; skills for judgment; lean host md files.
CE reject: stuffing safety or Learning Session duty into AGENTS.md alone.

2.2 HumanLayer harness lessons

Sources:

  1. CLI over MCP when CLI is in training data (gh, git, docker, …) — lower schema tax + composability (jq/rg).
  2. Subagents for context isolation, not role-play org charts.
  3. Verification silent-on-success / errors-only — swallow green test dumps; exit 2 on failure to re-engage.
  4. Hooks for control flow; Stop hooks for completion gates (maps directly to #5641).

2.3 Hooks economics

Sources: capitalandcompute harness guide; Anthropic hooks docs (code.claude.com/docs/en/hooks-guide); CE lifecycle-hooks.

  • SessionStart / Stop / PreToolUse run outside the conversation budget.
  • Stop is the right place for Learning Session and verify gates.
  • Aligns with CE: locator SessionStart; Learning Session on Stop (#5641); no always-on observer.

2.4 Self-improve without always-on Task Observer

Sources:

PatternToken-safe shape for CEReject
ERL heuristicsPost-task extract → store; retrieve top-k once at task start (locator + bounded)Inject every turn / full SessionStart scan
GenericAgent L0/L1Wake pack + Level-1 catalog; L2/L3 on demandAlways-on L2/L3 dumps
LogAct supervisorShared structured logs (phase ledger, learning queue, .memory) → periodic meta-optimizeContinuous Task Observer
SkillOptOffline compress SKILL.md (~90% volume cut reported by Red Hat ACE) with eval gateAuto-mutate live skills

2.5 Maximum output, minimal / zero tokens — governing factors

  1. Deterministic scripts & hooks before LLM narration (Red Hat: scripts cut RCA cost ~26%).
  2. Progressive disclosure (L1/L2/L3) — description is the router.
  3. Silent success on verify (HumanLayer).
  4. Fewer sharper tools / CLI>MCP (HumanLayer; Vercel tool-thinning lesson in prior dossier).
  5. Shared logs → unique work (LogAct: +17% work / −41% tokens with supervisor over logs — aspirational, not a CE claim of identical gains).
  6. Measure then mutate — metrics before growing prompts.
  7. Issues-first durable encode — portable overlay parity (#5645).

Supporting dossier: /workspace/agentic-harness-research.md (Winder, Goose, OpenHands, Hermes, agentskills.io, Gu system-scaling).


7. Key online sources used

  1. https://amux.io/guides/context-engineering/
  2. https://capitalandcompute.net/blog/claude-code-harness-guide/
  3. https://next.redhat.com/2026/07/28/building-skills-for-ai-agents-pitfalls-and-best-practices/
  4. https://www.humanlayer.dev/blog/skill-issue-harness-engineering-for-coding-agents
  5. https://www.humanlayer.dev/blog/context-efficient-backpressure
  6. https://www.humanlayer.dev/blog/advanced-context-engineering
  7. https://agentskills.io/specification
  8. https://arxiv.org/pdf/2603.24639v2 (ERL)
  9. https://arxiv.org/pdf/2604.17091 (GenericAgent)
  10. https://arxiv.org/html/2604.07988 (LogAct)
  11. https://github.com/microsoft/SkillOpt
  12. https://code.claude.com/docs/en/hooks-guide
  13. https://winder.ai/ai-agent-harness-comparison/ (prior dossier)
  14. Local: /workspace/agentic-harness-research.md, /workspace/chaos-engine-top10-optimizations.md (superseded)

8. Consult notes (optional Grok CLI)

Captured /tmp/ce-self-improve-consult.txt (model host frontier model). Alignment with this plan:

  • Measure first (LogAct counters) before growing prompts.
  • Silent verify + CLI>MCP + SkillOpt propose-only + ERL once-per-task.
  • Reaffirm reject always-on scanners / weekly auto-mutate.
  • Ordering tip from consult: metrics → silent verify → CLI>MCP → submit hygiene → SkillOpt → ERL → eval ratchet → GAP-EXIT2 UX.

Allure Report Internals

Results model

allure-results/ holds one *-result.json per test attempt (status: passed/failed/broken/skipped; failed = assertion, broken = unexpected exception), plus *-container.json (fixtures/grouping), attachments by UUID, categories.json, environment.properties, history/ (trend, keyed by historyId and testCaseId include suite identity before the report is built: platform, and language when language is a separate cohort. A parameter that already carries platform or language is counted once. The same platform and language still share both ids, so a real retry stays a retry. Rewriting ids in the generated HTML is not the fix. A shared display name across platforms or languages is a broken report, not a retry.

Verdict discipline (SHAFT hard rule)

An empty or unexpectedly small allure-results invalidates any pass/fail verdict — count result JSONs and executed tests FIRST, and inspect failed, broken, retried, and skipped attempts separately before declaring a run green (repo gotcha allure-result-population). A suite that "passed" with 3 result files when 40 tests exist means the run silently didn't execute.

Single-file report + patching

allure generate --single-file inlines everything into one huge index.html (tens of MB). Never Files.readString + String.substring to patch it — that's 2-3 whole-file copies on the heap and OOMs on large reports (SHAFT issue #3407). The proven pattern (PR #3433, AllureManager):

  1. Pass 1: stream with a 64KB sliding-window byte scan, carrying a tail overlap so markers spanning read-chunk boundaries are found; locate patch-id presence, first </head>, last </body> offsets.
  2. Pass 2: sequential copy through a sibling temp file, splicing patches at the recorded offsets; atomic move into place. Skip if already patched.
  • ASCII byte search is UTF-8-safe (multibyte sequences can't contain '<').
  • If a patch inserts before </body>, every later offset shifts by the sum of all earlier patch lengths — compute final offsets, don't re-scan.
  • Windows: replace-moves are not atomic under file-watcher/AV contention; use ATOMIC_MOVE with fallback and never leave a half-written index.
  • Allure 3 suite widgets may be base64. A plaintext miss is not an empty report. Decode the widget payload with a bounded scan before deciding the suite is absent. Do not Files.readString the whole single-file HTML (#3407). A name that is both plaintext and base64 counts once. An empty results directory (no *-result.json) stays unverified. Decoding does not invent a suite.

Ops rules

  • Never allure serve/allure open in agent sessions (guard-enforced); generate only. History carries over by copying history/ from the previous report into results before generating.
  • Report JS executes at open; theme/branding patches belong in <head> (style) and before </body> (script) — SHAFT's patch constants must never themselves contain </head>/</body> literals or the marker scan breaks.

Maven-provisioned Allure 3 CLI (#5801 / #5815)

SHAFT never calls a user PATH allure binary (Allure 2 drop: #5798/#5800). Resolution order:

  1. Maven cache (preferred): ~/.m2/repository/allure/allure-cli/<allure3Version>/node_modules/allure/cli.js invoked as node <cli.js> (portable Node under ~/.m2/repository/nodejs/ when PATH node is missing).
  2. npx --yes allure@<allure3Version> on PATH.
  3. Portable Node + its npx.

Maven zip coordinates (air-gap, #5815): io.github.shafthq:allure-cli:<allure.cli.version>:zip (built from allure-cli/ via mvn -f allure-cli/pom.xml clean install -Dgpg.skip or scripts/maintenance/build-allure-cli-zip.sh). Zip unpacks into the runtime cache layout above. Artifact on disk: ~/.m2/repository/io/github/shafthq/allure-cli/<version>/allure-cli-<version>.zip.

Operators / CI:

# Online / first warm-up (npm):
mvn -Pprovision-allure-cli -pl shaft-engine -am initialize

# Air-gap when the zip is in local or corporate Maven repo (never calls npm):
mvn -Pprovision-allure-cli-maven -pl shaft-engine -am initialize

Engine bootstrap also unpacks a local Maven zip before falling back to npm. Root POM property allure.cli.version must stay aligned with Internal.allure3Version() and allure-cli/pom.xml (currently 3.19.1; independent of allure-bom 3.1.1 Java adapters).

Overrides: -Dallure.cli.cacheRoot=..., -Dallure.cli.mavenZip=..., -Dallure.cli.skipProvision=true (engine bootstrap only).

Maven Central (#5833): allure-cli/ stays outside the shaft-parent reactor (version = Allure 3). mavenCentral_cd.yml deploys it standalone after the SHAFT reactor using the same GPG/OSSRH secrets. After publish, air-gapped CI only needs a corporate Maven mirror of io.github.shafthq:allure-cli:<allure.cli.version>:zip plus -Pprovision-allure-cli-maven (no npm). scripts/ci/verify_maven_central_release.py expects the zip (+ .asc / .pom).

Appium & Mobile Toolchains

Stack shape

Appium 2.x = server + installable drivers (uiautomator2, xcuitest, windows) + plugins, each versioned independently — pin them in CI, appium driver install at run time is a supply-chain and race liability. SHAFT talks to it via mobile_* MCP tools and the shaft-mcp mobile toolchain service; web emulation (Chrome device metrics) is a separate, cheaper path from real native automation.

Incidents that must not repeat

  • Never use appium driver run windows install-wad in CI: it lists api.github.com/repos/microsoft/winappdriver/releases unauthenticated and 403s under shared Actions IP rate limits (#5697). Install a pinned MSI via scripts/ci/install_winappdriver.ps1 (or an equivalent authenticated / cached asset path). The older race where install-wad returned before the MSI finished still applies if you resurrect that path — poll for WinAppDriver.exe before starting Appium (PR #3408).
  • Chrome's emulated-device list drifts: Chrome 143 removed "Pixel 5", silently breaking every mobile-web-emulation flow pinned to it. Treat device names as data that rots; validate against the running browser and keep a fallback metrics map (PR #3421).
  • Keep the server log retrievable on failure: in PowerShell CI, Receive-Job BEFORE Stop-Job, or the diagnosis evidence dies with the job.

Context switching

Native automation lives in NATIVE_APP context; webviews appear as separate contexts needing matching chromedriver versions. mobile_get_contexts before switching; hybrid-app flakiness is usually a context (not locator) problem. Accessibility-tree reads (mobile_get_accessibility_tree) beat raw XML source dumps for both tokens and stability.

CI reality

  • Android emulators in CI need: KVM/HAXM availability check, boot-completed polling (sys.boot_completed), and generous first-boot timeouts; never fixed sleeps.
  • iOS/Safari and device-cloud (BrowserStack) failures are frequently provider-side or target-site-side: before touching code, curl the target page and check the same test's pass history — a locator that passed 6 of 7 scheduled runs and the element still existing live means external incident, not regression (PR #3408 triage).

GitHub Actions CI Forensics

Triage method (proven on multi-job red runs)

  1. Pull logs with gh run view <id> --log / --log-failed; never diagnose from the UI summary alone.
  2. Group failures by EXACT exception signature first — a 10-job red run is usually 2-3 signatures plus noise (PR #3408: 4 jobs one signature, 2 another, 3 unique).
  3. Classify each group: code defect / stale-already-fixed (check merge timestamps vs run trigger time) / external provider or target-site incident (curl the target live; check the same test's pass history).
  4. Only then fix — one root cause per commit, workflow-level fixes get extra verification (higher blast radius than app code).

Signals that lie

  • workflow_run conclusion success ≠ delivery: job-level if: skips count as success.
  • Releases created with the default GITHUB_TOKEN emit NO release event (anti-recursion) — downstream on: release workflows silently never run.
  • A job can stay green while a post-test teardown step OOMs/crashes (post-test bookkeeping isn't test failure) — read to the END of logs of "passing" jobs when hunting incidental defects.
  • Fail-fast pipelines mask later-stage bugs: after fixing failure N, expect a NEW failure at stage N+1 that no run has ever reached before.

SHAFT-specific gates

PR-blocking: CodeQL + dependency-review, path-filtered intellij-plugin, shaft-pilot-release. E2E suites run on their existing nightly schedules; the guidance validator is manual. So "CI is green" on a PR proves much less than it suggests — run scripts/ci/local_gate.py (changed-module verify) and the relevant scoped tests yourself.

Workflow YAML edits

  • Parse with PyYAML locally before pushing (syntax + step order).
  • Unawaited elevated Start-Process in PowerShell steps returns instantly — poll for the real readiness condition, never trust step exit.
  • Keep diagnostic artifacts alive on failure: Receive-Job before Stop-Job, upload logs in if: always() steps.
  • Sharded runs: merge via explicit post-step (SHAFT: report_merge_shards, assemble_shard_blob.py) — shard-local reports alone hide cross-shard patterns.

Nightly-only E2E execution

Agents never manually start, rerun, or replace E2E Tests or Local E2E Tests, including targeted dispatches. Prove a fix with the smallest directly affected local headless test or local composite-action replay, then use exact-head PR checks as the delivery gate. Existing scheduled nightly workflows are the sole E2E execution owner. Agents may inspect scheduled results and fix failures; they do not wait for, trigger, or substitute remote E2E proof.

Local guard replay — prove a composite action's shell logic without dispatch (#4119)

This proves a workflow-level guard locally without running any remote job. A composite action's run: step is plain shell once GitHub's ${{ inputs.* }} templating is peeled off — extract, parameterize, fixture, replay.

  1. Extract the step's shell body verbatim (e.g. .github/actions/post-test-report/ action.yml's "Write Summary and Check Test Results" step, id collect_results, action.yml:119-252).
  2. Parameterize every ${{ inputs.<name> }} token as a same-named shell variable (${{ inputs.module-directory }} -> $MODULE_DIR, ${{ inputs.job-name }} -> $JOB_NAME) with one sed pass — no YAML or Actions runtime involved.
  3. Default $GITHUB_STEP_SUMMARY / $GITHUB_OUTPUT to /dev/null when unset: the script appends to them unconditionally, and GitHub sets them but a local shell doesn't.
  4. Build a fixture tree from a real run's own evidence, one directory per candidate input value: an allure-results/*-result.json file per scenario and an allure-report/summary.json carrying the real totals. Directories the guard should find nothing in are simply absent — that absence is the condition under test.
  5. Run the extracted script once per candidate input value and read the exit code and printed summary line. No runner queue or waiting.

Proven this way for #4119: MacOSX_Safari_Cucumber_BrowserStack's 13 Cucumber scenarios actually land under shaft-engine/ (its -Dtest selector only matches shaft-engine's cucumberTestRunner.CucumberTests, pulled in via -am), never under the job's own -pl target shaft-browserstack/. Replaying action.yml's guard (RESULT_COUNT at action.yml:149-155, the no-summary branch at action.yml:239-243) against a fixture built from that real 13-passed evidence:

  • MODULE_DIR=shaft-browserstack (the #4079 override, matching -pl rather than where output actually lands) — Allure Report Summary: Total=0, Passed=0, Failed=0, Broken=0, Skipped=0 then ::error::No Allure summary was generated for shaft-browserstack, and shaft-browserstack/allure-results does not exist., exit 1: the exact false failure #4119 reported on a fully passing run.
  • MODULE_DIR=shaft-engine (the default at action.yml:33; e2eTests.yml:754-760 no longer overrides it, fixed by #4135) — Allure Report Summary: Total=13, Passed=13, Failed=0, Broken=0, Skipped=0 then All tests passed according to Allure report., exit 0.

Generalizes to any composite-action guard: peel the templating, fixture the filesystem state the shell actually branches on, and replay. Scheduled nightly owns any later remote E2E evidence.

IntelliJ Platform Plugin Development (shaft-intellij)

Build reality

Use the checked-in Gradle 9.3+ wrapper with JDK 25 as the build runtime; source/target compatibility remains 17 for IntelliJ Platform compatibility. UI evidence comes from ShaftPluginScreenshotRendererTest -Dshaft.intellij.screenshotDir=... — visible UI changes must regenerate screenshots for the PR.

Issue #3784 recorded one Windows OpenJDK 25.0.1 G1 crash and introduced a temporary JDK 21 ceiling. That build already used Gradle 9.3, so the wrapper version does not explain the crash. Issue #4779 raised the ceiling to JDK 25 only after repeated fresh Windows JDK 25 builds passed and the supported Gradle/JDK combination was reconfirmed; newer daemon JDKs remain rejected. CI must invoke shaft-intellij/gradlew so its runtime support stays coupled to the checked-in wrapper instead of a separately pinned system Gradle.

Threading & UI rules

  • EDT-only for Swing mutation; use invokeLater/Application.invokeLater from background threads, and never block the EDT on MCP/process I/O — every shaft-mcp call from the plugin belongs on a pooled thread with an EDT callback.
  • StatusText/getEmptyText() placeholder lines clip at a fixed internal width (~42 chars) regardless of component width and never wrap — don't fight it with shorter fragments; paint placeholders manually (paintComponent + FontMetrics word-wrap) and expose text via a client property for tests (SHAFT's PlaceholderTextArea).
  • Long-lived background pollers (heartbeats spawning a fresh MCP process every 30s per panel) are a leak pattern — lifecycle-scope them.
  • Capture this::method as a field when the same reference must later be removed (e.g. addStateChangeListener/removeStateChangeListener pairs in addNotify()/removeNotify()) — re-evaluating a method-reference expression is not guaranteed to produce an identity-equal object, so a fresh one at the removal call site silently fails to remove anything (issue #3621).

Trust & failure surfacing (hard-won)

  • An MCP tool result with isError: true inside a successful JSON-RPC envelope IS a failure — surface the tool's own text; masking it as "Finished/OK" hides every downstream failure mode (PR #3396's highest-blast-radius fix).
  • Gate SHAFT-specific actions on real project detection (pom/gradle contains shaft coordinates; fail-closed, cached) so the plugin is inert in ordinary projects.
  • Setup-step badges derive ONLY from real verification checks, never from "the user clicked the button" (PR #3427). No phantom recorder steps: suppress browser-synthesized events that don't map to a user action.
  • Long-running work (recordings) must never belong to an ephemeral agent turn/process — start it in a service whose lifetime the tool window owns (PR #3431).

Testing

Plugin tests run headless via the IntelliJ test framework; Swing panels are unit-testable if logic is separated from EDT wiring. Playwright/webapp tools cannot see Swing — evidence is the screenshot renderer, not a browser.

Any test that installs a real platform L&F (UIManager.setLookAndFeel) must register LookAndFeelIsolationExtension (shaft-intellij/src/test/java/.../ui/, issue #3782) so it snapshots/restores the LookAndFeel instance, JBColor dark flag, and declared UIManager keys on the EDT — UIManager.put() overrides persist across L&F switches by Swing design, and IntelliJ's DefaultTreeUI only becomes the active TreeUI delegate once a platform L&F has been installed at least once in the JVM, a latch that restoring the L&F instance does not undo. Confirmed live (issue #3786): running the whole module's gradlew test together with -Dshaft.intellij.screenshotDir=... still breaks ShaftTestsPanelTest's tree-selection tests via that sticky latch even with the extension in place — always scope screenshot regeneration with --tests ShaftPluginScreenshotRendererTest, never a bare full-suite run.

Locator Design & Self-Healing

Locator ladder (stop at the first stable rung)

test id > role + accessible name > stable attributes in a stable container > text (assertions only; i18n-brittle) > structural CSS/XPath (last resort; never index- or style-class-based). SHAFT's By-based APIs and aria-snapshot assertions (matchesAriaSnapshot, partial-subset semantics) reward the role+name rung heavily.

For SHAFT-generated/codegen output specifically, this collapses to three legal tiers (shaft-locator-design owns the full policy; enforced mechanically by shaft-capture's LocatorPolicy, issue #4271): (1) a unique, author-written id through the SHAFT locator builder (hasAnyTagName().hasId(...)) — never a framework-recycled id such as :r1:, mat-input-3, cdk-overlay-0, ember1234, j_idt42, ctl00_... or sc-bdVaJa; (2) ARIA-role-powered XPath via the same builder (hasRole(...), chained with hasNormalizedText/hasAttribute/context); (3) native relative By.xpath(...) only when the element has neither. An element matching no tier fails generation rather than degrading. Never emit SHAFT.GUI.Locator.xpath(...) or the raw SHAFT.GUI.Locator.id/name/cssSelector/className/tagName(...) factories. Smart Locator (inputField/clickableField) is never emitted into generated code — test_code_guardrails_check flags it (SMART_LOCATOR) — and is legitimate only for a human's own throwaway exploration snippet.

Recorder truth rules (SHAFT-specific, hard-won)

  • Recorded locators must map 1:1 to real user actions; suppress browser-synthesized phantom steps (subframe storage restores, PR #3432; click-tracking state, PR #3427 — never reintroduce).
  • Codegen validates by compile-and-replay (GeneratedTestValidator) — a generated locator is unproven until replay passes headlessly.

Self-healing (shaft-heal / doctor)

  • Healing proposes a NEW locator from a failure snapshot (doctor_propose_healed_locator); score candidates by multi-signal similarity (attributes, text, neighbors, position) and NEVER auto-accept silently — healed locators are review-gated changes, else suites drift to asserting the wrong elements.
  • Confidence scoring must penalize uncited/unevidenced output symmetrically — SHAFT's DoctorConfidenceScorer once scored zero-citation recommended actions 85/100 because only hypotheses had a penalty branch (PR #3344). When extending scorers, check every factor has both directions.
  • Persisting healing history: append/merge, never rewrite the store (a data-wiping rewrite bug shipped once, issue #3315).
  • healer_run_failed_test can burn multiple real Maven builds — precheck that Allure output exists before invoking heavy loops.

Debugging a "broken" locator

Order: is the element in a different frame/context → is there a wait/race → did the DOM actually change (diff live DOM vs expectation) → only then rewrite the locator. Most "locator rot" reports are frame or timing bugs.

Maven Release Engineering

SHAFT publishing shape

Multi-module reactor: core shaft-engine, optional shaft-* feature modules, BOM and compatibility/publication modules. Consumers pin one version; the BOM must stay internally consistent. Release flow runs GPG signing — locally it fails without -Dgpg.skip=true; the proven local install for live validation is mvn -pl shaft-engine,shaft-capture,shaft-mcp install -DskipTests -Dgpg.skip=true.

Dependency convergence

  • maven-enforcer-plugin lives in root pluginManagement, activated in shaft-engine (banned-deps + convergence) — it is the cheap local mirror of merge-time CI breakage: run a targeted mvn -pl <module> verify -DskipTests before pushing dependency changes.
  • Transitives can be internally inconsistent: jython-slim:2.7.4 declares jffi 1.3.13 direct AND jffi-native 1.4.0 — converge both classifiers in dependencyManagement, not just the main artifact (PR #3344).
  • Excluding a transitive (e.g. org.openpnp:opencv from sikulixapi) without re-declaring it directly compiles fine and dies at runtime with NoClassDefFoundError — check what sibling modules with the same need declare (shaft-sikulix vs shaft-visual, PR #3408).

Resources & packaging traps

  • An explicit <resources><includes> allowlist silently drops every file not listed — new runtime resources (JS payloads like axe.min.js) never reach target/classes or the jar while unit tests that read from source tree still pass (PR #3383). Verify packaging with unzip -l on the jar.
  • Version shadowing: locally-installed .m2 jars mask Central artifacts of the same version; a fix that "works locally" may be running a patched local build. Check jar timestamps/hashes when validation seems too easy.

Verdict rules

  • workflow_run conclusion success only means no job FAILED — job-level if: skips count as success. A "green" release workflow can have delivered nothing (repo gotcha; also: releases created with the default GITHUB_TOKEN emit NO release event — dependent workflows never fire).
  • Release-candidate pipelines fail fast and mask later-stage bugs; a new failure after a fixed one is usually a genuinely new issue further along, not a regression of the fix.

MCP Protocol Engineering (shaft-mcp)

Transport truths

stdio transport = newline-delimited JSON-RPC on stdin/stdout; stdout is PROTOCOL-ONLY — any stray print (logging, Maven banners, JVM warnings) corrupts the stream. All diagnostics go to stderr. The proven E2E validation pattern: a throwaway Python stdio driver against the real java @shaft-mcp.args process with cwd = a consumer project (repo workflow live-validating-shaft-mcp-fixes-end-to-end-via-a-scripted-stdio-driver).

Error semantics (three layers, all different)

  1. Transport/JSON-RPC error — request failed.
  2. isError: true in a SUCCESSFUL tool result — the tool failed; clients that treat the envelope's success as the verdict mask every real failure.
  3. Domain "soft fail" in payload text — needs explicit checks. Always propagate layer-2 text to the user/client verbatim.

Workspace roots

The workspace root resolves from the CLIENT LAUNCH DIRECTORY; installers write only -Dshaft.mcp.fallbackWorkspaceRoot, never -Duser.dir or a hard workspace root, and a bare user home must never become a workspace (PR #3428 decisions, validator-enforced). Scoping to Path.of(".") (IDE cwd) instead of the open project is a recurring bug shape.

Long-lived state vs one-shot turns

Recordings/captures are stateful sessions that must outlive the initiating request. Failure shapes seen live: capture session JSON vanishing mid-run on Windows (non-atomic replace-move under AV/file-watcher contention) wedging status into an "already active vs not started" split-brain — lifecycle code must self-heal from a missing store, never wedge until process death (PR #3431).

Client-side gotchas

  • claude --print + acceptEdits auto-DENIES every Bash/MCP tool call unless --permission-prompt-tool (an MCP-over-HTTP approval bridge) is wired — flag exists in the binary but not --help (PR #3420/#3411).
  • Live-probe any CLI's wire behavior with a script before implementing against it; a prior SHAFT feature shipped dead because it assumed a control-channel that doesn't exist on raw stdio (PR #3373→#3410).

Tool design

Verb_noun names, flat args, one tool = one user-meaningful outcome, structured content out. Reject dangerous generated code server-side (test_code_guardrails_check pattern) rather than trusting client prompts.

Selenium WebDriver BiDi

Model

Selenium exposes BiDi (the WebSocket protocol alongside classic WebDriver HTTP) via Augmenter/HasBiDi; the pinned version lives in the root pom. Key domains: script (preload scripts, evaluate/callFunction), browsingContext (tree, navigation, screenshots), network (intercept/continue), log. Everything is per-browsing-context; "the page" is never one context — every iframe, popup, and worker-adjacent frame is its own context with its own lifecycle.

Rules that prevent real SHAFT bugs

  • script.addPreloadScript runs in EVERY browsing context, including same-origin subframes. Any injected UI or persisted state must be gated window === window.top; same-origin subframes SHARE the top frame's sessionStorage, so a subframe restoring "persisted UI state" creates phantom events (SHAFT recorder incident, PR #3432: phantom Navigate rows).
  • Injection order races with subframe load timing are real and CI-speed dependent. Deterministic repro trick: delay a fixture iframe's response (?frameDelay=900) to force the slow-CI initialization order locally.
  • Channel messages (script.message from channel callbacks) arrive on the WebSocket dispatch thread — never block it, queue and process elsewhere.
  • BiDi session lifetime is tied to the driver; a recording/capture session must own a driver whose lifetime outlives the initiating request. Never start a BiDi-dependent recording inside an ephemeral process/turn that exits right after (SHAFT incident: recordings died seconds after start because capture ran inside a one-shot agent child, PR #3431).
  • Not all SHAFT driver backends guarantee BiDi (remote grids, some mobile paths) — feature-detect, degrade gracefully, never assume.

Network capture

SHAFT mutation is CDP Fetch in BrowserNetworkInterceptor for HasDevTools only. BiDi network.addIntercept is not that path. Passive BiDi (network.responseCompleted, then size-capped network.getData) does not mock: capture headers and timings eagerly, bodies lazily. Native Appium traffic is out of band. BrowserNetworkInterceptor and HttpContractRecorder already normalize contracts — compose, don't rebuild.

Debugging

  • Enable BiDi wire logging first (-Dwebdriver.remote.verbose / Selenium's BiDi logger) — most "BiDi bugs" are ordering, visible in the log.
  • Screenshot via browsingContext.captureScreenshot works on non-focused contexts; classic screenshot only on the active top context.

TestNG/JUnit Lifecycle & Forked-JVM Properties

SHAFT's hook points

SHAFT integrates via TestNG listeners (ITestListener, ISuiteListener, IInvokedMethodListener, registered through ServiceLoader / META-INF/services) and JUnit 5 extensions — driver lifecycle, Allure wiring, and reporting all hang off these. Listener exceptions can silently kill reporting for the whole run; listeners must be defensive.

Failure-type semantics (real SHAFT fact)

Actions.report() wraps element-action failures in plain RuntimeException; AssertionError is thrown only by the separate validation/accessibility APIs. A test expecting AssertionError from an element action is wrong by construction — verify with grep which path throws what before "fixing" the production side (PR #3408 corrected a years-wrong test expectation).

Properties precedence (the #1 trap)

SHAFT auto-scaffolds src/main/resources/properties/default/* and reads custom.properties; a committed or auto-created custom.properties SILENTLY OVERRIDES -D system properties for some keys. Passing -DheadlessExecution=true is not reliable when no custom.properties pre-exists or when one sets headlessExecution=false — pre-seed a local-only gitignored custom.properties for deterministic headless runs (repo gotcha; bit PR #3344 and #3408 verification).

Forked-JVM double hop: Surefire forks the test JVM (argLine, systemPropertyVariables), and SHAFT features that fork AGAIN (replay validation subprocess) must explicitly re-force critical properties — GeneratedTestValidator.replay() once forwarded the outer JVM's own headlessExecution instead of forcing true, crashing only on display-less Linux CI (PR #3344).

Execution scoping (guard-enforced here)

  • Never unscoped mvn test on the reactor; one bad fork can crash the JVM across all modules. Use -Dtest=<Class> or -pl <module> WITHOUT -am.
  • Every potentially-browser-reaching run gets -DheadlessExecution=true.
  • Running SHAFT tests from the repo root auto-creates src/main/resources/... scaffolding at the CWD — stray untracked src/ at reactor root is generated garbage, safe to delete.
  • Parallel TestNG (parallel="methods|classes") requires everything ThreadLocal (SHAFT drivers are); a static mutable field in test scope is a flake factory.

N-run proof under failure-ignore (#5739)

The shaft-engine Surefire profile may set testFailureIgnore=true so JaCoCo can still report. Maven exit 0 then does not prove a green suite. Flake-proof / Wave N-run scripts must:

  1. Pass -Dmaven.test.failure.ignore=false on the proof invocation.
  2. After each run, parse Surefire TEST-*.xml (failures / errors) or TestNG testng-results.xml (failed) and require zero.
  3. Call python3 scripts/ci/assert_surefire_green.py <surefire-reports-dir> rather than trusting process exit alone.

Flaky-Test Stabilization & Wait Strategies

Doctrine

A flaky test is a bug with a probability attached — remove the race; never widen a sleep or add blind retries. Condition-polling with a budget (SHAFT SmartWait / explicit waits) is the only correct wait primitive; wait on the ACTUAL readiness condition (element interactable, request completed, sys.boot_completed), not a proxy (time, spinner gone).

Root-cause taxonomy (check in this order)

  1. Initialization-order races — injection vs subframe load, listener vs first event. CI is slower; order flips there first.
  2. Shared mutable state — static fields, shared sessionStorage across same-origin frames, shared temp dirs/ports across parallel tests.
  3. External world drift — target site changed, emulated-device list changed (Chrome 143 removed "Pixel 5"), provider incident. Check pass history + live target before blaming code.
  4. Environment divergence — headless vs headed, display-less Linux, Windows non-atomic file replace under AV contention, forked JVM missing a property.
  5. Genuine timing assumptions — animations, debounce, eventual consistency; make the code observable instead of guessing.

Make it deterministic, then fix it

The flake must fail on demand before you trust any fix. Techniques that worked in SHAFT:

  • Delay a fixture resource to force the slow ordering (?frameDelay=900 iframe trick, PR #3432).
  • Pin the fork/parallelism config down to one method (-Dtest=Class#method) to isolate shared-state suspects.
  • Run the suspicious pair/order explicitly; TestNG order is stable given the same XML — reproduce the exact order CI used.
  • Add temporary observation points at the boundary you suspect, not shotgun logging everywhere.

Retries

Retry analyzers are quarantine, not cure: acceptable to keep a release train moving while the root cause is tracked in an issue, never as the closing state. Allure shows retries per historyId — a rising retry count is a regression signal even while the suite stays "green".

SHAFT Skill Evaluation Prompts

Humans: these prompts pressure-test whether an agent applies a SHAFT skill. They are not the product guide (https://shafthq.github.io/).

Agents: load shaft-skills/shaft-developer/SKILL.md for product work, or AGENTS.md at the repository root when changing this repository. Do not treat these prompts as policy. Skip them unless the task evaluates skill routing.

Use these prompts through IntelliJ Assistant /partner or Guided Plan coding partner first. Fresh agent sessions are fallback coverage when subagent or model-evaluation access is approved. They are not a substitute for the repo validation checks; they pressure-test whether agents apply the skills under realistic drift.

shaft-automated-test-authoring

  1. Use $shaft-automated-test-authoring to write a TestNG SHAFT web login test for an existing project. The page has Email and Password fields and a Sign in button. Include one hard assertion and no sleeps.
  2. Use $shaft-automated-test-authoring to review this Java snippet and list only the changes required before it can be accepted: it uses Thread.sleep, driver.findElement, System.getProperty, and a hard-coded token.
  3. Use $shaft-automated-test-authoring to sketch a SHAFT API negative test from an OpenAPI contract where a missing required field should return 400. Keep reusable request/validation code outside the test.
  4. Use $shaft-automated-test-authoring for a repo that already has LoginPage.loginAs(...) and locator fields. Show that you call shaft_coding_partner_plan and extend the existing target instead of creating GeneratedLoginTest.
  5. Use $shaft-automated-test-authoring to explain why Actions.report() failures and validation/accessibility assertions must be verified against current SHAFT code instead of assumed to share one exception contract.

shaft-locator-design

  1. Use $shaft-locator-design to replace By.xpath("/html/body/div[2]/form/input[1]") for an email field whose visible label is Email and whose placeholder is Work email.
  2. Use $shaft-locator-design to choose locators for a checkout button with accessible name Checkout, a generated id, and a stable data-testid=checkout-submit.
  3. Use $shaft-locator-design to diagnose a flaky mobile locator where XPath finds two Login buttons but one has accessibility id loginButton.
  4. Use $shaft-locator-design when shaft_coding_partner_plan reports an existing emailInput locator and one missing confirmation-banner locator.

shaft-test-recording

  1. Use $shaft-test-recording to describe the MCP tool sequence for recording a WebDriver checkout flow and integrating it into existing page objects.
  2. Use $shaft-test-recording to handle a Capture result with readiness RISKY, one positional CSS warning, and no post-submit assertion.
  3. Use $shaft-recording-codegen to turn a persisted Appium recording into reusable SHAFT mobile test code without leaking typed passwords.
  4. Use $shaft-recording-codegen after a successful recording when the repo already has CheckoutPage. Include shaft_coding_partner_plan, target-specific code blocks, guardrails, and focused verification.

shaft-failure-analysis

  1. Use $shaft-failure-analysis to analyze an allure-results directory that contains zero *-result.json files but a stale summary.json says passed.
  2. Use $shaft-failure-analysis to triage a failed SHAFT web test with NoSuchElementException after a retry and a Doctor suggestion to use a new locator.
  3. Use $shaft-failure-analysis to decide whether to apply a healer-proposed assertion change when the trace shows the product now returns a different validation message.
  4. Use $shaft-failure-analysis to route a failed login repair through shaft_coding_partner_plan so the fix updates the existing page object and not a new helper.

shaft-change-verification

  1. Use $shaft-change-verification to preview a shaft_coding_partner_diff that inserts a new page-object method after loginAs, then decide whether it is safe to apply.
  2. Use $shaft-change-verification when a generated block contains Thread.sleep and driver.findElement; show that guardrails block the apply and describe the regeneration.
  3. Use $shaft-change-verification to choose and run the smallest verify_run_focused command after applying a locator change, and explain why a full suite is not needed.
  4. Use $shaft-change-verification after a cloud provider returns guardrailStatus that is not PASSED; show that you do not apply and route back to the coding-partner plan.

shaft-test-planning

  1. Use $shaft-test-planning to explore a demo storefront and produce an annotated test plan for guest checkout: discovered flows, priority order, and one automation scenario per flow.
  2. Use $shaft-test-planning when a stakeholder provides only a URL and "test the signup"; show that you explore first with MCP browsing tools and mark unverified assumptions in the plan for human review.
  3. Use $shaft-test-planning to turn an exploration transcript into scenarios ready for $shaft-test-recording, stating which flows need recording versus direct codegen.
  4. Use $shaft-test-planning on a repo that already has smoke coverage for login; show that the plan extends existing coverage via shaft_coding_partner_plan instead of duplicating scenarios.

Shaft-skills efficiency contract (#6909)

Each row is one shipped enhancement (E) or closed gap (G). The sentence is the normative text in the mapped skill file.

E01 || shaft-developer/references/routing.md || Load one specialist SKILL.md and its playbook; do not open a second skill in the same turn. E02 || shaft-developer/references/routing.md || Do not paste MCP JSON schemas into the prompt; call the tool by its exact name. E03 || shaft-cli/references/playbook.md || Prefer shaft-cli call for a stateless tool so the MCP schema is not loaded. E04 || shaft-cli/references/playbook.md || Ask shaft-cli tools --cached before a live tool listing, and do not paste the live listing into the chat. E05 || shaft-developer/references/routing.md || Return a bounded result summary, not raw page source, accessibility XML, or screenshot bytes. E06 || shaft-developer/references/routing.md || Do not load ce-pack mastery chapters unless the selected playbook names that chapter. E07 || shaft-developer/references/routing.md || Keep the skill description inside the context budget; do not duplicate a playbook inside SKILL.md. E08 || shaft-developer/references/routing.md || Stop reading a playbook once the matching practice is found. E09 || shaft-developer/references/routing.md || A completed mobile or web action names the SHAFT API, the evidence that proves it, and the failure to report. E10 || shaft-mobile-actions/references/playbook.md || Tap with driver.touch().tap(locator). E11 || shaft-mobile-actions/references/playbook.md || File silent no-ops upstream. E12 || shaft-mobile-actions/references/playbook.md || Coordinate taps are forbidden. E13 || shaft-mobile-actions/references/playbook.md || Record XCUITest or UiAutomator2 before choosing a gesture. E14 || shaft-developer/references/routing.md || Assertion text names the element and the expected state. E15 || shaft-mcp/references/playbook.md || Use locator tap or element_click for mobile controls; do not call mobile_tap_coordinates to finish a flow. E16 || shaft-mcp/references/playbook.md || Initialize mobile_native before mobile tools on a native session. E17 || shaft-mcp/references/playbook.md || Read isError and warnings before the next MCP call. E18 || shaft-mcp/references/playbook.md || Keep one MCP client for the whole stateful sequence. E19 || shaft-mcp/references/playbook.md || Redact tokens from tool output before repeating it. E20 || shaft-mcp/references/playbook.md || Close with driver_quit when the task owns the driver. G01 || shaft-mobile-actions/references/playbook.md || A non-hittable iOS element is not a passed tap. G02 || shaft-mobile-actions/references/playbook.md || driver.element().click(locator) and driver.touch().tap(locator) share one hittable outcome. G03 || shaft-mobile-actions/references/playbook.md || Do not report pass when the page source is unchanged after a non-hittable tap. G04 || shaft-cli/references/playbook.md || Generated mobile Java taps with driver.touch().tap(locator) and never tapByCoordinates. G05 || shaft-defect-reporting/references/playbook.md || Strip auth_token and other query credentials from defect evidence before filing. G06 || shaft-failure-analysis/references/playbook.md || A later assertion timeout can hide a no-op tap; inspect the tap step before widening the wait. G07 || shaft-web-actions/references/playbook.md || Native iOS and Android taps belong to shaft-mobile-actions, not this web playbook. G08 || shaft-recording-codegen/references/playbook.md || Do not generate tapByCoordinates or mobile_tap_coordinates from a recording. G09 || shaft-developer/references/routing.md || Do not load every shaft-skills playbook for completeness. G10 || shaft-cli/references/playbook.md || Prefer the checked-in catalog over dumping shaft-cli tools live output into the chat. G11 || shaft-defect-reporting/references/playbook.md || A coordinate tap that works on a device is diagnosis only, not the fix. G12 || shaft-locator-design/references/playbook.md || Mobile identity still follows the three locator tiers; coordinates are not a tier. G13 || shaft-mobile-actions/SKILL.md || On iOS XCUITest, hittable=false means the tap did not land. G14 || shaft-mcp/references/playbook.md || mobile_tap_coordinates is not a success path. G15 || shaft-cli/references/playbook.md || Inside a shaft-cli session, mobile taps still use a locator, not viewport coordinates. G16 || shaft-failure-analysis/references/playbook.md || Do not replace a silent no-op with a coordinate workaround. G17 || shaft-web-actions/references/playbook.md || An action that returns without an exception is not proof the control fired. G18 || shaft-recording-codegen/references/playbook.md || Replay-proven code is required before a mobile tap snippet is treated as done. G19 || shaft-developer/references/routing.md || Never copy session auth tokens from a bug report into logs, commits, or skills. G20 || shaft-developer/references/routing.md || When the first unique locator tier matches, stop; do not add a coordinate fallback.

SHAFT CLI Command Reference

shaft-cli is a command-line interface to the same shaft-mcp tool set. When a shaft-cli launcher is installed, prefer it over raw MCP tool calls — a plain shell command costs fewer agent tokens than an MCP schema load. Fall back to the shaft-mcp:<tool> MCP calls in shaft-mcp-tools.md when shaft-cli is not installed or a command fails. For stateful work (recordings, live browser or mobile drivers) start a shaft-cli session first or stay on a persistent MCP client — a recording must never belong to a one-shot process (see Sessions below).

Install: pass --install-shaft-cli to the shaft-mcp installer (see https://shafthq.github.io/docs/agentic/cli). The launcher lands under the platform application-data root, e.g. %LOCALAPPDATA%\ShaftHQ\shaft-cli\versions\<version>\shaft-cli.cmd on Windows.

Tool mapping

Every tool in shaft-mcp-tools.md is callable by its exact name, unprefixed:

shaft-cli call <TOOL> [key=value ...] [--args '<json>'] [--json] [--stdio-ok]
  • key=value values are coerced (true/false → boolean, numbers → numeric, {...}/[...] → JSON, otherwise string); key=value overrides --args keys.
  • --json prints the raw JSON-RPC result instead of rendered text.
  • Exit codes: 0 success; 1 tool or transport error (or --json with isError); 2 unknown alias action.

Sessions and state

One-shot commands spawn an ephemeral shaft-mcp child per call, so state (browser, driver, capture) does not survive between them. For stateful tools start a session first:

shaft-cli session start    # persistent daemon; state survives across commands
shaft-cli session status
shaft-cli session stop

Stateful commands fail fast without a session unless --stdio-ok is passed. Stateless tools (guide search, doctor analysis) need no session.

Discovery

shaft-cli tools [--json] lists every tool with its description at runtime. shaft-cli tools --cached [--json] reads the bundled generated copy of the canonical tool-index.json with no MCP server or one-shot child process. Its names, descriptions, parameter schemas, and curated metadata exactly match the shaft-mcp version shipped with that CLI; use the live command only when the running server may be a different version.

Codegen (deterministic, no MCP session)

shaft-cli codegen [--session <path>] [any CaptureCli generate flag] generates SHAFT test code directly from a capture recording file — no MCP server, no session, no --stdio-ok (Decision 6: single-session actions like codegen prefer the deterministic CLI path over MCP). --session defaults to the newest *.json file under ./recordings when omitted, mirroring capture_generate_replay's zero-arg default, so a bare shaft-cli codegen works right after a capture start/capture stop pair. Every other CaptureCli generate flag (--backend webdriver|playwright, --output-dir, --package, --class-name, --overwrite, --replay, ...) passes through unchanged; exit code 2 means no --session was given and none could be inferred.

Local infrastructure setup (direct, no MCP session)

shaft-cli setup exposes the release-coupled setup catalog and lifecycle. It defaults to the read-only EXTERNAL mode. Managed mutations require a plan file plus its exact printed digest; a stale or edited plan is rejected before the cache or durable-data roots are created.

shaft-cli setup catalog|profiles [--json]
shaft-cli setup doctor|status|verify --profile REPORTING [--json]
shaft-cli setup plan --profile REPORTING --mode MANAGED --output <absolute-plan.json> [policy options] [--json]
shaft-cli setup install|apply|update --plan <absolute-plan.json> --approve <sha256:digest> [--accept-license <id>] [policy options] [--json]
shaft-cli setup start|stop --profile <profile>
shaft-cli setup logs --profile REPORTING

The first managed provider installs verified, SHAFT-owned portable Node and Allure 3 artifacts for the REPORTING profile. start and stop return unsupported for profiles without an owned service; they never adopt or stop an unknown process. Exit codes are 0 for ready/success, 2 for invalid input or approval, 3 for missing/degraded readiness, 4 for unsupported providers, and 5 for execution or integrity failures. --cache-root and --data-root are paired advanced overrides and must both be absolute.

Plan and install must receive the same policy options because those values are bound into the approved digest: --offline, --auto-start, --prefer-system-tools=true|false, --reuse-owned-processes=true|false, --startup-timeout <ISO-8601 duration>, and --shutdown-timeout <ISO-8601 duration>.

The equivalent Java surface is SHAFT.Infrastructure. Its no-argument read-only methods use SHAFT.Properties.infrastructure; install and start require the exact SetupPlan and SetupApproval. Defaults remain EXTERNAL, offline=false, and autoStart=false. An explicit remote execution address wins for web, Grid, and mobile endpoint profiles.

Curated aliases

Shortcuts over call, same options:

  • shaft-cli browser navigate|screenshot|dom|url (session required)
  • shaft-cli element click|type|hover (session required)
  • shaft-cli capture start|stop|status|code|step-delete|step-reorder (session required)
  • shaft-cli guide search (stateless)
  • shaft-cli doctor analyze|suggest|local-ai-status (stateless)
shaft-mcp-tools.mdplaybook.mdplaybook.mdplaybook.mdplaybook.mdplaybook.mdplaybook.mdplaybook.mdcache.mdplaybook.mdplaybook.mdplaybook.mdrouting.mdsources.mdplaybook.mdplaybook.mdplaybook.mdplaybook.mdplaybook.mdplaybook.mdplaybook.mdplaybook.mdcache.mdplaybook.mdplaybook.mdplaybook.mdplaybook.mdplaybook.mdplaybook.mdplaybook.mdplaybook.mdplaybook.mdjourney-coverage-map.mdplaybook.mdplaybook.mdplaybook.mdplaybook.mdplaybook.mdplaybook.mdplaybook.mdplaybook.mdplaybook.md

Sponsors

Details

Marketplace
ShaftHQ/SHAFT_ENGINE
Source
shaft-skills
Author
ShaftHQ
Category
Testing
License
MIT
Keywords
shaft, test-automation, selenium, appium, rest-assured, java, testng, junit

More from ShaftHQ/SHAFT_ENGINE

All 4