
shaft-skills
Official SHAFT agent skills: methodology router, locator strategy, test planning and authoring, MCP-driven recording, failure analysis, and change verification.
- GitHub stars
- 422
Install
/plugin install shaft-skills@shafthqRun it inside Claude Code after adding the marketplace with /plugin marketplace add ShaftHQ/SHAFT_ENGINE.
Contents
Discovered from sourceSkills 37
Agents 82
SHAFT skill inventory
Generated by scripts/ci/shaft_skill_quality.py --write; do not edit manually.
| Skill | Canonical source | Package path | Package | Supported clients | Description | Linked local references |
|---|---|---|---|---|---|---|
| shaft-accessibility-testing | shaft-skills/shaft-accessibility-testing/SKILL.md | skills/shaft-accessibility-testing/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when testing web or mobile accessibility for WCAG conformance, keyboard and focus behavior, semantics, errors, or user impact. | shaft-accessibility-testing/references/playbook.md; shaft-developer/SKILL.md |
| shaft-android-apk-select | shaft-skills/shaft-android-apk-select/SKILL.md | skills/shaft-android-apk-select/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when choosing a universal or ABI-split APK for a local emulator versus a real or cloud arm64 Android device. | shaft-android-apk-select/references/playbook.md; shaft-browserstack-native/SKILL.md; shaft-developer/SKILL.md; shaft-mobile-env/SKILL.md |
| shaft-api-testing | shaft-skills/shaft-api-testing/SKILL.md | skills/shaft-api-testing/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when implementing or repairing SHAFT.API tests for HTTP requests, authentication, payloads, schemas, responses, contracts, or service workflows. | shaft-api-testing/references/playbook.md; shaft-developer/SKILL.md |
| shaft-assertions | shaft-skills/shaft-assertions/SKILL.md | skills/shaft-assertions/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when selecting or repairing SHAFT hard assertions, soft verifications, browser or element oracles, API checks, object checks, or failure messages. | shaft-assertions/references/playbook.md; shaft-developer/SKILL.md |
| shaft-automated-test-authoring | shaft-skills/shaft-automated-test-authoring/SKILL.md | skills/shaft-automated-test-authoring/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when implementing or repairing executable SHAFT Java tests for web, mobile, API, database, CLI, TestNG, JUnit, or Cucumber behavior. | shaft-automated-test-authoring/references/playbook.md; shaft-developer/SKILL.md |
| shaft-browserstack-native | shaft-skills/shaft-browserstack-native/SKILL.md | skills/shaft-browserstack-native/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when configuring BrowserStack App Automate for SHAFT native Android or iOS: secrets keys, device pin, bs:// apps, Local, or triage. | shaft-android-apk-select/SKILL.md; shaft-browserstack-native/references/playbook.md; shaft-developer/SKILL.md; shaft-ios-ipa-acceptance/SKILL.md; shaft-mobile-env/SKILL.md |
| shaft-change-verification | shaft-skills/shaft-change-verification/SKILL.md | skills/shaft-change-verification/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when edited SHAFT Java needs a reuse plan, preview diff, guardrail check, approved application, focused test, and evidence-backed verdict. | shaft-change-verification/references/playbook.md; shaft-developer/SKILL.md |
| shaft-cli | shaft-skills/shaft-cli/SKILL.md | skills/shaft-cli/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when a SHAFT task needs a deterministic one-shot command, machine-readable output, offline discovery, or code generation from a persisted recording. | shaft-cli/references/cache.md; shaft-cli/references/playbook.md; shaft-developer/SKILL.md |
| shaft-database-testing | shaft-skills/shaft-database-testing/SKILL.md | skills/shaft-database-testing/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when implementing or repairing SHAFT.DB checks for persisted state, queries, rows, transactions, cleanup, or database-backed integration behavior. | shaft-database-testing/references/playbook.md; shaft-developer/SKILL.md |
| shaft-defect-reporting | shaft-skills/shaft-defect-reporting/SKILL.md | skills/shaft-defect-reporting/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when reporting a confirmed SHAFT product defect with severity, impact, environment, reproduction steps, expected and actual behavior, and evidence links. | shaft-defect-reporting/references/playbook.md; shaft-developer/SKILL.md |
| shaft-developer | shaft-skills/shaft-developer/SKILL.md | skills/shaft-developer/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when starting any SHAFT testing task to select exactly one lifecycle, implementation, or tool specialist before doing the work. | shaft-developer/references/routing.md; shaft-developer/references/sources.md |
| shaft-docs-hygiene | shaft-skills/shaft-docs-hygiene/SKILL.md | skills/shaft-docs-hygiene/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when keeping SHAFT project docs lean: SoT trio, Done/In progress/Not started, hours math, adversarial review, or delete-guard. | shaft-developer/SKILL.md; shaft-docs-hygiene/references/playbook.md |
| shaft-execution-reporting | shaft-skills/shaft-execution-reporting/SKILL.md | skills/shaft-execution-reporting/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when reporting SHAFT execution to engineers with reconciled counts, coverage, environments, failures, defects, flakes, evidence, risks, and next actions. | shaft-developer/SKILL.md; shaft-execution-reporting/references/playbook.md |
| shaft-failure-analysis | shaft-skills/shaft-failure-analysis/SKILL.md | skills/shaft-failure-analysis/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when investigating consistently failed SHAFT results through evidence, controlled reproduction, hypotheses, and root-cause ownership. | shaft-developer/SKILL.md; shaft-failure-analysis/references/playbook.md |
| shaft-flaky-test-analysis | shaft-skills/shaft-flaky-test-analysis/SKILL.md | skills/shaft-flaky-test-analysis/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when SHAFT tests inconsistently pass or fail under equivalent conditions and need history, retry, timing, state, data, or environment analysis. | shaft-developer/SKILL.md; shaft-flaky-test-analysis/references/playbook.md |
| shaft-fluent-api | shaft-skills/shaft-fluent-api/SKILL.md | skills/shaft-fluent-api/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when composing or repairing SHAFT fluent browser, element, touch, assertion, verification, or report-message chains. | shaft-developer/SKILL.md; shaft-fluent-api/references/playbook.md |
| shaft-guide-search | shaft-skills/shaft-guide-search/SKILL.md | skills/shaft-guide-search/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when current official SHAFT syntax, configuration, migration, troubleshooting, or feature guidance must be found and cited before coding. | shaft-developer/SKILL.md; shaft-guide-search/references/playbook.md |
| shaft-ios-ipa-acceptance | shaft-skills/shaft-ios-ipa-acceptance/SKILL.md | skills/shaft-ios-ipa-acceptance/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when gating an iOS IPA before BrowserStack or cloud upload: iphoneos, signing, arm64, thinning, or Info.plist checks. | shaft-browserstack-native/SKILL.md; shaft-developer/SKILL.md; shaft-ios-ipa-acceptance/references/playbook.md |
| shaft-locator-design | shaft-skills/shaft-locator-design/SKILL.md | skills/shaft-locator-design/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when identifying, ranking, repairing, or generating resilient SHAFT web or mobile locators from current DOM, ARIA, or accessibility-tree evidence. | shaft-developer/SKILL.md; shaft-locator-design/references/playbook.md |
| shaft-mcp | shaft-skills/shaft-mcp/SKILL.md | skills/shaft-mcp/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when a SHAFT task needs structured tool calls, interactive exploration, or browser, mobile, driver, and capture state that persists across calls. | shaft-developer/SKILL.md; shaft-mcp/references/cache.md; shaft-mcp/references/playbook.md |
| shaft-mobile-actions | shaft-skills/shaft-mobile-actions/SKILL.md | skills/shaft-mobile-actions/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when implementing SHAFT native or mobile-web sessions, context switches, taps, gestures, rotation, keyboard, or app lifecycle behavior. | shaft-developer/SKILL.md; shaft-mobile-actions/references/playbook.md |
| shaft-mobile-env | shaft-skills/shaft-mobile-env/SKILL.md | skills/shaft-mobile-env/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when setting up or repairing a local SHAFT mobile env: JDK split, Appium pin, SDK/AVD, emulator memory, or doctor loop. | shaft-android-apk-select/SKILL.md; shaft-browserstack-native/SKILL.md; shaft-developer/SKILL.md; shaft-ios-ipa-acceptance/SKILL.md; shaft-mobile-env/references/playbook.md |
| shaft-mobile-retest-hygiene | shaft-skills/shaft-mobile-retest-hygiene/SKILL.md | skills/shaft-mobile-retest-hygiene/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when triaging SHAFT mobile retest failures: map no-ops, picker probes, mixed bytecode, progress journals, or swipe fallback. | shaft-developer/SKILL.md; shaft-failure-analysis/SKILL.md; shaft-mobile-actions/SKILL.md; shaft-mobile-retest-hygiene/references/playbook.md |
| shaft-nonfunctional-test-design | shaft-skills/shaft-nonfunctional-test-design/SKILL.md | skills/shaft-nonfunctional-test-design/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when designing measurable risk-based tests for performance, security, reliability, recovery, compatibility, scalability, usability, or operations. | shaft-developer/SKILL.md; shaft-nonfunctional-test-design/references/playbook.md |
| shaft-page-objects | shaft-skills/shaft-page-objects/SKILL.md | skills/shaft-page-objects/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when creating or refactoring reusable SHAFT page objects, component objects, locators, and user-intent methods without duplicating owners. | shaft-developer/SKILL.md; shaft-page-objects/references/playbook.md |
| shaft-recording-codegen | shaft-skills/shaft-recording-codegen/SKILL.md | skills/shaft-recording-codegen/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when a persisted SHAFT web, Playwright, mobile, or API recording must become compilable, repository-aware Java code or insertion blocks. | shaft-developer/SKILL.md; shaft-recording-codegen/references/playbook.md |
| shaft-requirements-analysis | shaft-skills/shaft-requirements-analysis/SKILL.md | skills/shaft-requirements-analysis/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when analyzing requirements for testability, acceptance criteria, traceability, contradictions, omissions, risks, assumptions, or coverage gaps. | shaft-developer/SKILL.md; shaft-requirements-analysis/references/playbook.md |
| shaft-stakeholder-reporting | shaft-skills/shaft-stakeholder-reporting/SKILL.md | skills/shaft-stakeholder-reporting/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when translating verified SHAFT coverage, outcomes, defects, trends, and residual risk into business quality or release decisions. | shaft-developer/SKILL.md; shaft-stakeholder-reporting/references/playbook.md |
| shaft-test-case-design | shaft-skills/shaft-test-case-design/SKILL.md | skills/shaft-test-case-design/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when designing reviewable test cases or scenarios with traceability, data, steps, expected results, techniques, priorities, and boundary coverage. | shaft-developer/SKILL.md; shaft-test-case-design/references/journey-coverage-map.md; shaft-test-case-design/references/playbook.md |
| shaft-test-completion | shaft-skills/shaft-test-completion/SKILL.md | skills/shaft-test-completion/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when closing a SHAFT test cycle with exit-criteria evaluation, residual risk, archived testware, lessons, maintenance handover, or completion records. | shaft-developer/SKILL.md; shaft-test-completion/references/playbook.md |
| shaft-test-data-design | shaft-skills/shaft-test-data-design/SKILL.md | skills/shaft-test-data-design/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when designing deterministic, private, isolated SHAFT test data with partitions, factories, seeding, uniqueness, cleanup, retention, or schema rules. | shaft-developer/SKILL.md; shaft-test-data-design/references/playbook.md |
| shaft-test-environment | shaft-skills/shaft-test-environment/SKILL.md | skills/shaft-test-environment/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when defining or troubleshooting SHAFT test environment readiness, parity, versions, configuration, dependencies, isolation, reset, or drift. | shaft-developer/SKILL.md; shaft-test-environment/references/playbook.md |
| shaft-test-execution | shaft-skills/shaft-test-execution/SKILL.md | skills/shaft-test-execution/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when executing approved SHAFT tests with controlled scope, preflight checks, deterministic configuration, evidence, status reconciliation, and cleanup. | shaft-developer/SKILL.md; shaft-test-execution/references/playbook.md |
| shaft-test-monitoring | shaft-skills/shaft-test-monitoring/SKILL.md | skills/shaft-test-monitoring/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when monitoring SHAFT test progress, coverage, quality gates, defects, risks, duration, flakiness, infrastructure health, or completion. | shaft-developer/SKILL.md; shaft-test-monitoring/references/playbook.md |
| shaft-test-planning | shaft-skills/shaft-test-planning/SKILL.md | skills/shaft-test-planning/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when creating or revising a risk-based SHAFT test plan covering objectives, scope, approach, resources, estimates, schedule, criteria, and monitoring. | shaft-developer/SKILL.md; shaft-test-planning/references/playbook.md |
| shaft-test-recording | shaft-skills/shaft-test-recording/SKILL.md | skills/shaft-test-recording/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when an approved live web, Playwright, native-mobile, mobile-web, Appium Inspector, or API flow must be captured with privacy-safe evidence. | shaft-developer/SKILL.md; shaft-test-recording/references/playbook.md |
| shaft-web-actions | shaft-skills/shaft-web-actions/SKILL.md | skills/shaft-web-actions/SKILL.md | shaft-skills | Claude Code; Codex CLI | Use when implementing SHAFT WebDriver or Playwright navigation, browser state, element interaction, upload, drag-and-drop, window, cookie, or screenshot behavior. | shaft-developer/SKILL.md; shaft-web-actions/references/playbook.md |
SHAFT skill quality scorecard
Generated by scripts/ci/shaft_skill_quality.py --write; scores are reviewed in quality-review.json.
| Skill | Trigger precision | Sibling overlap | Dependency depth | Reference freshness | Runnable evidence | Context cost | Description characters | Confusion checks | Review notes |
|---|---|---|---|---|---|---|---|---|---|
| shaft-accessibility-testing | 5 | 4 | 5 | 4 | 4 | 4 | 130 | shaft-nonfunctional-test-design | WCAG and assistive-use triggers separate this specialist from broad nonfunctional design. |
| shaft-android-apk-select | 5 | 4 | 5 | 5 | 5 | 5 | 112 | shaft-mobile-env; shaft-ios-ipa-acceptance | ABI vs universal APK choice for emulator versus arm64 cloud is a crisp artifact decision. |
| shaft-api-testing | 5 | 4 | 5 | 4 | 5 | 4 | 148 | shaft-automated-test-authoring | HTTP request and response evidence gives this implementation specialist a clear boundary. |
| shaft-assertions | 4 | 4 | 5 | 4 | 5 | 3 | 150 | shaft-automated-test-authoring | Oracle and verification wording is precise, but the description needs context compression. |
| shaft-automated-test-authoring | 4 | 3 | 5 | 4 | 5 | 5 | 136 | shaft-change-verification; shaft-web-actions | Executable Java test ownership is clear; modality specialists remain intentional siblings. |
| shaft-browserstack-native | 5 | 4 | 5 | 5 | 4 | 4 | 135 | shaft-mobile-env; shaft-ios-ipa-acceptance | App Automate secrets, device pins, and bs:// app ids bound cloud native runs. |
| shaft-change-verification | 5 | 4 | 5 | 5 | 5 | 4 | 142 | shaft-automated-test-authoring | Diff, guardrail, compile, and verdict triggers distinguish verification from authoring. |
| shaft-cli | 5 | 4 | 5 | 4 | 5 | 4 | 152 | shaft-mcp | One-shot machine-readable commands form the tested boundary against stateful MCP use. |
| shaft-database-testing | 5 | 4 | 5 | 4 | 5 | 4 | 150 | shaft-automated-test-authoring | Persisted-state and query terms route database work without claiming general authoring. |
| shaft-defect-reporting | 5 | 4 | 5 | 4 | 4 | 2 | 157 | shaft-execution-reporting | Confirmed product finding language is distinct, but the listing description is oversized. |
| shaft-developer | 4 | 3 | 5 | 5 | 5 | 3 | 131 | shaft-automated-test-authoring | The router intentionally overlaps every specialist and must select exactly one owner. |
| shaft-docs-hygiene | 5 | 4 | 5 | 5 | 3 | 4 | 130 | shaft-test-completion; shaft-stakeholder-reporting | SoT trio, three-status vocabulary, and hours math are docs process, not cycle closure reports. |
| shaft-execution-reporting | 5 | 4 | 5 | 4 | 4 | 4 | 157 | shaft-stakeholder-reporting | Engineering counts and evidence distinguish this report from decision-oriented summaries. |
| shaft-failure-analysis | 5 | 4 | 5 | 4 | 5 | 4 | 137 | shaft-flaky-test-analysis | Consistent failure and root-cause ownership are the reviewed boundary against flakiness. |
| shaft-flaky-test-analysis | 5 | 4 | 5 | 4 | 5 | 3 | 147 | shaft-failure-analysis | Equivalent-condition variance and retry history uniquely identify intermittent failures. |
| shaft-fluent-api | 5 | 4 | 5 | 4 | 5 | 5 | 120 | shaft-web-actions; shaft-assertions | Fluent chain composition is narrow and concise despite adjacent web and assertion APIs. |
| shaft-guide-search | 5 | 5 | 5 | 5 | 4 | 5 | 141 | shaft-developer | Official-current guidance and citation requirements create a unique retrieval trigger. |
| shaft-ios-ipa-acceptance | 5 | 4 | 5 | 5 | 5 | 5 | 121 | shaft-browserstack-native; shaft-android-apk-select | iphoneos Info.plist gate before upload is narrower than general BrowserStack config. |
| shaft-locator-design | 5 | 4 | 5 | 4 | 5 | 5 | 150 | shaft-page-objects | DOM and accessibility evidence distinguish locator design from object architecture. |
| shaft-mcp | 5 | 4 | 5 | 4 | 4 | 5 | 149 | shaft-cli | Stateful interactive tool calls are a crisp complement to one-shot CLI behavior. |
| shaft-mobile-actions | 5 | 4 | 5 | 4 | 5 | 4 | 139 | shaft-web-actions | Native contexts, gestures, and app lifecycle terms keep mobile action routing specific. |
| shaft-mobile-env | 5 | 4 | 5 | 5 | 4 | 4 | 123 | shaft-test-environment; shaft-browserstack-native | Local JDK/Appium/SDK/emulator doctor loop is distinct from cloud BrowserStack setup. |
| shaft-mobile-retest-hygiene | 5 | 4 | 5 | 5 | 4 | 4 | 128 | shaft-failure-analysis; shaft-mobile-actions | Map no-op triage, picker probes, and mixed-bytecode journals specialize mobile retest analysis. |
| shaft-nonfunctional-test-design | 4 | 4 | 5 | 4 | 4 | 4 | 150 | shaft-accessibility-testing; shaft-test-planning | Measurable quality attributes define the scope; accessibility has a deeper specialist. |
| shaft-page-objects | 5 | 4 | 5 | 4 | 5 | 5 | 142 | shaft-locator-design | Reusable object and user-intent method ownership separates architecture from locators. |
| shaft-recording-codegen | 5 | 4 | 5 | 4 | 5 | 5 | 140 | shaft-test-recording | Persisted recording input and compilable Java output distinguish transformation from capture. |
| shaft-requirements-analysis | 5 | 4 | 5 | 4 | 4 | 3 | 148 | shaft-test-planning; shaft-test-case-design | Pre-planning testability and gap analysis is clear, though the description needs compression. |
| shaft-stakeholder-reporting | 5 | 4 | 5 | 4 | 4 | 5 | 134 | shaft-execution-reporting | Business decisions and residual risk distinguish this from engineering result detail. |
| shaft-test-case-design | 5 | 4 | 5 | 4 | 4 | 4 | 150 | shaft-test-planning; shaft-requirements-analysis | Concrete scenarios, data, steps, and expected outcomes bound this design specialist. |
| shaft-test-completion | 5 | 4 | 5 | 4 | 3 | 5 | 154 | shaft-test-monitoring; shaft-stakeholder-reporting | Cycle closure (exit criteria, archive, testware decisions, handover) is distinct from live monitoring and business reporting. |
| shaft-test-data-design | 5 | 5 | 5 | 4 | 4 | 4 | 153 | shaft-database-testing | Isolation, seeding, uniqueness, and cleanup identify data design rather than DB checks. |
| shaft-test-environment | 5 | 5 | 5 | 4 | 4 | 5 | 145 | shaft-test-execution | Readiness, parity, versions, and drift route environment work before execution. |
| shaft-test-execution | 5 | 4 | 5 | 4 | 5 | 4 | 155 | shaft-test-monitoring | Running approved scope with preflight and cleanup differs from observing active runs. |
| shaft-test-monitoring | 5 | 4 | 5 | 4 | 4 | 5 | 140 | shaft-test-execution | Progress, gates, health, and completion signals clearly indicate runtime monitoring. |
| shaft-test-planning | 5 | 4 | 5 | 4 | 4 | 4 | 154 | shaft-requirements-analysis; shaft-test-case-design | Scope, resources, estimates, schedule, and criteria distinguish planning from analysis. |
| shaft-test-recording | 5 | 4 | 5 | 4 | 4 | 5 | 144 | shaft-recording-codegen | Approved live-flow capture and privacy-safe evidence separate recording from generation. |
| shaft-web-actions | 5 | 4 | 5 | 4 | 5 | 4 | 162 | shaft-mobile-actions; shaft-fluent-api | Browser navigation and interaction primitives are specific despite fluent API overlap. |
SHAFT skills pack
Humans: this folder is the published pack that teaches an agent to use SHAFT. The generated inventory is INVENTORY.md. The scorecard is QUALITY.md. Evaluation prompts are evaluation-prompts.md.
Agents: for product test work, load shaft-skills/shaft-developer/SKILL.md and follow that router. For work on this repository, load AGENTS.md at the repository root and follow ChaosEngine. This file is the human map and an index. Do not treat it, INVENTORY.md, or QUALITY.md as policy. INVENTORY.md and QUALITY.md are generated; do not edit them by hand.
SHAFT skills mindmap
Human map of the pack. An agent loads shaft-developer and that entrypoint selects exactly one skill. The other skills stay unloaded until that choice.
shaft-developer
├── Lifecycle
│ ├── shaft-requirements-analysis — testable requirements, gaps, traceability
│ ├── shaft-test-planning — strategy, scope, risk, schedule, entry and exit
│ ├── shaft-test-case-design — scenarios, steps, expected results
│ ├── shaft-test-data-design — datasets, factories, privacy, cleanup
│ ├── shaft-test-environment — browser, device, service, configuration readiness
│ ├── shaft-test-execution — run approved tests and keep the evidence
│ ├── shaft-test-monitoring — progress, gates, duration, retries
│ ├── shaft-test-completion — exit criteria, residual risk, archive, handover
│ ├── shaft-failure-analysis — Allure failure analysis of one consistent failure
│ ├── shaft-flaky-test-analysis — inconsistent pass/fail under the same conditions
│ ├── shaft-defect-reporting — a reproducible product defect
│ ├── shaft-execution-reporting — engineering execution report
│ ├── shaft-stakeholder-reporting — business quality or release decision
│ ├── shaft-accessibility-testing — WCAG, keyboard, assistive technology
│ └── shaft-nonfunctional-test-design — performance, security, reliability, recovery
├── Implementation
│ ├── shaft-automated-test-authoring — executable SHAFT test code
│ ├── shaft-page-objects — page and component objects
│ ├── shaft-fluent-api — fluent chains
│ ├── shaft-locator-design — rank and repair locators
│ ├── shaft-web-actions — browser and element actions
│ ├── shaft-mobile-actions — native and touch actions
│ ├── shaft-mobile-env — local JDK, Appium, SDK, emulator
│ ├── shaft-android-apk-select — universal or ABI-split APK
│ ├── shaft-ios-ipa-acceptance — IPA gate before cloud upload
│ ├── shaft-browserstack-native — BrowserStack App Automate
│ ├── shaft-mobile-retest-hygiene — mobile retest map, picker, bytecode, journal
│ ├── shaft-api-testing — HTTP API tests
│ ├── shaft-database-testing — persisted-state checks
│ ├── shaft-assertions — hard assertions and soft verifications
│ ├── shaft-guide-search — current official SHAFT guide
│ ├── shaft-test-recording — Record and playback of a live flow
│ ├── shaft-recording-codegen — Code generation from a persisted recording
│ ├── shaft-change-verification — preview, guardrail, apply, verify
│ └── shaft-docs-hygiene — plan, journey, and issue vocabulary
└── Tool caches (load only when selected)
├── shaft-mcp — compact MCP names and arguments; defer JSON schemas
└── shaft-cli — compact CLI forms and arguments; do not use --help for discovery
How they relate
shaft-developeris the only decision surface. Finish one skill before returning for a different deliverable.- Allure failure analysis is
shaft-failure-analysis. The sibling for results that pass and fail under the same conditions isshaft-flaky-test-analysis. - Record and playback is
shaft-test-recording. Code generation from that persisted recording isshaft-recording-codegen. A generated edit then goes toshaft-change-verification. shaft-automated-test-authoringowns test code that is not produced from a recording. It hands locator, page-object, fluent, web, mobile, API, database, and assertion detail to those specialists.- Local mobile setup starts at
shaft-mobile-env, thenshaft-android-apk-selectorshaft-ios-ipa-acceptance, and usesshaft-browserstack-nativefor a cloud device. shaft-mcpandshaft-cliare the two tool caches. Load one of them only when the entrypoint selects that tool path. Neither cache is part of the ChaosEngine entry.
Next
- Repository orientation is
README.mdat the repository root. - Skill inventory
SHAFT engine users add-on
ChaosEngine add-on for people who use SHAFT in their own test projects.
The install one-liner selects it when the project matches the pack
installWhen, and when the project is the pack home (via
shaft-core-developers). It is not selected for any other project.
- Install:
--with-shaft-engine-users(PowerShell-WithShaftEngineUsers, orCHAOS_ENGINE_ADDONS=shaft-engine-users). - Remove:
--without-shaft-engine-users. - Installed at
.chaos-engine/addons/shaft-engine-users/. - Entry point:
shaft-developer/SKILL.md(repository:shaft-skills/shaft-developer/; installed:.chaos-engine/addons/shaft-engine-users/shaft-developer/SKILL.md). It routes to exactly one SHAFT specialist per task.
The skills are the same files the SHAFT agent plugin publishes, included in
place from shaft-skills/. Nothing is copied, so the two never drift. SHAFT
contributors also want shaft-core-developers, which requires this add-on.
SHAFT project profile
Load the canonical ChaosEngine entrypoint (.chaos-engine/skills/chaos-engine/SKILL.md)
first. This profile adds only repository-specific facts and permissions.
Its machine-readable identity is in profile.json.
Knowledge stores: scripts/agents/knowledge_stores.py (status, search;
MemPalace path via tools/repository-map/resolve_mempalace.py;
Graphify: graphify). Watch PRs with one
blocking scripts/ci/watch_pr_checks.py call per push.
Reflection (.chaos-engine/references/reflection-checkpoints.md) applies unchanged.
- Repository:
ShaftHQ/SHAFT_ENGINE; default branch:main. - Personal authorizations (commit-signing key, owner email attribution,
artifact sharing) live in user-level config, never in this repository:
~/.config/chaos-engine/authorizations.md(Windows%APPDATA%\chaos-engine\authorizations.md). Load it when present; without it, ask before signing as, attributing to, or sharing for the owner. - Task branches use
ChaosEngine/*and start from fetchedorigin/main. - SHAFT and ChaosEngine tasks are complete only when the pull request is merged.
- Agents never manually start, rerun, or replace the
E2E TestsorLocal E2E Testsworkflows. Smallest local proof and exact-head PR checks gate delivery; scheduled nightly workflows own E2E execution. - The companion public-documentation repository is
ShaftHQ/shafthq.github.ioonmaster; discover its local root or use an explicitly configured root, never a fixed sibling path. Every user-facing SHAFT behavior change opens a companion PR on thatmasterbranch in the same delivery. That companion PR must include a description of the change, screenshots where a human sees UI, human-facing instructions, and AI-supported details (locator policy, replay-proven snippets, properties, exact commands). - Install or upgrade (INSTALL.md):
irm https://raw.githubusercontent.com/ShaftHQ/SHAFT_ENGINE/main/chaos-engine/install.ps1 | iex. - Maven modules and SHAFT product behavior route through the playbooks and mastery chapters under references.
- Cheap, bounded, already-specified local coding work uses the workstation local coding agent.
Multi-ticket assignment orchestration
This specializes the portable solo-or-orchestrate rule: two or more SHAFT issues in one owner request are orchestration. Do not wait for the owner to say "orchestrate".
- Load every assigned ticket (and linked/deferred children) before grouping or dispatching.
- Group related work to the fewest PRs that still keep one problem per issue
(
Closes #Nper completed subtask). - Main session orchestrates: status, owner commands, review, merge. It does not implement product or guidance chunks.
- Remaining chunks run one at a time, ordered by dependency then priority.
- After a chunk's PR is merged, destroy that writer and start the next from a
fresh
ChaosEngine/*branch off fetchedorigin/main.
Graphify
This repository profile uses Graphify as a repository CLI/cache workflow for current structural questions: likely files, calls, and dependencies. It does not depend on Graphify appearing as an MCP tool. Absence from the MCP tool catalog is not evidence that Graphify is unavailable.
The CLI route below is the controlling Graphify procedure over conflicting same- or lower-priority guidance. Its ordered steps are exact:
- G1: Resolve any available shared cache and require a nonempty readable graph. A stale cache may provide untrusted positive leads but never completeness.
- G2: If G1 succeeds, run exactly one query bounded to the affected symbol or
subsystem, treat results as untrusted leads, verify every returned path
against the current worktree, and supplement conclusions with targeted
rg. - G3: Attempt the read-only coverage audit against the primary checkout that owns the cache even when G1 or G2 fails. Inability to resolve that owner is a failed audit attempt, never permission to audit a linked worktree.
- G4: Declare degraded mode when any step cannot provide current verified results, and only after G1 through G3 have been attempted; never use MCP catalog absence as the reason.
Run the procedure with this PowerShell flow, replacing only the bounded structural question:
$graphOut = py -3 tools/repository-map/resolve_graph_out.py --check
$resolverOk = $LASTEXITCODE -eq 0 -and -not [string]::IsNullOrWhiteSpace($graphOut)
$sharedGraphOut = if ($resolverOk) { $graphOut } else {
py -3 tools/repository-map/resolve_graph_out.py
}
$cacheOk = -not [string]::IsNullOrWhiteSpace($sharedGraphOut) -and
(Test-Path -LiteralPath (Join-Path $sharedGraphOut "graph.json") -PathType Leaf)
$primaryRoot = if ([string]::IsNullOrWhiteSpace($sharedGraphOut)) { $null } else {
Split-Path $sharedGraphOut -Parent
}
$queryOk = $false
if ($cacheOk) {
$queryOutput = @(graphify query "<bounded structural question>" --graph (Join-Path $sharedGraphOut "graph.json"))
$queryExitOk = $LASTEXITCODE -eq 0
$queryOutput | Write-Output
$returnedPaths = @($queryOutput | ForEach-Object {
if ($_ -match 'src=(.+?)\s+loc=') { $Matches[1] }
} | Sort-Object -Unique)
$worktreeRoot = [IO.Path]::GetFullPath((Get-Location).Path).TrimEnd(
[IO.Path]::DirectorySeparatorChar
) + [IO.Path]::DirectorySeparatorChar
$invalidPaths = @($returnedPaths | Where-Object {
$relative = $_
$parts = @($relative -split '[\\/]')
$lexicallyInside = -not [IO.Path]::IsPathRooted($relative) -and
$parts.Count -gt 0 -and -not ($parts | Where-Object { $_ -in @('', '.', '..') })
$resolved = if ($lexicallyInside) {
Resolve-Path -LiteralPath (Join-Path (Get-Location) $relative) -ErrorAction SilentlyContinue
} else { $null }
$inside = $null -ne $resolved -and ($resolved.ProviderPath + [IO.Path]::DirectorySeparatorChar).StartsWith(
$worktreeRoot, [StringComparison]::OrdinalIgnoreCase
)
$hasReparsePoint = $false
$lexicalPath = (Get-Location).Path
foreach ($part in $parts) {
$lexicalPath = Join-Path $lexicalPath $part
$item = Get-Item -LiteralPath $lexicalPath -Force -ErrorAction SilentlyContinue
$hasReparsePoint = $hasReparsePoint -or (
$null -ne $item -and ($item.Attributes.value__ -band 1024) -ne 0
)
}
-not $inside -or $hasReparsePoint
})
$queryOk = $queryExitOk -and $invalidPaths.Count -eq 0
}
$auditOk = $false
if ($null -ne $primaryRoot) {
py -3 tools/repository-map/graphify_maintenance.py audit --root $primaryRoot
$auditOk = $LASTEXITCODE -eq 0
}
if (-not ($resolverOk -and $cacheOk -and $queryOk -and $auditOk)) {
Write-Warning "Graphify degraded mode: use targeted live-file verification."
}
A missing cache reports absent; a cache without a matching indexed revision
reports stale. Either state is advisory for ordinary tasks. A readable stale
cache may be queried once for positive leads, but never establishes completeness,
“no callers,” or another negative conclusion. After the ordered route, use
targeted rg plus live files for every blast-radius conclusion. Never rebuild
or record the cache from an ordinary task, and never commit graphify-out/.
The existing maintenance controller is the sole Graphify refresh owner. A
shared origin/main revision mismatch or an active refresh lock is an expected
degraded condition and does not block implementation. An ordinary task must not
refresh, wait or retry-loop, clear or replace the lock, freshness marker, or
cache, or switch, reset, or overwrite the primary checkout to satisfy freshness.
Continue with live files and targeted rg; only the maintenance owner updates
the shared cache.
From the primary checkout, the repository-owned controller owns refresh:
py -3 tools/repository-map/graphify_maintenance.py refresh --root .
The refresh uses the pinned Graphify runtime, then orders build -> audit -> cluster -> freshness marker. JSON sources with no emitted nodes remain visible expected data-only inputs; SQL or other parser gaps stop the refresh before the marker is recorded.
Unclassified extract skips are corpus policy, not a failed install. Noise
families belong in .graphifyignore and mempalace.yaml exclude_patterns.
Known first-class SHAFT families that Graphifyy 0.9.43 still drops
(.properties, META-INF/services/*, Dockerfiles, plugin XML, .toml,
.feature, wrappers) stay on the Graphify allowlist until an upstream
promote lands. Nightly MemPalace mine force-includes the exact
git ls-files set for .properties, SPI, Dockerfiles, plugin XML, and
.feature via --include-ignored (#5119). A new suffix is unclassified_unexpected and fails
graphify_maintenance.py audit. knowledge_stores.py refresh delegates to
python3 .chaos-engine/tool.py stores refresh and is allowed from any
worktree. status/search remain the ordinary-task path. See #5087.
Freshness behavior is pinned by tests/scripts/test_resolve_graph_out.py.
Agent Guidance Boundary Guard
Use for agent guidance and retrieval/config surfaces. Keep one substantive entrypoint; make every host surface a thin relative adapter.
Workflow
- Put cross-provider behavior in chaosengine-harness; link new task playbooks from its routing reference instead of copying them.
- Keep
AGENTS.md,CLAUDE.md, and installer-generated Claude skill/agent adapters lean. They may point to canonical files with relative paths but must not restate policy. - Keep one hook implementation under
scripts/agents/; provider hook files only register its relative command. - Preserve restricted native Memory access, deterministic MemPalace retrieval, and Graphify routing. Do not save routine diaries.
- Update guidance budgets/manifests and add a failing portability test before changing behavior.
- Assemble the portable
shaft-skillspackage withpython3 scripts/ci/assemble_shaft_skills_plugin.py <empty-output-directory>and validate it withscripts/ci/validate_agent_plugins.py; cover assembly intests/scripts/test_assemble_shaft_skills_plugin.py, and release assets intests/scripts/test_agent_plugin_release.pybefore release. Forshaft-skills, refresh and validate the generated inventory, reviewed scorecard, and context budget withpython3 scripts/ci/shaft_skill_quality.py --write, then runtests/scripts/test_shaft_skill_quality.py; refresh the canonical routing corpus adapters withpython3 scripts/ci/shaft_skill_routing_eval.py --writeand cover them intests/scripts/test_shaft_skill_routing_eval.py. Validate every online candidate throughpython3 scripts/ci/shaft_skill_candidate_intake.pyandtests/scripts/test_shaft_skill_candidate_intake.py; never install candidate code into canonical roots or bypass its no-network container gate. Run unauthenticated native-client evidence withpython3 scripts/ci/agent_plugin_client_smoke.py --mode smokeand cover it intests/scripts/test_agent_plugin_client_smoke.py; scheduled live acceptance in.github/workflows/agent-plugin-acceptance.ymlrecords missing model credentials as external blockers. - Run
python3 scripts/ci/validate_agent_setup.py --skip-external; run the full command when network/npm access is stable.
Output
List canonical guidance, adapter, hook, retrieval, and validation changes plus intentional budget slack.
PDCA
Personas are phases, not agent identities. They run sequentially in the main
session. PDCA does not select an execution mode. Select one through
execution workflows (.chaos-engine/references/execution-workflows.md), then keep
these phases sequential inside it.
- Kevin phase plans spec, value, acceptance, risks, and any useful Mermaid or wireframe.
- Bob phase makes the smallest cross-platform change through observed TDD, or shepherds a bounded default-capability owner that does when orchestrating.
- Bruce reviews the actual diff and evidence for defects, ambiguity, and confidence, then assigns any required patch to an implementation owner.
Do not merge phases. Run Kevin -> Bob -> Bruce. Bruce covers quality/simplicity, then intuitiveness/acceptability, inside Check. Stop when Check accepts. Do not add a pass to fill a quota.
Execution
Switch persona by switching main-thread phase, never by creating persona agents, workflows, or orchestrators. Bob takes at most three implementation rounds. Bruce judges the actual diff plus real checks, never a self-report; hunt stubs and weakened assertions. Gaps are closed by whoever the mode says implements. Record which phase produced each commit.
Use the canonical TDD/PDCA phase boundary (.chaos-engine/references/tdd.md#workflow).
The harness close is the delivery cycle (references/delivery-cycle.md in the harness):
at most 3 PDCA iterations, then two independent adversarial reviews. Bruce's
in-session check is Check. It is not one of those two reviews, and the author
is neither reviewer. Stop when Check accepts. The cap is not a quota.
Allure Extent Report Operator
Use for report generation and CI artifact behavior. Do not treat a generated summary as valid until raw result files are populated.
Workflow
- Separate producers from consumers:
AllureManager, report properties/paths,.github/actions/post-test-report,.github/actions/consolidate-test-results, workflows, andscripts/ci/extract_allure_failures.py. - Preserve the live
allure-resultsdirectory and delete only its contents; replacing the root can race with Allure writers on Windows. - Count
*-result.jsonfiles before trusting Allure status orsummary.json. - For parser changes, run
python3 -m pytest tests/scripts/test_extract_allure_failures.py. For Java report internals, run the narrow affected Allure manager tests. - Treat Extent as a separate path: verify actual Extent artifacts before claiming support.
Output
Report producer/consumer impact, raw result count evidence, parser or Java tests run, artifact names, and remaining OS/path risk.
CI Failure Investigator
Use this playbook only for CI failure triage.
Workflow
- Identify the exact run, failed jobs, changed commit, and failed steps. Use local logs supplied by the user before making GitHub calls.
- If remote data is required, normalize authentication with
scripts/ci/github-auth-env.sh, then fetch failed-job logs and relevant artifact metadata only. Do not download successful-job logs by default. A job-log download is not a MemPalace or Graphify retrieve.gh api.../actions/jobs/<id>/logs,gh run view --log-failed, and curl of that logs URL may be saved under a scratch directory and filtered withsed/tailwithout a store citation. - Search logs for the first actionable exception and its surrounding setup or teardown lines. Separate primary failure from reporting and cleanup noise.
- For SHAFT artifacts, follow the CI investigation runbook. Parse self-contained Allure HTML or result JSON instead of opening large reports manually.
- Count result files before trusting statuses. Inspect failed, broken, and retried/skipped attempts when the final summary hides an intermittent failure.
- Classify the cause as code, test isolation, configuration, dependency, runner/infrastructure, credentials, or external provider. Do not weaken assertions to hide provider or credential failures.
- Inspect only the source and recent changes connected to the failure signature. Propose or implement the smallest root-cause fix.
- Validate with the narrowest local reproduction. Request a remote rerun only when local evidence cannot prove the environment-specific behavior.
Known SHAFT fingerprints
setup-lock-pin-drift: a bundled setuppackage-lock.jsonchanged but the planner SHA-256 pin did not, soUnit Tests (shaft-infrastructure)fails with "lock does not match the approved plan". Keep the pin. Same-repo Dependabot pull requests self-heal throughsetup-lock-pins.yml; otherwise runpython3 scripts/ci/refresh_setup_lock_pins.py --writein the same tip.
Output
Report the run/job, concise signature, root cause and confidence, affected files, validation evidence, and remaining environment risk. Avoid templated tables when a short finding is clearer.
Flaky Test Stabilizer
Use this playbook for intermittent, retry-only, parallel-only, or environment-sensitive failures.
Workflow
- Gather the exact failure, runner mode, retry history, affected test, and setup/teardown code. Do not begin with broad refactoring.
- Check high-probability causes in this order:
- per-method state stored in shared instance or static fields;
ThreadLocalvalues not removed;- one method deleting another method's files;
- global SHAFT properties or static services shared across classes;
- fixed sleeps or missing readiness polling;
- unordered/random assertions or external-service dependencies;
- browser sessions created accidentally by unit-test constructors.
- Remember that TestNG
singleThreadedisolates methods in one class, not static state shared with other classes. Use a cross-class lock where a static service genuinely cannot run concurrently. - Preserve the
allure-resultsroot during cleanup. Inspect retried/skipped result JSON when the final report is green. - Fix the ownership or synchronization defect without changing what the test
validates. Prefer local variables,
ThreadLocal, explicit locks, readiness polling, deterministic fixtures, and mocked external dependencies. - Run the focused test once. Repeat it or enable method parallelism only when repetition directly tests the suspected flake mechanism; stop after enough evidence establishes stability.
Output
State the failure signature, root cause and confidence, minimal fix, focused commands run, repetition/parallel evidence if applicable, and residual risk.
Framework Source Rules
- Deprecate before removing or renaming public APIs; prefer overloads to changed signatures.
- Add concise JavaDoc (
@param/@return/@throwswhere applicable) to public classes and methods. - Overload
@paramarity (before push): when adding or splitting overloads, strip or retarget every@paramtag to the actual signature before push. A leftover@paramfrom a longer overload fails ErrorProne and cascades CI (#5748). Runscripts/ci/check_javadoc_param_arity.py(repo-only) againstshaft-engineinteraction packages (or the files you touched). The harness PR Gatejavadocsurface also runs the checker’s unit tests when interaction sources or the script change. - Keep user-facing APIs consistent with the
SHAFTfacade and its namespaced nested classes. Utility classes use a private constructor that throwsIllegalStateException("Utility class"). - Throw specific exceptions, retain causes, and use SHAFT reporting/logging utilities. Never print to standard output or swallow failures silently.
- Declare configurable versions/URLs/timeouts/thresholds/switches in the
matching
com.shaft.properties.internalinterface with a default; access viaSHAFT.Properties, neverSystem.getProperty()directly. - Use
ThreadLocalPropertiesManagerfor per-thread overrides; clear withProperties.clearForCurrentThread()at lifecycle boundaries. Document non-obvious shared/thread-local state ownership. - New external tools stay batteries-included: resolve from PATH, then a package manager where applicable, then a controlled Maven-cache download; keep the version in the properties layer.
- Merge existing vendor capability maps and keep all WebDriver extensions
namespaced. Never restore top-level legacy capabilities such as
enableVideo. - For Allure 3, use
AllureManager.resolveAllureCommandPrefix(). Do not add the removedallure generate --cleanflag; the generated summary is at the report root. - Preserve UTF-8 flags in Maven and CI configuration when touching process or reporting setup.
- For a bug, reproduce the defect (add a focused regression test when practical); make the smallest behavior-preserving fix.
- Route release metadata work through the release guard.
Design / Automation MCP apply paths (#6012)
Every Design/Automation tool that paints IDE state needs:
- A dedicated
apply*Json(or typed apply) on the panel/model. - A router discriminator on a stable field (or explicit tool name) in
DesignStagePanel.applyToolResult— never reuseapplyAnalysisJsonfor a different JSON shape. - A panel unit test that applies a fixture and asserts the UI model updates.
Current discriminators include shaftType/shaftMethod → gap map,
handoffAllowed → readiness, automationPrefill → handoff.
See also intellij-ui-test-conventions.md for accessibleName (#6013) and Live record persistence (#6014).
IntelliJ Plugin Development
Use for shaft-intellij plugin, tool-window, Gradle or JDK, or Marketplace work.
Load the IntelliJ plugin mastery chapter instead of copying it. Visible UI also follows SHAFT UI design. Contributor notes: UI test conventions (accessibleName, Live record persistence) and Design/Automation apply paths.
Cite current JetBrains sources, not memory:
- Plugin SDK: https://plugins.jetbrains.com/docs/intellij/developing-plugins.html
- Marketplace approval: https://plugins.jetbrains.com/docs/marketplace/jetbrains-marketplace-approval-guidelines.html
- Marketplace security: https://plugins.jetbrains.com/docs/marketplace/understanding-plugin-security.html
Workflow
- Load mastery before editing plugin source, tests, Gradle, or Swing UI.
- Keep the plugin a light wrapper. The same generate-from-scenario path must
work from the plugin and from a CLI. Pass the user text to the selected
client; do not drive
capture_start, replay, or generate from the plugin. - Marketplace setup is copy the installer command and pre-type the IDE terminal only. Never execute the installer from the plugin with ProcessBuilder, Runtime.exec, or any equivalent.
- Visible UI changes regenerate screenshots with
ShaftPluginScreenshotRendererTestonly, passing-Dshaft.intellij.screenshotDir=...and-Dallure.automaticallyOpen=false. Never a bare full-suitegradlew test. - Run the smallest focused plugin tests for the edited behavior. Headless only.
Output
List plugin files touched, checks run, screenshot paths for visible UI, and confirm plugin source has no installer-exec path.
IntelliJ UI test conventions
Icon-only controls (#6013)
ShaftIconButtons and other icon-only toolbar controls expose their contract through
accessible name, not visible text. Prefer:
findByAccessibleName(panel, "Start recording", JButton.class);
assertEquals("Start recording", button.getAccessibleContext().getAccessibleName());
Do not assert getText() on icon-only Automation/Design toolbar buttons — text is
empty or decorative and flakes across LAFs.
Automation workflow persistence key (#6014)
The Automation canvas persistence key is Live record
(AutomationStagePanel.LIVE_RECORD_TAB). A legacy Guided key restores once to
Live record and must not be written again.
Java Test Rules
- Follow the existing TestNG, JUnit, or Cucumber style in the affected module. Use descriptive scenario-based test names.
org.testng.Assert.assertThrowsreturns void andexpectThrowsreturns the throwable. UseassertThrowsonly as a statement when the test needs the type. UseexpectThrowswhen the test needs the throwable. Do not assignassertThrows.- Browser tests create a fresh driver per test and clean it in an always-run
teardown. For
ThreadLocaldrivers, call bothquit()andremove(). - Browser tests run headlessly; headed-only behavior needs explicit user approval before execution (see AGENTS.md).
- Under TestNG method parallelism, instance fields are shared across
concurrent methods; keep per-method paths/files/mocks/drivers/mutable setup
in locals or
ThreadLocal, and clear the same thread-owned state in cleanup. @Test(singleThreaded = true)serializes one class only; static services or SHAFT global properties shared across classes need an explicit cross-class lock or per-test prerequisites, plus always-run restoration.- Use
Properties.clearForCurrentThread()for per-thread SHAFT properties, except JVM-globalSHAFT.Properties.flags— capture/restore the original infinally/@AfterMethod. - When a test writes Allure results, follow
the reports playbook; it owns the
allure-resultshandling and verdict rules. - Prefer SHAFT fluent assertions and existing test-data conventions. Store reusable data in module test resources instead of embedding large fixtures.
- Coverage-only unit tests must not call facade/default constructors that create real browser sessions. Use explicit mock-backed constructors; keep session creation in integration tests.
- Avoid mocking or proxying
org.testng.ISuiteon Java 25 when list/value helper logic can be tested directly. - Mock external services when the contract is not under test. Disable status validation only when response status is intentionally outside the assertion.
- For a bug fix, prove the regression first when practical, then run the focused test after the fix; repeat/parallelize runs only to evaluate a flaky or concurrency failure.
- Reuse passing test evidence unless later edits affect the tested behavior or its dependencies.
- Capture screenshots only when the tested behavior is visual.
- For flaky failures, use the stabilization playbook.
Static analysis and proof runs
- Strategy-matrix / Wave PRs fail static analysis when Java switches lack a
default branch:
switchmust be exhaustive (defaultwhere CodacyMissingDefaultCaserequires it). Confirm before the first push. - Codacy is this repository's static-analysis app: its
ACTION_REQUIREDis the static-analysis gate (.chaos-engine/references/static-analysis-gate.md). - N-run / flake-proof scripts that invoke Maven under the engine Surefire
profile must not treat process exit alone as green: after each proof
invocation, require a zero failed count from Surefire
TEST-*.xmlor TestNGtestng-results.xml(and set-Dmaven.test.failure.ignore=falsefor defense in depth). Usescripts/ci/assert_surefire_green.py(repo-only).
MCP Transport Contract Auditor
Use for shaft-mcp contract changes across tool schemas, fixtures, startup, installers, containers, and transports.
Workflow
- Pair Java/service edits with
shaft-mcp/src/test/resources/fixtures/mcp-tool-manifest.jsonandshaft-mcp/src/test/resources/fixtures/shaft-pilot/mcp/**when the public tool contract changes. - Run
python3 scripts/ci/validate_shaft_mcp_configuration.py. Also runpython3 scripts/ci/validate_modular_documentation.pyfor client fixture or public docs-link changes. - For Java behavior, run the narrow affected
shaft-mcptests with-DheadlessExecution=trueand-Dgpg.skip. - Before claiming transport safety, package
shaft-mcp, copy runtime dependencies toshaft-mcp/target/dependency, then runpython3 scripts/ci/validate_shaft_mcp_transports.py. - For installer or release-facing changes, run
python3 scripts/ci/verify_shaft_mcp_installer_release.pyor state why it is not applicable.
Output
List contract files touched, transport/client impact, checks run, and remaining fixture or installer risk.
Modular Boundary Auditor
Use for module boundary and packaging drift across Maven reactor modules, examples, BOM, legacy coordinate, and clean consumer fixtures.
Workflow
- Inspect root/module POMs,
shaft-bom,legacy-shaft-engine, bundled examples, andtools/modularization/consumer-fixtures/**before changing checks. - Keep optional module dependencies out of
shaft-engine; fix the owning module, BOM, or fixture instead. - Run
python3 scripts/ci/validate_reactor_versions.pywhen versions or inter-module coordinates change. - Run
python3 scripts/ci/validate_modular_documentation.py,python3 scripts/ci/validate_maven_publication.py, and the relevanttests/scripts/test_*_boundary.py. - For release-candidate surfaces, add
python3 scripts/ci/validate_shaft_pilot_release.pyand affected clean-fixturemvn --batch-mode -f ... verify "-Dshaft.version=..."checks.
Output
State the boundary touched, owning module, BOM/fixture/example impact, checks run, and any untested fixture matrix.
Public Behavior Docs Synchronizer
Use for user-visible SHAFT behavior changes. Keep docs work separate from code unless the task explicitly asks for both.
Every user-facing SHAFT behavior change opens a companion PR on
ShaftHQ/shafthq.github.io master in the same delivery: discover the docs
root, or use an explicitly configured root; never a fixed sibling path. Do
not add public guide pages to SHAFT_ENGINE. That companion PR must include a
description of the change, screenshots where a human sees UI, human-facing instructions,
and AI-supported details (locator policy: unique author-written id via the SHAFT
locator builder, then ARIA role, then native relative xpath only; replay-proven
snippets; properties; exact commands).
Required companion PR content
- A description of the change.
- Screenshots for visible UI. Reuse existing
static/or test screenshots when still accurate. If a screenshot cannot be produced headless, ship the prose and open one standalone follow-up issue per missing shot. Do not launch a GUI browser without asking. - Human-facing steps.
- AI-supported details: locator policy, replay-proven snippets, properties, and exact commands.
Workflow
- Identify the public surface: API, property, module, report output, CLI/MCP tool, README link, or release text.
- Discover the local
shafthq.github.iocheckout, or use an explicitly configured root; never a fixed sibling path. Search it with targetedrg; update only guide pages affected by the behavior change. - Keep canonical links in
README.md,.github/RELEASE_BODY_TEMPLATE.md, andlegacy-shaft-engine/pom.xml. Runpython3 scripts/ci/validate_modular_documentation.pywhen those links or examples are touched. - Run
python3 scripts/ci/validate_documentation_boundaries.py; for guidance changes also runpython3 scripts/ci/validate_agent_setup.py. - Open the companion PR on
masterwith the required content above. Report the docs PR/link, or the concrete reason no public docs change is needed.
Output
List the public surface, guide files searched or changed, validation results, and docs PR/blocker.
Release And Dependency Guard
Use only for release preparation, release review, release notes, or dependency-currency work. Never run deployment or publication commands locally.
Workflow
-
Inspect recent merged release PRs and executable workflows before relying on prose. Compute versions as
{major}.{quarter}.{YYYYMMDD}. -
Keep the root project version, every reactor child parent, and SHAFT inter-module dependency aligned. Run:
python3 scripts/ci/validate_reactor_versions.py -
Update
shaft-engine/src/main/java/com/shaft/properties/internal/Internal.javashaftEngineVersion. Verifyallure3Versionagainst the stable npm package andnodeLtsVersionagainst the current LTS patch using primary sources. Verify every other version default inInternal.javaagainst its source registry when it changes. Keep non-Maven static versions inInternal.java; move stray version literals there. -
Update all example and consumer-fixture
<shaft.version>values found by:rg -l "<shaft.version>" -g "pom.xml" -
Check Maven dependency/plugin/property updates. Accept stable ones after compatibility review; do not treat alpha, beta, RC, milestone, or snapshot releases as upgrades.
-
Validate release metadata and publication rules with existing scripts, including Maven Central POM/JAR/classifier/signature, the IntelliJ IDEA plugin release candidate, and canonical, combined-module, and legacy-relocation consumer checks.
-
Run one compile/package pass after metadata is aligned. Do not rerun broad product tests for metadata-only changes unless source/test behavior changed.
-
Verify the release version is newer than any immutable Maven Central coordinate it replaces. Merging to
maintriggers publication; local deploy, signing, andscm-publishare prohibited.
Release notes (#6232)
Release bodies come from scripts/ci/render_release_notes.py (run by
announce_release) filling .github/RELEASE_BODY_TEMPLATE.md; do not switch
back to raw GitHub-generated notes or hand-write bodies.
- Format: version, a 1-3 line summary, the Maven/Gradle snippet, then only
non-empty sections in this order: Breaking changes and upgrade notes, New
features, Fixes, Performance, Deprecations, notable Dependency upgrades.
One cleaned
- Title (#PR)line per change; no authors or PR URLs. All internal work goes in one<details>block; the full changelog link is last. - Internal:
[CE];ci,test,docs,chore,build, orrefactorprefixes; ChaosEngine/installer/harness scopes; the labelssubsystem:agent-harness,subsystem:repository-tooling,documentation,tests,github-actions, ormaintenance; or no files under shipped module sources. Bumps collapse to "N dependency updates" unless labeledsecurityor they are Selenium/Appium/Playwright majors. - Labels: exactly one of
breaking-change,enhancement,bug,skip-release-notes; optionalperformance,deprecation,regression. PR titles become release lines, so write them for SHAFT users. - Keep the renderer, template, and
scripts/ci/validate_release_notes.pylabel sets in sync;Release-note governancetests that parity. - Slack and reconcile announcements reuse that summary line plus a link via
slack_payload(#6241); never promise sections the notes do not have. - Never edit a published release body. Preview with
python3 scripts/ci/render_release_notes.py --version <tag> --previous-tag <prev> --head <tag> --output <file>.
Output
List blockers first, then version alignment, stable updates considered, commands and results, source checks for volatile versions, and remaining publication risk.
SHAFT Marketing Ad Producer
Plan SHAFT ads without fake claims.
When the optional design-skills add-on is installed
(.chaos-engine/addons/design-skills/SKILL.md), follow its technical-video
pipeline and cards (brief, capture, voice-over, captions, loudness, delivery QC)
with this playbook's SHAFT facts and guardrails; run its design_qc.py gates.
Without it, use the workflow below.
Stack
- Strategy/scenes: plan in the brief/storyboard; no dedicated tool.
- Visuals: keep exact copy, code, logos, and UI screenshots deterministic; no image-generation tool, so write a visual brief.
- Capture: shaft-mcp browser/capture tools for web (screenshots,
capture_start);browser_network_requestsfor perf/network; desktop/OBS/ffmpegif installed. - SHAFT proof: use MCP recorder, report, Doctor, Allure, and code-block tools.
- Editing/audio: use existing video tools and approved or licensed audio; otherwise write a music brief.
Workflow
- Brief: audience, channel, duration, feature, proof, aspect ratio, and planning-only vs production.
- Evidence: read SHAFT README/docs or MCP guidance. Use real Allure, Capture, MCP recorder, Doctor, or code surfaces for production; label mockups.
- Script/scenes: make a timestamped table with visual/action, text, voiceover, source, animation, music beat, and asset path.
- Assets: render code snippets as deterministic HTML/CSS screenshots. Never use generated-image text for final copy or code.
- Capture: avoid secrets and private browser data. Record approved tabs/windows, preferably demo projects or sanitized reports.
- Edit: assemble clips, captions, overlays, snippets, music, and exports with the smallest existing toolchain. Save commands/projects.
- Reuse: save reusable output in the marketing repo, not SHAFT_ENGINE. If no repo path is given, ask first. Use
brand/theme.json,brand/assets/,snippets/,music/, andcampaigns/<yyyy-mm>-<slug>/{brief.md,script.md,storyboard.md,captures/,renders/,exports/,manifest.json}.
Output
Planning: campaign slate, storyboard table, asset checklist, capture plan, and theme notes.
Production: scripts/storyboards, captures, code-card renders, captions, music source/brief, edit manifest, exports, and marketing-repo paths.
Guardrails: no unsupported benchmarks, 3rd-party logos, security claims, generated final code text, secrets, or tokens.
SHAFT UI Design
Use for every visible SHAFT interface: reports, core tools, the public guide,
and app-like product UI. Read this file fully before planning, editing,
reviewing, or validating UI work. When the optional design-skills add-on is
installed (.chaos-engine/addons/design-skills/SKILL.md), its visual-direction,
typography, color, image, and audit cards supply the generic rules; this file
adds the SHAFT specifics and wins on conflict.
Sources Of Truth
- Current repo rules win:
AGENTS.md,CLAUDE.md, matching skills, Memory, and the target repo'sAGENTS.md. - Read existing tokens, CSS, theme files, components, and representative pages before designing. Reuse what works.
- For guide work, preserve Docusaurus behavior, dark mode, reduced motion, semantic HTML, copyable fenced code, search exclusions, archive
noindex, redirects, AutoBot HTTP contracts, and source-backed claims. - Do not add dependencies, systems, or generated artifacts unless the current task needs them.
Workflow
- Define the surface: core report/tool/workbench, docs/user guide, homepage/marketing, or app-like product UI.
- Inspect the current design language and one representative rendered state. For revamps, capture a before screenshot first.
- Make a short design plan: audience, one job, 4-6 color roles, type roles, layout idea, and one memorable SHAFT-specific signature element.
- Build the smallest complete change. Prefer existing components, tokens, CSS, and native browser features over new abstractions.
- Validate in a browser at desktop and mobile widths. Check contrast, focus, keyboard use, reduced motion, dark mode when supported, loading/empty/error states, and text overflow.
- Share real screenshots every time: before/after for revamps, after-only for new UI. PRs that touch visible UI or reporting must include those links/paths in the PR body or a PR comment. If capture is blocked, keep the PR draft or report the exact blocker.
Design Direction
- Make the interface feel like SHAFT: precise, evidence-first, automation-native, confident, and readable under real debugging pressure.
- Ground visual choices in the subject: test evidence, reports, traces, locators, flows, timelines, browser/device surfaces, or docs learning paths.
- Use one bold idea, not many decorations. A signature visualization or interaction should explain the product better than a generic hero/card pattern.
- For operational/report UI, prioritize dense scanning, stable layout, clear status, comparison, filtering, and repeat use. For guide/marketing pages, let the brand voice carry more personality while keeping claims verifiable.
- UI copy is part of design: plain verbs, sentence case, consistent action names, specific errors, useful empty states, no vague apology copy, no unsupported superlatives.
Quality Floor
- Contrast: body and placeholder text at least 4.5:1; large text at least 3:1.
- Layout: no incoherent overlap, no clipping, no text overflow, no layout shift from hover or dynamic content, stable dimensions for grids/boards/toolbars/cards.
- Typography: readable line lengths, clear hierarchy, no viewport-scaled font sizes, no negative letter spacing, headings balanced when supported.
- Components: icon buttons use existing icon libraries when available; familiar controls beat custom widgets.
- Motion: purposeful and sparse; never gate visibility on animations; avoid layout-property animation; always respect
prefers-reduced-motion. - Assets: prefer real product/report/docs imagery or deterministic screenshots. Do not use generated images for exact UI text, code, logos, or claims.
- Color: never pure
#000000or#fffffffor surfaces or body text; keep one accent role and let neutrals carry the rest. Accent saturation above ~80% reads as a default theme, not a choice. - States: every interactive surface ships focus, loading, empty, and error states. A loading state mirrors the real layout's dimensions rather than a centered spinner, so the page does not jump when content arrives.
Unslop Rules
Reject and rewrite these unless the brief explicitly requires them and the choice is justified:
- Generic SaaS hero metrics, endless equal card grids, decorative glassmorphism, gradient text, bokeh/orbs, diagonal stripe backgrounds, sketchy SVG scenes, side-stripe accents, and dark blurred stock imagery.
- Tiny uppercase eyebrows or
01 / 02 / 03markers as section scaffolding. Numbers only when order matters. - One-note palettes: default cream/sand, purple-blue gradients, dark navy/slate, brown/espresso, or a single hue ramp. Existing SHAFT brand tokens can stay.
- Ghost cards: decorative
1pxborder plus wide soft shadow. Pick a border or a restrained shadow. - Over-rounded cards/sections/inputs. Keep cards at 8px radius or less unless an existing design system already requires otherwise.
- Hover transforms on images that are not action targets. Animate the card state, border, shadow, or content affordance instead.
- Text in visible UI that explains the feature instead of letting the feature work.
SHAFT-Specific Checks
- Core generated reports and workbenches must keep evidence readable: status chips, tables, wrapping, trace/code panels, screenshots, and diagnostics must remain inspectable.
- User guide pages must keep one canonical page per concept and verify commands, APIs, properties, dependencies, and compatibility against engine source.
- Public docs changes that alter behavior need the matching docs PR; pure agent-guidance changes do not.
- PR evidence must include the smallest useful validation plus real screenshot links/paths for any UI or reporting enhancement.
IntelliJ Plugin UI
- Follow JetBrains IntelliJ icon standards first: https://intellij-icons.jetbrains.design/ and https://plugins.jetbrains.com/docs/intellij/icons.html.
- Reuse IntelliJ Platform icons from
AllIconswhen they render correctly in runtime and screenshot evidence. Use plugin-owned SVGs only when platform icons do not satisfy the UI, licensing, theme, or deterministic screenshot need. - Keep action icons as 16x16 SVGs under
shaft-intellij/src/main/resources/icons/actions, with matching_dark.svgvariants when needed. Use the IntelliJ New UI action colors: light#6C707E, dark#CED0D6. - Tool window icons must keep the existing 16x16 and 20x20 SVG light/dark variants registered in
plugin.xml. - Do not hand-draw replacement glyphs in screenshot renderers or paint fake icons over components. If screenshots show blank or broken icons, fix the shared icon source (
ShaftIconsor assets) so runtime UI and screenshots use the same real icon. - Preserve icon-only button shape, size, tooltip, and accessible-name behavior through
ShaftIconButtons; do not add one-off button styling in individual panels. - Validate IntelliJ icon/UI changes with the smallest relevant set:
ShaftIconAssetsTest, focused panel tests that assert icons paint visible pixels, screenshot renderer output, and visual inspection of light, dark, and narrow states. - Swing surfaces obey the same states and color rules as the web surfaces above: focus, loading, empty, and error states are required; one accent role; never pure black or white. The plugin-screenshot renderer is where you verify them, not a browser.
- Keep icon stroke weight and metaphor consistent across a panel. Avoid cliche metaphors (rocket for run, shield for security) when a platform
AllIconsentry already names the concept.
Portions of the color/state rules above are re-expressed from leonxlnx/taste-skill (MIT, Copyright (c) 2026 Leonxlnx).
Workstation local coding agent
Use the already-installed workstation loop when the work is cheap, bounded, and already specified. The default capability's cost and latency are waste for those slices. The most-intelligent or default model stays the decider and may hand the slice to this loop.
Orchestration
The host session orchestrates.
One implementer per batch.
The writer stops when the PR exists.
The inner shaft-java-agent.ps1 loop is optional and allowlisted. The local
loop still never merges, reviews, or owns GitHub delivery.
Owner-gated train
Owner-gated train, export, and deploy stay out of tree only. Never auto-train from harvest. Never overwrite qwen2.5-coder:7b. Deploy a new Ollama name.
When to use it
- Read-only
shaft-architectwith the cited-path gate. - Mechanical script-string pins the loop can apply without inventing scope.
- A fully specified edit the loop can run, with an allowlist and a test command, where default-capability time is waste.
When not to use it
- Product changes under
src/main/java. - PR merge, GitHub delivery, or review.
- Harness architecture, routing, or guidance design.
- Anything the loop cannot run: missing allowlist, missing spec, work that needs a public-API judgment, or a host that is not this workstation.
Commands
Keep one model loaded. Loopback only. No startup apps, tray icons, or cloud endpoints.
scripts/local-coding-agent/shaft-java-agent.ps1— bounded edit loop.scripts/local-coding-agent/shaft-architect.ps1— read-only architect; cited-path gate fail-closes invented slashy paths.scripts/local-coding-agent/shaft-local-ai-stop.ps1— stop the loopback runtime.scripts/agents/knowledge_stores.py—statusorsearchfrom any worktree.refreshrefuses.
Output
Return the command that ran, the report path, cited-path or allowlist failures, and whether the decider must take the work back.
Product-track self-improve (SHAFT + ChaosGauge)
Profile extension of the portable
product-track playbook (.chaos-engine/skills/self-improve/references/product-track.md).
SHAFT filing cues
| Lesson class | Prefer |
|---|---|
| Engine / public API regression | Focused shaft-engine (or affected shaft-*) unittest + silent-verify |
| Doctor / CLI repair gap | shaft-doctor / shaft-cli path — not a new MCP essay surface |
| Agent effectiveness on product tasks | Link closest ChaosGauge public task under scripts/ci/chaos_gauge/ |
| IntelliJ plugin | Focused plugin check; keep docs PR separate when docs-only |
Routing
Product authoring still hands off to shaft-developer via
shaft routing. This playbook only covers Learning Session →
issue/eval encoding after delivery.
Proof examples
py -3 -m unittest <focused_module> -v
python3 scripts/ci/chaos_gauge/validate_experiment.py scripts/ci/chaos_gauge/experiment.json
python3 scripts/ci/harness_pr_gate.py --base "$(git merge-base HEAD origin/main)" --head "$(git rev-parse HEAD)" --write-generated
Reject Task Observer; Delivery-Stop / Learning Session only.
Routing
Match the deliverable in front of you to one row, load that surface, and finish its deliverable before returning here. Do not load an adjacent surface preemptively. If no row fits, say what is missing rather than approximating.
Never make the user name an implementation detail they could not reasonably know. Infer the row from what they asked for.
Always, before any row
Take the depth the entrypoint's triage already selected, loading
consult-first (.chaos-engine/references/consult-first.md) only when that triage sends you
there. Apply the entrypoint's Caveman, Ponytail, and TDD rules; use
delegation (.chaos-engine/references/delegation.md) before dispatch and roles (.chaos-engine/references/roles.md) for role
boundaries. When the change involves a mock, a method a production class gains
only for a test, or an argument for writing the code first, add
TDD failure modes (.chaos-engine/references/tdd-failure-modes.md).
Knowledge before discovery
Each store answers a different question. Query the ones whose trigger fires before broad manual discovery, not all of them by reflex.
| Question you actually have | Store | Query it when |
|---|---|---|
| Has this constraint or gotcha already bitten us? | native Memory | A concrete prior constraint or gotcha would change the task decision. |
| What happened around this before, and what does it touch? | MemPalace | The change spans entities, sessions, or a history you were not part of. |
| What calls or depends on this? | Graphify (.chaos-engine/references/graphify.md) | Blast radius is unknown, or you are about to change a shared symbol. |
| What does the code do right now? | targeted rg and exact reads | Always. This is the only source that settles a disagreement. |
From any SHAFT worktree, py -3 scripts/agents/knowledge_stores.py status
or search hits the centralized palace and a Graphify --check. refresh
refuses and names SHAFT-Nightly-Knowledge-Refresh.
Each selected store gets one attempt through the existing host timeout. Record
irrelevant stores as skipped and unavailable stores as degraded; neither blocks
ordinary implementation or completion. Explicit install, upgrade, maintenance,
status, and doctor operations remain strict.
The native store is source-controlled, and its contract travels with it:
.memory/config.json declares the project and store version, and
.memory/schema/object.schema.json, .memory/schema/relation.schema.json,
.memory/schema/event.schema.json, .memory/schema/patch.schema.json and
.memory/schema/config.schema.json validate every write. The entries
themselves are data the CLI owns; reach them with a bounded memory query,
then memory inspect only for selected records. SessionStart injects tracked
locators only and never starts an optional retrieval process.
A retrieved claim is a lead, never a verdict: confirm it against the live file before acting on it, and a stale index never outranks what is on disk. Your own plan ranks here too, below every one of them: it is the oldest source you hold, written before the reads.
If a store is unavailable, name the degraded mode in your report and continue with the rest. Skipping a store whose trigger fired, without saying so, is the failure this table exists to prevent.
Repository engineering
| Deliverable in front of you | Load exactly this |
|---|---|
| Agent guidance, host adapters, hooks, budgets, retrieval setup | agent guidance |
| Explicit PDCA, personas, repeated refinement | PDCA |
Production Java under src/main/java | framework source |
Test Java under src/test/java | Java tests |
| A red CI run, job, or scheduled suite | CI failures |
| Inconsistent pass/fail under equivalent conditions | flaky tests |
| Version, release, release notes, BOM, or dependency metadata | release and dependencies |
| MCP transport, tool contract, or client wiring | MCP transport |
| Module boundary or cross-module dependency | module boundaries |
| Allure or Extent report generation and verdicts | reports |
| Externally documented behavior that changed | public docs |
| Any visible SHAFT interface, visual QA, UX copy, accessibility | UI design |
| Marketing or promotional material | marketing |
| Open or rewrite a work item | work-item skill (.chaos-engine/skills/work-item/SKILL.md) |
| One issue through a merged PR | GitHub playbook (.chaos-engine/references/work-github-playbook.md) |
| Reviewing a diff for behavior that no check would catch | verification-gap lens (.chaos-engine/references/verification-gap-lens.md) |
| Holding main thread at session start | orchestrator bootstrap (.chaos-engine/references/orchestrator-bootstrap.md) |
A repository CI script or guard under scripts/, outside agent guidance | agent guidance, then the guard's own test |
| In-repo Markdown for humans, such as README or CONTRIBUTING | public docs |
The .memory store itself: entries, relations, or hygiene | agent guidance |
| Cheap, bounded, already-specified workstation coding | workstation local coding |
Deep SHAFT internals
Read the one chapter the task actually touches, and skip the rest. Each encodes incident history that is expensive to re-derive.
| Task touches | Load exactly this |
|---|---|
| Recorder, preload scripts, network capture, browser lifecycle | Selenium BiDi |
| Report generation or patching, results JSON, verdict analysis | Allure internals |
| Mobile recording and replay, emulators, Appium or WinAppDriver CI | Appium mobile |
| Versioning, Central publishing, BOM, dependency convergence | Maven release |
| Listeners, forked JVMs, properties precedence, scoped runs | TestNG lifecycle |
| The IntelliJ plugin, its desktop UI, tool windows, Gradle or JDK setup | IntelliJ plugin development |
| IntelliJ Design/Automation MCP tools that paint IDE state | Design/Automation apply paths |
| IntelliJ plugin UI tests, icon-only controls | IntelliJ UI test conventions |
| MCP tools, stdio transport, workspace roots, clients | MCP protocol |
| Red CI runs, scheduled suites, workflow YAML, sharding | CI forensics |
| Races, synchronization, deterministic reproduction | Wait strategies |
| Locator choice, semantic selectors, healer or doctor | Locator healing |
The plugin-development playbook loads the IntelliJ plugin mastery chapter and must not duplicate it.
SHAFT product and test authoring
Writing, planning, recording, executing, diagnosing, or reporting SHAFT tests is
a different surface with its own router. Hand off to shaft-developer in the
distributed skills pack and let it select the single lifecycle, implementation,
or tool specialist. Do not duplicate its rows here, and do not guess SHAFT
syntax, tool names, or CLI flags from this file.
External facts
- Past-cutoff external library API: context7 MCP; prefer a repo exemplar otherwise.
- Live Maven Central facts: Maven MCP; for in-tree dependency declarations, inspect POMs first.
- Never start unavailable infrastructure implicitly.
UI routes preserve native technology: desktop surfaces use IDE and JVM inspection with renderer evidence; web surfaces use browser evidence and an accessibility audit.
Self-improve roadmap (profile)
When running a Learning Session or planning harness/product self-enhance work,
load self-improve master plan. Cited online
research lives in self-improve research synthesis.
Product lessons → SHAFT issues/evals: product-track playbook (portable base: self-improve product-track (.chaos-engine/skills/self-improve/references/product-track.md)).
Research/explore isolation: context firewall (.chaos-engine/references/context-firewall.md).
S4 self-improve (portable pointers)
- Periodic meta-optimize (
.chaos-engine/references/meta-optimize.md) — not continuous / no Observer - Opt-in draft skill PRs (
.chaos-engine/references/draft-skill-pr.md) — default OFF; never auto-merge
ChaosEngine Self-Improve Master Plan
Owner: Mohab Mohie / ShaftHQ/SHAFT_ENGINE
Dated (Africa/Cairo): 2026-09-08
Base tip: 77a273d2a8 — fix(ce): enforce Learning Session after every delivery (harness parity) (#5641)
Supersedes: prior optimization top-10 at analysis SHA 0961058e66 / /workspace/chaos-engine-top10-optimizations.md (ideas #1–#10 landed as #5619–#5625 under epic #5618).
Method: tip deep-read of skills/self-improve/**, learning.py, Stop/Learning Session hooks, token/parity/delivery refs; online harness synthesis; optional Grok CLI consult (/tmp/ce-self-improve-consult.txt).
Constraint: NEVER Cursor Cloud Agents; merge commits only; portable overlay parity across Codex / Claude / Grok CLI / Gemini / Copilot / Grok Bot.
Research synthesis (cited URLs): self-improve-research-synthesis.md.
0. Locked prefs (do not reopen)
| Pref | Status |
|---|---|
| NO native Task Observer / always-on observation | Locked reject (#5643; research-adopt-reject) |
| Lean self-improve: SessionStart locator-only; Learning Session on delivery Stop | Shipped (#5641 / #5613) |
| Harness parity across six hosts (overlay, not agent memory) | Shipped policy (#5645 / host-parity-matrix) |
| One-command install/configure/update | Shipped north star (#5618 / #5620) |
| Caveman + Ponytail companions | Shipped (token-budget-modes) |
| Issues-first self-improve; eval-gated draft skill PRs opt-in later | Locked (#5618 fork #5) |
| Merge commits only | Locked |
1. Current-state assessment (post-#5641)
1.1 What CE can already self-enhance
| Loop | Product (SHAFT) | Harness (CE overlay) | Token profile |
|---|---|---|---|
| Capture | Dual-track Learning Session → learning.py queue → GitHub issues | Same path; categories allow-list; privacy gates | Heavy only on delivery Stop |
| Enforce learn | Stop / delivery-complete owes Learning Session even if the harness tree is untouched | Portable learning_session.py finalize + monorepo controller | Hook = 0 context unless block reason |
| Discover surfaces | Level-1 catalog + router Heal / zero-LLM | SessionStart locators under SESSION_START_MAX_BYTES=4096 | ↓ always-on |
| Health / heal | Installer/doctor/repair components | status ⊆ doctor, activationProof, repair --component | Zero-LLM |
| Delivery gates | Triage-scaled research + phase ledger | Stop Learning Session; catastrophic deny | Soft vs hard by triage |
| Memory | Owner wake pack locator; retrieve orchestrator | Soft-degrade Memory/MemPalace/Graphify | No SessionStart prose dump |
| Eval | Product CI / ChaosGauge | Eval-parity fixtures + host matrix | CI, not chat |
1.2 Dual-track self-improve ability (honest)
Strengths
- Stop-gated Learning Session is deterministic (hook), not prompt hope — aligns with industry “hooks = 100% enforce, 0 context unless output.”
- Issues-first + privacy-gated
learning.pyis a durable LogAct-like append log (queue → GitHub), not transcript hoarding. - Progressive disclosure skeleton exists: Level-1 catalog, locator SessionStart, on-demand self-improve refs.
- Optimization program #5618 children (#5619–#5625) closed the prior top-10 (health truth, repair, zero-LLM locator, L1 catalog, triage research+ledger, wake+retrieve, eval-parity+portable finalize).
Gaps (self-improve-specific — this plan’s focus)
- No closed-loop metrics: queued → submitted → adopted/rejected → estimatedTokens delta is not machine-summarized for doctor/zero-LLM.
- No once-per-task heuristic retrieve: lessons live in issues/
.memorybut are not ERL-style top-k retrieved at triage (still avoid every-turn injection). - Verification dumps still risk token rot: Check/Stop paths lack a universal silent-on-success / errors-only wrapper contract.
- Skill body bloat risk: no SkillOpt-style compression/eval gate before growing SKILL.md (L2 <500 lines discipline is documented, not automated).
- CLI vs MCP preference is tribal in places; HumanLayer lesson not yet an iron-law + zero-LLM catalog row.
- Significance filter for mid-session capture is skill-discipline only — easy to miss under Caveman ultra (without becoming Task Observer).
- Phase-2 eval-gated draft skill PRs still deferred (correct) but needs a ready gate checklist.
- Product-track playbook thinner than harness track (taxonomy exists; SHAFT-specific acceptance/eval linkage weak).
- GAP-EXIT2 still soft on Grok/Copilot — compensating UX exists; not a self-improve blocker but limits Stop-gate fidelity.
- Periodic meta-optimize (supervisor over shared logs) not scheduled — only per-delivery Learning Session.
1.3 Gap analysis vs #5641
| #5641 deliverable | Status | Remaining self-improve work |
|---|---|---|
| Merge/delivery → Learning Session owed | Shipped | Metrics that prove sessions actually queue/submit |
| Docs forbid CE-untouched skip | Shipped | Product-track yield reporting |
| Harness parity in overlay | Shipped | Encode new lessons as hooks/skills/scripts, never agent-only memory |
| SessionStart locator-only | Shipped | Keep; add locator to heuristic index only (no prose) |
| Unit tests for merge→Stop | Shipped | Expand fixtures for metrics + silent-verify + heuristic retrieve |
Verdict: #5641 closed the enforcement gap. This plan closes the yield / token-efficiency / closed-loop gap for both product and harness.
3. Master roadmap waves (self-improve next)
Dependency order. Do not re-implement #5619–#5625.
Wave S0 — Measure (zero/low LLM)
[#1 Learning metrics closed loop]
Wave S1 — Token sinks on the hot path
[#2 Silent-on-success verify] → [#5 CLI-over-MCP iron law]
↘ [#3 ERL once-per-task heuristics]
Wave S2 — Capture quality without Observer
[#4 Significance-filtered mid-session capture] → [#6 SkillOpt compress audit]
Wave S3 — Isolation & product yield
[#7 Subagent research firewall guidance] → [#10 Product-track playbook + ChaosGauge link]
Wave S4 — Close the mutate loop (opt-in)
[#9 Periodic meta-optimize] → [#8 Eval-gated draft skill PRs (opt-in)]
Out of scope forever (unless prefs unlock): always-on SessionStart full skill/frontmatter scan; native Task Observer; auto-merge skill mutations; pretending GAP-EXIT2 hard process blocks.
4. Top 10 recommendations (ranked)
Each item: why · token impact · dependency · proposed CE shape · Task Observer accept/reject.
1. Learning metrics closed loop (queued → adopted → token delta)
- Why: LogAct shows shared structured logs + light supervisor yield more unique work at lower tokens. CE already appends via
learning.py/.memory/ phase ledger but does not summarize adoption. - Token impact: ↓↓ (zero-LLM doctor/CLI; avoids re-discovering known lessons)
- Dependency: none (S0)
- CE shape:
learning.py metrics|summary+doctor --json.learningMetricsfields: queued, submitted, open/closed issues, estimatedTokens sum, optional adopt/reject labels; link issue numbers only. - vs Task Observer: Reject Observer. Metrics are offline/zero-LLM over existing logs.
2. Silent-on-success / errors-only verification wrappers
- Why: HumanLayer: green test dumps rotate agents into the dumb zone; success must add zero context.
- Token impact: ↓↓↓ on Check/Stop
- Dependency: after #1 optional; independent OK
- CE shape:
references/silent-verify.md+ thinscripts/chaos-enginehelper used by delivery Check + optional Stop verify; exit 0 → no stdout; fail → stderr + exit 2 where honored. - vs Task Observer: Reject Observer. Pure hook/script back-pressure.
3. ERL-style heuristic retrieve once per task (not every turn)
- Why: ERL: post-task heuristics + top-k inject at new task start beats always-on observation.
- Token impact: ↓ always-on; slight ↑ once at triage (bounded top-k, privacy-safe)
- Dependency: prefers #1 metrics for ranking; wake pack / retrieve patterns (#5624)
- CE shape:
.chaos-engine-state/heuristics/or Memory category; SessionStart locator only; triage/router callsretrieve.py heuristics --top 3once; never PreToolUse spam. - vs Task Observer: Reject Observer. Adopt ERL retrieve-once; reject continuous scan (#5643).
4. Significance-filtered mid-session capture (skill + optional soft PostToolUse)
- Why: Under Caveman ultra, dual-track capture is easy to miss until Stop; need cheap friction marks without full observation protocol.
- Token impact: ↓ vs batching failures; ↑ tiny if soft hook emits one-line marks
- Dependency: taxonomy + learning.py; must not load self-improve refs mid-turn
- CE shape: optional PostToolUse recorder for deny/fail signatures → state file; Learning Session reads marks. Default off or ultra-cheap. Significance filter: errors, repeated denials, doctor recovery — not every edit.
- vs Task Observer: Reject Observer. Filter + deferred heavy path only.
5. CLI-over-MCP preference (iron law + zero-LLM catalog)
- Why: HumanLayer: CLI in training data beats MCP schema tax for gh/git/docker/db.
- Token impact: ↓↓ system prompt + tool-result size
- Dependency: zero-llm-catalog / Level-1 (#5621/#5622)
- CE shape: router iron-law row + catalog entries; MCP only when no equivalent CLI exists.
- vs Task Observer: N/A (tooling policy).
6. SkillOpt-style SKILL.md compression audit (propose, don’t auto-apply)
- Why: Red Hat ACE / SkillOpt: ~90% volume cut with quality preserved in experiments; L2 <500 lines.
- Token impact: ↓↓ on skill activation
- Dependency: eval-parity (#5625); metrics #1 help prioritize which skills
- CE shape: offline CLI proposing compressed SKILL.md + refs split; CI eval gate; issues-first PR by human. No live auto-edit.
- vs Task Observer: Reject auto-mutate. Aligns with issues-first / opt-in draft PRs later.
7. Subagent / context-firewall guidance for research & explore
- Why: HumanLayer + capitalandcompute: isolation keeps parent in smart zone; explore noise stays in child.
- Token impact: ↓ parent context (child spend may ↑ — net quality win)
- Dependency: host adapters; local-coding-delegate / omniroute optional
- CE shape: short Level-1 + router rule: research/multi-file explore → subagent/Task when host supports; return filepath:line citations only.
- vs Task Observer: N/A.
8. Eval-gated draft skill PRs — opt-in phase 2
- Why: Hermes/SkillOpt closed loops need eval; CE locked issues-first until ready.
- Token impact: ↑ CI only; ↓ bad skill merges long-term
- Dependency: #1 metrics, #6 compress, #5625 fixtures green, portable finalize stable
- CE shape: flag
CHAOS_ENGINE_SELF_IMPROVE_DRAFT_PR=1; draft PR from staged branch after eval-parity + unit tests; never auto-merge. - vs Task Observer: Reject Observer. Opt-in mutate path only after eval.
9. Periodic meta-optimize over shared logs (not continuous)
- Why: LogAct supervisor: periodic introspection of shared logs beats per-turn gossip.
- Token impact: ↓ vs continuous; bounded scheduled LLM or zero-LLM cluster
- Dependency: #1 metrics; Learning Session history
- CE shape: operator
/learning metaor weekly script clustering open learning issues +.memoryfacts → ranked backlog issue; no SessionStart involvement. - vs Task Observer: Reject Observer. Periodic only.
10. Product-track self-improve playbook (SHAFT + ChaosGauge)
- Why: Dual-track requires equal product discipline; recent #5644/#5646/#5647 show product lessons already valuable.
- Token impact: ↓ rediscovery; slight ↑ at Learning Session when product work occurred
- Dependency: taxonomy; metrics #1; silent-verify #2 for product Check
- CE shape:
skills/self-improve/references/product-track.md— issue templates, link tests/ChaosGauge, prefer CLI/doctor/silent-verify product fixes over essay tickets. - vs Task Observer: Reject Observer. Delivery-Stop capture only.
5. Accept / reject matrix (Task Observer principles)
| Principle | Decision | CE encoding |
|---|---|---|
| Always-on SessionStart full scan | REJECT | Locator-only; SESSION_START_MAX_BYTES |
| Continuous observation every turn | REJECT | #5643; research-adopt-reject |
| Lean core + on-demand refs | ACCEPT | self-improve SKILL + references |
| Capture vs review split | ACCEPT | queue now; human/CI apply later |
| Dual-track harness + product | ACCEPT | taxonomy + Learning Session checklist |
| Auto-stage / auto-install skill patches | REJECT (phase-2 draft PR opt-in only) | learning.py issues-first |
| Weekly auto-mutate live skills | REJECT | meta-optimize proposes backlog only |
| Post-delivery Learning Session gate | ACCEPT | #5641 Stop hooks |
| ERL retrieve-once heuristics | ACCEPT (adapted) | Top 10 #3 |
| LogAct metrics / periodic supervisor | ACCEPT (adapted) | Top 10 #1 #9 |
| Silent-on-success verify | ACCEPT | Top 10 #2 |
6. Implementation notes (when executing children)
- Prefer one condensed PR per wave (S0→S4); children under the epic;
Fixes #<child>never the epic. - Every lasting behavior → portable chaosengine-harness (hooks/skills/scripts/doctor) + host adapters as needed — harness parity.
- Expand eval-parity fixtures when adding gates; never weaken fixtures.
- Keep Ponytail and Caveman ultra.
- No Cursor Cloud Agents for implementation workstreams.
9. Supersession notice
The 2026-09-07 dependency-ranked top-10 (health plane → repair → zero-LLM locator → L1 catalog → research ledger → wake/retrieve → token-budget → eval-gated improve → fixtures → Heal/GAP-EXIT2) is complete via #5619–#5625 + #5641. Do not reopen those tickets. Track new work from §4 Top 10 under the self-improve epic filed alongside this document.
ChaosEngine Self-Improve — Research Synthesis
Companion to self-improve-master-plan.md. Dated (Africa/Cairo): 2026-09-08. Citations only — no always-on injection.
2. Online research synthesis (cited)
2.1 Context surfaces — put the right thing on the right primitive
Sources:
- https://amux.io/guides/context-engineering/
- https://capitalandcompute.net/blog/claude-code-harness-guide/
- https://next.redhat.com/2026/07/28/building-skills-for-ai-agents-pitfalls-and-best-practices/
- https://agentskills.io/specification
| Surface | Load | Enforce | CE mapping |
|---|---|---|---|
| AGENTS/CLAUDE.md | Every turn | Soft ~70% | Keep lean (<~200–500 lines); pointers only |
| Hooks | Event; 0 context unless output | Hard 100% | Safety, Learning Session Stop, research gate |
| Skills | Name+desc always; body on demand | Soft | Router + self-improve + companions |
| Subagents | On spawn; isolated | Soft | Research/explore firewall |
| Memory/stores | Locator / on demand | Soft | Wake pack locator; retrieve orchestrator |
| MCP | Names cheap; schemas deferred | Tools | Prefer CLI when training-data CLI exists |
CE accept: hooks for must-happen; skills for judgment; lean host md files.
CE reject: stuffing safety or Learning Session duty into AGENTS.md alone.
2.2 HumanLayer harness lessons
Sources:
- https://www.humanlayer.dev/blog/skill-issue-harness-engineering-for-coding-agents
- https://www.humanlayer.dev/blog/context-efficient-backpressure
- https://www.humanlayer.dev/blog/advanced-context-engineering
- https://www.humanlayer.dev/blog/writing-a-good-claude-md
- CLI over MCP when CLI is in training data (gh, git, docker, …) — lower schema tax + composability (
jq/rg). - Subagents for context isolation, not role-play org charts.
- Verification silent-on-success / errors-only — swallow green test dumps; exit 2 on failure to re-engage.
- Hooks for control flow; Stop hooks for completion gates (maps directly to #5641).
2.3 Hooks economics
Sources: capitalandcompute harness guide; Anthropic hooks docs (code.claude.com/docs/en/hooks-guide); CE lifecycle-hooks.
- SessionStart / Stop / PreToolUse run outside the conversation budget.
- Stop is the right place for Learning Session and verify gates.
- Aligns with CE: locator SessionStart; Learning Session on Stop (#5641); no always-on observer.
2.4 Self-improve without always-on Task Observer
Sources:
- https://arxiv.org/pdf/2603.24639v2 (ERL — Experiential Reflective Learning)
- https://arxiv.org/pdf/2604.17091 (GenericAgent density / layered memory)
- https://arxiv.org/html/2604.07988 (LogAct shared logs + supervisor)
- CE
research-adopt-reject.md+ #5643
| Pattern | Token-safe shape for CE | Reject |
|---|---|---|
| ERL heuristics | Post-task extract → store; retrieve top-k once at task start (locator + bounded) | Inject every turn / full SessionStart scan |
| GenericAgent L0/L1 | Wake pack + Level-1 catalog; L2/L3 on demand | Always-on L2/L3 dumps |
| LogAct supervisor | Shared structured logs (phase ledger, learning queue, .memory) → periodic meta-optimize | Continuous Task Observer |
| SkillOpt | Offline compress SKILL.md (~90% volume cut reported by Red Hat ACE) with eval gate | Auto-mutate live skills |
2.5 Maximum output, minimal / zero tokens — governing factors
- Deterministic scripts & hooks before LLM narration (Red Hat: scripts cut RCA cost ~26%).
- Progressive disclosure (L1/L2/L3) — description is the router.
- Silent success on verify (HumanLayer).
- Fewer sharper tools / CLI>MCP (HumanLayer; Vercel tool-thinning lesson in prior dossier).
- Shared logs → unique work (LogAct: +17% work / −41% tokens with supervisor over logs — aspirational, not a CE claim of identical gains).
- Measure then mutate — metrics before growing prompts.
- Issues-first durable encode — portable overlay parity (#5645).
Supporting dossier: /workspace/agentic-harness-research.md (Winder, Goose, OpenHands, Hermes, agentskills.io, Gu system-scaling).
7. Key online sources used
- https://amux.io/guides/context-engineering/
- https://capitalandcompute.net/blog/claude-code-harness-guide/
- https://next.redhat.com/2026/07/28/building-skills-for-ai-agents-pitfalls-and-best-practices/
- https://www.humanlayer.dev/blog/skill-issue-harness-engineering-for-coding-agents
- https://www.humanlayer.dev/blog/context-efficient-backpressure
- https://www.humanlayer.dev/blog/advanced-context-engineering
- https://agentskills.io/specification
- https://arxiv.org/pdf/2603.24639v2 (ERL)
- https://arxiv.org/pdf/2604.17091 (GenericAgent)
- https://arxiv.org/html/2604.07988 (LogAct)
- https://github.com/microsoft/SkillOpt
- https://code.claude.com/docs/en/hooks-guide
- https://winder.ai/ai-agent-harness-comparison/ (prior dossier)
- Local:
/workspace/agentic-harness-research.md,/workspace/chaos-engine-top10-optimizations.md(superseded)
8. Consult notes (optional Grok CLI)
Captured /tmp/ce-self-improve-consult.txt (model host frontier model). Alignment with this plan:
- Measure first (LogAct counters) before growing prompts.
- Silent verify + CLI>MCP + SkillOpt propose-only + ERL once-per-task.
- Reaffirm reject always-on scanners / weekly auto-mutate.
- Ordering tip from consult: metrics → silent verify → CLI>MCP → submit hygiene → SkillOpt → ERL → eval ratchet → GAP-EXIT2 UX.
Allure Report Internals
Results model
allure-results/ holds one *-result.json per test attempt (status:
passed/failed/broken/skipped; failed = assertion, broken = unexpected
exception), plus *-container.json (fixtures/grouping), attachments by UUID,
categories.json, environment.properties, history/ (trend, keyed by
historyId and testCaseId include suite identity before the report is built:
platform, and language when language is a separate cohort. A parameter that
already carries platform or language is counted once. The same platform and
language still share both ids, so a real retry stays a retry. Rewriting ids in
the generated HTML is not the fix. A shared display name across platforms or
languages is a broken report, not a retry.
Verdict discipline (SHAFT hard rule)
An empty or unexpectedly small allure-results invalidates any pass/fail
verdict — count result JSONs and executed tests FIRST, and inspect failed,
broken, retried, and skipped attempts separately before declaring a run green
(repo gotcha allure-result-population). A suite that "passed" with 3 result
files when 40 tests exist means the run silently didn't execute.
Single-file report + patching
allure generate --single-file inlines everything into one huge index.html
(tens of MB). Never Files.readString + String.substring to patch it —
that's 2-3 whole-file copies on the heap and OOMs on large reports (SHAFT
issue #3407). The proven pattern (PR #3433, AllureManager):
- Pass 1: stream with a 64KB sliding-window byte scan, carrying a tail
overlap so markers spanning read-chunk boundaries are found; locate
patch-id presence, first
</head>, last</body>offsets. - Pass 2: sequential copy through a sibling temp file, splicing patches at the recorded offsets; atomic move into place. Skip if already patched.
- ASCII byte search is UTF-8-safe (multibyte sequences can't contain '<').
- If a patch inserts before
</body>, every later offset shifts by the sum of all earlier patch lengths — compute final offsets, don't re-scan. - Windows: replace-moves are not atomic under file-watcher/AV contention;
use
ATOMIC_MOVEwith fallback and never leave a half-written index. - Allure 3 suite widgets may be base64. A plaintext miss is not an empty report. Decode the widget payload with a bounded scan before deciding the suite is absent. Do not
Files.readStringthe whole single-file HTML (#3407). A name that is both plaintext and base64 counts once. An empty results directory (no*-result.json) stays unverified. Decoding does not invent a suite.
Ops rules
- Never
allure serve/allure openin agent sessions (guard-enforced); generate only. History carries over by copyinghistory/from the previous report into results before generating. - Report JS executes at open; theme/branding patches belong in
<head>(style) and before</body>(script) — SHAFT's patch constants must never themselves contain</head>/</body>literals or the marker scan breaks.
Maven-provisioned Allure 3 CLI (#5801 / #5815)
SHAFT never calls a user PATH allure binary (Allure 2 drop: #5798/#5800). Resolution order:
- Maven cache (preferred):
~/.m2/repository/allure/allure-cli/<allure3Version>/node_modules/allure/cli.jsinvoked asnode <cli.js>(portable Node under~/.m2/repository/nodejs/when PATH node is missing). npx --yes allure@<allure3Version>on PATH.- Portable Node + its
npx.
Maven zip coordinates (air-gap, #5815): io.github.shafthq:allure-cli:<allure.cli.version>:zip
(built from allure-cli/ via mvn -f allure-cli/pom.xml clean install -Dgpg.skip or
scripts/maintenance/build-allure-cli-zip.sh). Zip unpacks into the runtime cache layout above.
Artifact on disk: ~/.m2/repository/io/github/shafthq/allure-cli/<version>/allure-cli-<version>.zip.
Operators / CI:
# Online / first warm-up (npm):
mvn -Pprovision-allure-cli -pl shaft-engine -am initialize
# Air-gap when the zip is in local or corporate Maven repo (never calls npm):
mvn -Pprovision-allure-cli-maven -pl shaft-engine -am initialize
Engine bootstrap also unpacks a local Maven zip before falling back to npm.
Root POM property allure.cli.version must stay aligned with Internal.allure3Version() and
allure-cli/pom.xml (currently 3.19.1; independent of allure-bom 3.1.1 Java adapters).
Overrides: -Dallure.cli.cacheRoot=..., -Dallure.cli.mavenZip=..., -Dallure.cli.skipProvision=true
(engine bootstrap only).
Maven Central (#5833): allure-cli/ stays outside the shaft-parent reactor
(version = Allure 3). mavenCentral_cd.yml deploys it standalone after the SHAFT
reactor using the same GPG/OSSRH secrets. After publish, air-gapped CI only needs a
corporate Maven mirror of io.github.shafthq:allure-cli:<allure.cli.version>:zip
plus -Pprovision-allure-cli-maven (no npm).
scripts/ci/verify_maven_central_release.py expects the zip (+ .asc / .pom).
Appium & Mobile Toolchains
Stack shape
Appium 2.x = server + installable drivers (uiautomator2, xcuitest,
windows) + plugins, each versioned independently — pin them in CI, appium driver install at run time is a supply-chain and race liability. SHAFT talks
to it via mobile_* MCP tools and the shaft-mcp mobile toolchain service;
web emulation (Chrome device metrics) is a separate, cheaper path from real
native automation.
Incidents that must not repeat
- Never use
appium driver run windows install-wadin CI: it listsapi.github.com/repos/microsoft/winappdriver/releasesunauthenticated and 403s under shared Actions IP rate limits (#5697). Install a pinned MSI viascripts/ci/install_winappdriver.ps1(or an equivalent authenticated / cached asset path). The older race whereinstall-wadreturned before the MSI finished still applies if you resurrect that path — poll forWinAppDriver.exebefore starting Appium (PR #3408). - Chrome's emulated-device list drifts: Chrome 143 removed "Pixel 5", silently breaking every mobile-web-emulation flow pinned to it. Treat device names as data that rots; validate against the running browser and keep a fallback metrics map (PR #3421).
- Keep the server log retrievable on failure: in PowerShell CI,
Receive-JobBEFOREStop-Job, or the diagnosis evidence dies with the job.
Context switching
Native automation lives in NATIVE_APP context; webviews appear as separate
contexts needing matching chromedriver versions. mobile_get_contexts before
switching; hybrid-app flakiness is usually a context (not locator) problem.
Accessibility-tree reads (mobile_get_accessibility_tree) beat raw XML
source dumps for both tokens and stability.
CI reality
- Android emulators in CI need: KVM/HAXM availability check, boot-completed
polling (
sys.boot_completed), and generous first-boot timeouts; never fixed sleeps. - iOS/Safari and device-cloud (BrowserStack) failures are frequently provider-side or target-site-side: before touching code, curl the target page and check the same test's pass history — a locator that passed 6 of 7 scheduled runs and the element still existing live means external incident, not regression (PR #3408 triage).
GitHub Actions CI Forensics
Triage method (proven on multi-job red runs)
- Pull logs with
gh run view <id> --log/--log-failed; never diagnose from the UI summary alone. - Group failures by EXACT exception signature first — a 10-job red run is usually 2-3 signatures plus noise (PR #3408: 4 jobs one signature, 2 another, 3 unique).
- Classify each group: code defect / stale-already-fixed (check merge timestamps vs run trigger time) / external provider or target-site incident (curl the target live; check the same test's pass history).
- Only then fix — one root cause per commit, workflow-level fixes get extra verification (higher blast radius than app code).
Signals that lie
workflow_runconclusionsuccess≠ delivery: job-levelif:skips count as success.- Releases created with the default
GITHUB_TOKENemit NO release event (anti-recursion) — downstreamon: releaseworkflows silently never run. - A job can stay green while a post-test teardown step OOMs/crashes (post-test bookkeeping isn't test failure) — read to the END of logs of "passing" jobs when hunting incidental defects.
- Fail-fast pipelines mask later-stage bugs: after fixing failure N, expect a NEW failure at stage N+1 that no run has ever reached before.
SHAFT-specific gates
PR-blocking: CodeQL + dependency-review, path-filtered intellij-plugin,
shaft-pilot-release. E2E suites run on their existing nightly schedules; the
guidance validator is manual. So "CI is green" on a PR proves much less than
it suggests — run scripts/ci/local_gate.py (changed-module verify) and the
relevant scoped tests yourself.
Workflow YAML edits
- Parse with PyYAML locally before pushing (syntax + step order).
- Unawaited elevated
Start-Processin PowerShell steps returns instantly — poll for the real readiness condition, never trust step exit. - Keep diagnostic artifacts alive on failure:
Receive-JobbeforeStop-Job, upload logs inif: always()steps. - Sharded runs: merge via explicit post-step (SHAFT:
report_merge_shards,assemble_shard_blob.py) — shard-local reports alone hide cross-shard patterns.
Nightly-only E2E execution
Agents never manually start, rerun, or replace E2E Tests or Local E2E Tests,
including targeted dispatches. Prove a fix with the smallest directly affected
local headless test or local composite-action replay, then use exact-head PR
checks as the delivery gate. Existing scheduled nightly workflows are the sole
E2E execution owner. Agents may inspect scheduled results and fix failures;
they do not wait for, trigger, or substitute remote E2E proof.
Local guard replay — prove a composite action's shell logic without dispatch (#4119)
This proves a workflow-level guard locally without running any remote job.
A composite action's run: step is plain shell once GitHub's
${{ inputs.* }} templating is peeled off — extract, parameterize, fixture,
replay.
- Extract the step's shell body verbatim (e.g.
.github/actions/post-test-report/ action.yml's "Write Summary and Check Test Results" step, idcollect_results, action.yml:119-252). - Parameterize every
${{ inputs.<name> }}token as a same-named shell variable (${{ inputs.module-directory }}->$MODULE_DIR,${{ inputs.job-name }}->$JOB_NAME) with onesedpass — no YAML or Actions runtime involved. - Default
$GITHUB_STEP_SUMMARY/$GITHUB_OUTPUTto/dev/nullwhen unset: the script appends to them unconditionally, and GitHub sets them but a local shell doesn't. - Build a fixture tree from a real run's own evidence, one directory per
candidate input value: an
allure-results/*-result.jsonfile per scenario and anallure-report/summary.jsoncarrying the real totals. Directories the guard should find nothing in are simply absent — that absence is the condition under test. - Run the extracted script once per candidate input value and read the exit code and printed summary line. No runner queue or waiting.
Proven this way for #4119: MacOSX_Safari_Cucumber_BrowserStack's 13 Cucumber
scenarios actually land under shaft-engine/ (its -Dtest selector only
matches shaft-engine's cucumberTestRunner.CucumberTests, pulled in via
-am), never under the job's own -pl target shaft-browserstack/.
Replaying action.yml's guard (RESULT_COUNT at action.yml:149-155, the
no-summary branch at action.yml:239-243) against a fixture built from that
real 13-passed evidence:
MODULE_DIR=shaft-browserstack(the #4079 override, matching-plrather than where output actually lands) —Allure Report Summary: Total=0, Passed=0, Failed=0, Broken=0, Skipped=0then::error::No Allure summary was generated for shaft-browserstack, and shaft-browserstack/allure-results does not exist., exit 1: the exact false failure #4119 reported on a fully passing run.MODULE_DIR=shaft-engine(the default at action.yml:33; e2eTests.yml:754-760 no longer overrides it, fixed by #4135) —Allure Report Summary: Total=13, Passed=13, Failed=0, Broken=0, Skipped=0thenAll tests passed according to Allure report., exit 0.
Generalizes to any composite-action guard: peel the templating, fixture the filesystem state the shell actually branches on, and replay. Scheduled nightly owns any later remote E2E evidence.
IntelliJ Platform Plugin Development (shaft-intellij)
Build reality
Use the checked-in Gradle 9.3+ wrapper with JDK 25 as the build runtime;
source/target compatibility remains 17 for IntelliJ Platform compatibility.
UI evidence comes from
ShaftPluginScreenshotRendererTest -Dshaft.intellij.screenshotDir=... —
visible UI changes must regenerate screenshots for the PR.
Issue #3784 recorded one Windows OpenJDK 25.0.1 G1 crash and introduced a
temporary JDK 21 ceiling. That build already used Gradle 9.3, so the wrapper
version does not explain the crash. Issue #4779 raised the ceiling to JDK 25
only after
repeated fresh Windows JDK 25 builds passed and the supported Gradle/JDK
combination was reconfirmed; newer daemon JDKs remain rejected. CI must invoke
shaft-intellij/gradlew so its runtime support stays coupled to the checked-in
wrapper instead of a separately pinned system Gradle.
Threading & UI rules
- EDT-only for Swing mutation; use
invokeLater/Application.invokeLaterfrom background threads, and never block the EDT on MCP/process I/O — every shaft-mcp call from the plugin belongs on a pooled thread with an EDT callback. StatusText/getEmptyText()placeholder lines clip at a fixed internal width (~42 chars) regardless of component width and never wrap — don't fight it with shorter fragments; paint placeholders manually (paintComponent+ FontMetrics word-wrap) and expose text via a client property for tests (SHAFT'sPlaceholderTextArea).- Long-lived background pollers (heartbeats spawning a fresh MCP process every 30s per panel) are a leak pattern — lifecycle-scope them.
- Capture
this::methodas a field when the same reference must later be removed (e.g.addStateChangeListener/removeStateChangeListenerpairs inaddNotify()/removeNotify()) — re-evaluating a method-reference expression is not guaranteed to produce an identity-equal object, so a fresh one at the removal call site silently fails to remove anything (issue #3621).
Trust & failure surfacing (hard-won)
- An MCP tool result with
isError: trueinside a successful JSON-RPC envelope IS a failure — surface the tool's own text; masking it as "Finished/OK" hides every downstream failure mode (PR #3396's highest-blast-radius fix). - Gate SHAFT-specific actions on real project detection (pom/gradle contains shaft coordinates; fail-closed, cached) so the plugin is inert in ordinary projects.
- Setup-step badges derive ONLY from real verification checks, never from "the user clicked the button" (PR #3427). No phantom recorder steps: suppress browser-synthesized events that don't map to a user action.
- Long-running work (recordings) must never belong to an ephemeral agent turn/process — start it in a service whose lifetime the tool window owns (PR #3431).
Testing
Plugin tests run headless via the IntelliJ test framework; Swing panels are unit-testable if logic is separated from EDT wiring. Playwright/webapp tools cannot see Swing — evidence is the screenshot renderer, not a browser.
Any test that installs a real platform L&F (UIManager.setLookAndFeel) must
register LookAndFeelIsolationExtension (shaft-intellij/src/test/java/.../ui/,
issue #3782) so it snapshots/restores the LookAndFeel instance, JBColor
dark flag, and declared UIManager keys on the EDT — UIManager.put()
overrides persist across L&F switches by Swing design, and IntelliJ's
DefaultTreeUI only becomes the active TreeUI delegate once a platform
L&F has been installed at least once in the JVM, a latch that restoring the
L&F instance does not undo. Confirmed live (issue #3786): running the
whole module's gradlew test together with
-Dshaft.intellij.screenshotDir=... still breaks ShaftTestsPanelTest's
tree-selection tests via that sticky latch even with the extension in
place — always scope screenshot regeneration with
--tests ShaftPluginScreenshotRendererTest, never a bare full-suite run.
Locator Design & Self-Healing
Locator ladder (stop at the first stable rung)
test id > role + accessible name > stable attributes in a stable container >
text (assertions only; i18n-brittle) > structural CSS/XPath (last resort;
never index- or style-class-based). SHAFT's By-based APIs and
aria-snapshot assertions (matchesAriaSnapshot, partial-subset semantics)
reward the role+name rung heavily.
For SHAFT-generated/codegen output specifically, this collapses to three
legal tiers (shaft-locator-design owns the full policy; enforced
mechanically by shaft-capture's LocatorPolicy, issue #4271): (1) a
unique, author-written id through the SHAFT locator builder
(hasAnyTagName().hasId(...)) — never a framework-recycled id such as
:r1:, mat-input-3, cdk-overlay-0, ember1234, j_idt42, ctl00_...
or sc-bdVaJa; (2) ARIA-role-powered XPath via the same builder
(hasRole(...), chained with hasNormalizedText/hasAttribute/context);
(3) native relative By.xpath(...) only when the element has neither. An
element matching no tier fails generation rather than degrading. Never emit
SHAFT.GUI.Locator.xpath(...) or the raw
SHAFT.GUI.Locator.id/name/cssSelector/className/tagName(...) factories.
Smart Locator (inputField/clickableField) is never emitted into generated
code — test_code_guardrails_check flags it (SMART_LOCATOR) — and is
legitimate only for a human's own throwaway exploration snippet.
Recorder truth rules (SHAFT-specific, hard-won)
- Recorded locators must map 1:1 to real user actions; suppress browser-synthesized phantom steps (subframe storage restores, PR #3432; click-tracking state, PR #3427 — never reintroduce).
- Codegen validates by compile-and-replay (
GeneratedTestValidator) — a generated locator is unproven until replay passes headlessly.
Self-healing (shaft-heal / doctor)
- Healing proposes a NEW locator from a failure snapshot
(
doctor_propose_healed_locator); score candidates by multi-signal similarity (attributes, text, neighbors, position) and NEVER auto-accept silently — healed locators are review-gated changes, else suites drift to asserting the wrong elements. - Confidence scoring must penalize uncited/unevidenced output symmetrically
— SHAFT's
DoctorConfidenceScoreronce scored zero-citation recommended actions 85/100 because only hypotheses had a penalty branch (PR #3344). When extending scorers, check every factor has both directions. - Persisting healing history: append/merge, never rewrite the store (a data-wiping rewrite bug shipped once, issue #3315).
healer_run_failed_testcan burn multiple real Maven builds — precheck that Allure output exists before invoking heavy loops.
Debugging a "broken" locator
Order: is the element in a different frame/context → is there a wait/race → did the DOM actually change (diff live DOM vs expectation) → only then rewrite the locator. Most "locator rot" reports are frame or timing bugs.
Maven Release Engineering
SHAFT publishing shape
Multi-module reactor: core shaft-engine, optional shaft-* feature
modules, BOM and compatibility/publication modules. Consumers pin one
version; the BOM must stay internally consistent. Release flow runs GPG
signing — locally it fails without -Dgpg.skip=true; the proven local
install for live validation is
mvn -pl shaft-engine,shaft-capture,shaft-mcp install -DskipTests -Dgpg.skip=true.
Dependency convergence
maven-enforcer-pluginlives in root pluginManagement, activated in shaft-engine (banned-deps + convergence) — it is the cheap local mirror of merge-time CI breakage: run a targetedmvn -pl <module> verify -DskipTestsbefore pushing dependency changes.- Transitives can be internally inconsistent:
jython-slim:2.7.4declares jffi 1.3.13 direct AND jffi-native 1.4.0 — converge both classifiers independencyManagement, not just the main artifact (PR #3344). - Excluding a transitive (e.g.
org.openpnp:opencvfrom sikulixapi) without re-declaring it directly compiles fine and dies at runtime withNoClassDefFoundError— check what sibling modules with the same need declare (shaft-sikulix vs shaft-visual, PR #3408).
Resources & packaging traps
- An explicit
<resources><includes>allowlist silently drops every file not listed — new runtime resources (JS payloads likeaxe.min.js) never reachtarget/classesor the jar while unit tests that read from source tree still pass (PR #3383). Verify packaging withunzip -lon the jar. - Version shadowing: locally-installed
.m2jars mask Central artifacts of the same version; a fix that "works locally" may be running a patched local build. Check jar timestamps/hashes when validation seems too easy.
Verdict rules
workflow_runconclusionsuccessonly means no job FAILED — job-levelif:skips count as success. A "green" release workflow can have delivered nothing (repo gotcha; also: releases created with the default GITHUB_TOKEN emit NO release event — dependent workflows never fire).- Release-candidate pipelines fail fast and mask later-stage bugs; a new failure after a fixed one is usually a genuinely new issue further along, not a regression of the fix.
MCP Protocol Engineering (shaft-mcp)
Transport truths
stdio transport = newline-delimited JSON-RPC on stdin/stdout; stdout is
PROTOCOL-ONLY — any stray print (logging, Maven banners, JVM warnings)
corrupts the stream. All diagnostics go to stderr. The proven E2E validation
pattern: a throwaway Python stdio driver against the real
java @shaft-mcp.args process with cwd = a consumer project (repo workflow
live-validating-shaft-mcp-fixes-end-to-end-via-a-scripted-stdio-driver).
Error semantics (three layers, all different)
- Transport/JSON-RPC error — request failed.
isError: truein a SUCCESSFUL tool result — the tool failed; clients that treat the envelope's success as the verdict mask every real failure.- Domain "soft fail" in payload text — needs explicit checks. Always propagate layer-2 text to the user/client verbatim.
Workspace roots
The workspace root resolves from the CLIENT LAUNCH DIRECTORY; installers
write only -Dshaft.mcp.fallbackWorkspaceRoot, never -Duser.dir or a hard
workspace root, and a bare user home must never become a workspace
(PR #3428 decisions, validator-enforced). Scoping to Path.of(".") (IDE
cwd) instead of the open project is a recurring bug shape.
Long-lived state vs one-shot turns
Recordings/captures are stateful sessions that must outlive the initiating request. Failure shapes seen live: capture session JSON vanishing mid-run on Windows (non-atomic replace-move under AV/file-watcher contention) wedging status into an "already active vs not started" split-brain — lifecycle code must self-heal from a missing store, never wedge until process death (PR #3431).
Client-side gotchas
claude --print+acceptEditsauto-DENIES every Bash/MCP tool call unless--permission-prompt-tool(an MCP-over-HTTP approval bridge) is wired — flag exists in the binary but not--help(PR #3420/#3411).- Live-probe any CLI's wire behavior with a script before implementing against it; a prior SHAFT feature shipped dead because it assumed a control-channel that doesn't exist on raw stdio (PR #3373→#3410).
Tool design
Verb_noun names, flat args, one tool = one user-meaningful outcome,
structured content out. Reject dangerous generated code server-side
(test_code_guardrails_check pattern) rather than trusting client prompts.
Selenium WebDriver BiDi
Model
Selenium exposes BiDi (the WebSocket protocol alongside classic WebDriver
HTTP) via Augmenter/HasBiDi; the pinned version lives in the root pom. Key domains: script (preload scripts,
evaluate/callFunction), browsingContext (tree, navigation, screenshots),
network (intercept/continue), log. Everything is per-browsing-context;
"the page" is never one context — every iframe, popup, and worker-adjacent
frame is its own context with its own lifecycle.
Rules that prevent real SHAFT bugs
script.addPreloadScriptruns in EVERY browsing context, including same-origin subframes. Any injected UI or persisted state must be gatedwindow === window.top; same-origin subframes SHARE the top frame's sessionStorage, so a subframe restoring "persisted UI state" creates phantom events (SHAFT recorder incident, PR #3432: phantom Navigate rows).- Injection order races with subframe load timing are real and CI-speed
dependent. Deterministic repro trick: delay a fixture iframe's response
(
?frameDelay=900) to force the slow-CI initialization order locally. - Channel messages (
script.messagefromchannelcallbacks) arrive on the WebSocket dispatch thread — never block it, queue and process elsewhere. - BiDi session lifetime is tied to the driver; a recording/capture session must own a driver whose lifetime outlives the initiating request. Never start a BiDi-dependent recording inside an ephemeral process/turn that exits right after (SHAFT incident: recordings died seconds after start because capture ran inside a one-shot agent child, PR #3431).
- Not all SHAFT driver backends guarantee BiDi (remote grids, some mobile paths) — feature-detect, degrade gracefully, never assume.
Network capture
SHAFT mutation is CDP Fetch in BrowserNetworkInterceptor for HasDevTools
only. BiDi network.addIntercept is not that path. Passive BiDi
(network.responseCompleted, then size-capped network.getData) does not
mock: capture headers and timings eagerly, bodies lazily. Native Appium
traffic is out of band. BrowserNetworkInterceptor and HttpContractRecorder
already normalize contracts — compose, don't rebuild.
Debugging
- Enable BiDi wire logging first (
-Dwebdriver.remote.verbose/ Selenium'sBiDilogger) — most "BiDi bugs" are ordering, visible in the log. - Screenshot via
browsingContext.captureScreenshotworks on non-focused contexts; classic screenshot only on the active top context.
TestNG/JUnit Lifecycle & Forked-JVM Properties
SHAFT's hook points
SHAFT integrates via TestNG listeners (ITestListener, ISuiteListener,
IInvokedMethodListener, registered through ServiceLoader /
META-INF/services) and JUnit 5 extensions — driver lifecycle, Allure
wiring, and reporting all hang off these. Listener exceptions can silently
kill reporting for the whole run; listeners must be defensive.
Failure-type semantics (real SHAFT fact)
Actions.report() wraps element-action failures in plain
RuntimeException; AssertionError is thrown only by the separate
validation/accessibility APIs. A test expecting AssertionError from an
element action is wrong by construction — verify with grep which path
throws what before "fixing" the production side (PR #3408 corrected a
years-wrong test expectation).
Properties precedence (the #1 trap)
SHAFT auto-scaffolds src/main/resources/properties/default/* and reads
custom.properties; a committed or auto-created custom.properties
SILENTLY OVERRIDES -D system properties for some keys. Passing
-DheadlessExecution=true is not reliable when no custom.properties
pre-exists or when one sets headlessExecution=false — pre-seed a
local-only gitignored custom.properties for deterministic headless runs
(repo gotcha; bit PR #3344 and #3408 verification).
Forked-JVM double hop: Surefire forks the test JVM (argLine, systemPropertyVariables),
and SHAFT features that fork AGAIN (replay validation subprocess) must
explicitly re-force critical properties — GeneratedTestValidator.replay()
once forwarded the outer JVM's own headlessExecution instead of forcing
true, crashing only on display-less Linux CI (PR #3344).
Execution scoping (guard-enforced here)
- Never unscoped
mvn teston the reactor; one bad fork can crash the JVM across all modules. Use-Dtest=<Class>or-pl <module>WITHOUT-am. - Every potentially-browser-reaching run gets
-DheadlessExecution=true. - Running SHAFT tests from the repo root auto-creates
src/main/resources/...scaffolding at the CWD — stray untrackedsrc/at reactor root is generated garbage, safe to delete. - Parallel TestNG (
parallel="methods|classes") requires everything ThreadLocal (SHAFT drivers are); a static mutable field in test scope is a flake factory.
N-run proof under failure-ignore (#5739)
The shaft-engine Surefire profile may set testFailureIgnore=true so JaCoCo
can still report. Maven exit 0 then does not prove a green suite.
Flake-proof / Wave N-run scripts must:
- Pass
-Dmaven.test.failure.ignore=falseon the proof invocation. - After each run, parse Surefire
TEST-*.xml(failures/errors) or TestNGtestng-results.xml(failed) and require zero. - Call
python3 scripts/ci/assert_surefire_green.py <surefire-reports-dir>rather than trusting process exit alone.
Flaky-Test Stabilization & Wait Strategies
Doctrine
A flaky test is a bug with a probability attached — remove the race; never
widen a sleep or add blind retries. Condition-polling with a budget (SHAFT
SmartWait / explicit waits) is the only correct wait primitive; wait on the
ACTUAL readiness condition (element interactable, request completed,
sys.boot_completed), not a proxy (time, spinner gone).
Root-cause taxonomy (check in this order)
- Initialization-order races — injection vs subframe load, listener vs first event. CI is slower; order flips there first.
- Shared mutable state — static fields, shared sessionStorage across same-origin frames, shared temp dirs/ports across parallel tests.
- External world drift — target site changed, emulated-device list changed (Chrome 143 removed "Pixel 5"), provider incident. Check pass history + live target before blaming code.
- Environment divergence — headless vs headed, display-less Linux, Windows non-atomic file replace under AV contention, forked JVM missing a property.
- Genuine timing assumptions — animations, debounce, eventual consistency; make the code observable instead of guessing.
Make it deterministic, then fix it
The flake must fail on demand before you trust any fix. Techniques that worked in SHAFT:
- Delay a fixture resource to force the slow ordering
(
?frameDelay=900iframe trick, PR #3432). - Pin the fork/parallelism config down to one method
(
-Dtest=Class#method) to isolate shared-state suspects. - Run the suspicious pair/order explicitly; TestNG order is stable given the same XML — reproduce the exact order CI used.
- Add temporary observation points at the boundary you suspect, not shotgun logging everywhere.
Retries
Retry analyzers are quarantine, not cure: acceptable to keep a release train moving while the root cause is tracked in an issue, never as the closing state. Allure shows retries per historyId — a rising retry count is a regression signal even while the suite stays "green".
SHAFT Skill Evaluation Prompts
Humans: these prompts pressure-test whether an agent applies a SHAFT skill. They are not the product guide (https://shafthq.github.io/).
Agents: load shaft-skills/shaft-developer/SKILL.md for product work, or AGENTS.md at the repository root when changing this repository. Do not treat these prompts as policy. Skip them unless the task evaluates skill routing.
Use these prompts through IntelliJ Assistant /partner or Guided Plan coding partner first. Fresh agent sessions are fallback coverage when subagent or model-evaluation access is approved. They are not a substitute for the repo validation checks; they pressure-test whether agents apply the skills under realistic drift.
shaft-automated-test-authoring
- Use
$shaft-automated-test-authoringto write a TestNG SHAFT web login test for an existing project. The page has Email and Password fields and a Sign in button. Include one hard assertion and no sleeps. - Use
$shaft-automated-test-authoringto review this Java snippet and list only the changes required before it can be accepted: it usesThread.sleep,driver.findElement,System.getProperty, and a hard-coded token. - Use
$shaft-automated-test-authoringto sketch a SHAFT API negative test from an OpenAPI contract where a missing required field should return 400. Keep reusable request/validation code outside the test. - Use
$shaft-automated-test-authoringfor a repo that already hasLoginPage.loginAs(...)and locator fields. Show that you callshaft_coding_partner_planand extend the existing target instead of creatingGeneratedLoginTest. - Use
$shaft-automated-test-authoringto explain whyActions.report()failures and validation/accessibility assertions must be verified against current SHAFT code instead of assumed to share one exception contract.
shaft-locator-design
- Use
$shaft-locator-designto replaceBy.xpath("/html/body/div[2]/form/input[1]")for an email field whose visible label is Email and whose placeholder is Work email. - Use
$shaft-locator-designto choose locators for a checkout button with accessible name Checkout, a generated id, and a stabledata-testid=checkout-submit. - Use
$shaft-locator-designto diagnose a flaky mobile locator where XPath finds two Login buttons but one has accessibility idloginButton. - Use
$shaft-locator-designwhenshaft_coding_partner_planreports an existingemailInputlocator and one missing confirmation-banner locator.
shaft-test-recording
- Use
$shaft-test-recordingto describe the MCP tool sequence for recording a WebDriver checkout flow and integrating it into existing page objects. - Use
$shaft-test-recordingto handle a Capture result with readinessRISKY, one positional CSS warning, and no post-submit assertion. - Use
$shaft-recording-codegento turn a persisted Appium recording into reusable SHAFT mobile test code without leaking typed passwords. - Use
$shaft-recording-codegenafter a successful recording when the repo already hasCheckoutPage. Includeshaft_coding_partner_plan, target-specific code blocks, guardrails, and focused verification.
shaft-failure-analysis
- Use
$shaft-failure-analysisto analyze anallure-resultsdirectory that contains zero*-result.jsonfiles but a stalesummary.jsonsays passed. - Use
$shaft-failure-analysisto triage a failed SHAFT web test withNoSuchElementExceptionafter a retry and a Doctor suggestion to use a new locator. - Use
$shaft-failure-analysisto decide whether to apply a healer-proposed assertion change when the trace shows the product now returns a different validation message. - Use
$shaft-failure-analysisto route a failed login repair throughshaft_coding_partner_planso the fix updates the existing page object and not a new helper.
shaft-change-verification
- Use
$shaft-change-verificationto preview ashaft_coding_partner_diffthat inserts a new page-object method afterloginAs, then decide whether it is safe to apply. - Use
$shaft-change-verificationwhen a generated block containsThread.sleepanddriver.findElement; show that guardrails block the apply and describe the regeneration. - Use
$shaft-change-verificationto choose and run the smallestverify_run_focusedcommand after applying a locator change, and explain why a full suite is not needed. - Use
$shaft-change-verificationafter a cloud provider returnsguardrailStatusthat is notPASSED; show that you do not apply and route back to the coding-partner plan.
shaft-test-planning
- Use
$shaft-test-planningto explore a demo storefront and produce an annotated test plan for guest checkout: discovered flows, priority order, and one automation scenario per flow. - Use
$shaft-test-planningwhen a stakeholder provides only a URL and "test the signup"; show that you explore first with MCP browsing tools and mark unverified assumptions in the plan for human review. - Use
$shaft-test-planningto turn an exploration transcript into scenarios ready for$shaft-test-recording, stating which flows need recording versus direct codegen. - Use
$shaft-test-planningon a repo that already has smoke coverage for login; show that the plan extends existing coverage viashaft_coding_partner_planinstead of duplicating scenarios.
Shaft-skills efficiency contract (#6909)
Each row is one shipped enhancement (E) or closed gap (G). The sentence is the normative text in the mapped skill file.
E01 || shaft-developer/references/routing.md || Load one specialist SKILL.md and its playbook; do not open a second skill in the same turn.
E02 || shaft-developer/references/routing.md || Do not paste MCP JSON schemas into the prompt; call the tool by its exact name.
E03 || shaft-cli/references/playbook.md || Prefer shaft-cli call for a stateless tool so the MCP schema is not loaded.
E04 || shaft-cli/references/playbook.md || Ask shaft-cli tools --cached before a live tool listing, and do not paste the live listing into the chat.
E05 || shaft-developer/references/routing.md || Return a bounded result summary, not raw page source, accessibility XML, or screenshot bytes.
E06 || shaft-developer/references/routing.md || Do not load ce-pack mastery chapters unless the selected playbook names that chapter.
E07 || shaft-developer/references/routing.md || Keep the skill description inside the context budget; do not duplicate a playbook inside SKILL.md.
E08 || shaft-developer/references/routing.md || Stop reading a playbook once the matching practice is found.
E09 || shaft-developer/references/routing.md || A completed mobile or web action names the SHAFT API, the evidence that proves it, and the failure to report.
E10 || shaft-mobile-actions/references/playbook.md || Tap with driver.touch().tap(locator).
E11 || shaft-mobile-actions/references/playbook.md || File silent no-ops upstream.
E12 || shaft-mobile-actions/references/playbook.md || Coordinate taps are forbidden.
E13 || shaft-mobile-actions/references/playbook.md || Record XCUITest or UiAutomator2 before choosing a gesture.
E14 || shaft-developer/references/routing.md || Assertion text names the element and the expected state.
E15 || shaft-mcp/references/playbook.md || Use locator tap or element_click for mobile controls; do not call mobile_tap_coordinates to finish a flow.
E16 || shaft-mcp/references/playbook.md || Initialize mobile_native before mobile tools on a native session.
E17 || shaft-mcp/references/playbook.md || Read isError and warnings before the next MCP call.
E18 || shaft-mcp/references/playbook.md || Keep one MCP client for the whole stateful sequence.
E19 || shaft-mcp/references/playbook.md || Redact tokens from tool output before repeating it.
E20 || shaft-mcp/references/playbook.md || Close with driver_quit when the task owns the driver.
G01 || shaft-mobile-actions/references/playbook.md || A non-hittable iOS element is not a passed tap.
G02 || shaft-mobile-actions/references/playbook.md || driver.element().click(locator) and driver.touch().tap(locator) share one hittable outcome.
G03 || shaft-mobile-actions/references/playbook.md || Do not report pass when the page source is unchanged after a non-hittable tap.
G04 || shaft-cli/references/playbook.md || Generated mobile Java taps with driver.touch().tap(locator) and never tapByCoordinates.
G05 || shaft-defect-reporting/references/playbook.md || Strip auth_token and other query credentials from defect evidence before filing.
G06 || shaft-failure-analysis/references/playbook.md || A later assertion timeout can hide a no-op tap; inspect the tap step before widening the wait.
G07 || shaft-web-actions/references/playbook.md || Native iOS and Android taps belong to shaft-mobile-actions, not this web playbook.
G08 || shaft-recording-codegen/references/playbook.md || Do not generate tapByCoordinates or mobile_tap_coordinates from a recording.
G09 || shaft-developer/references/routing.md || Do not load every shaft-skills playbook for completeness.
G10 || shaft-cli/references/playbook.md || Prefer the checked-in catalog over dumping shaft-cli tools live output into the chat.
G11 || shaft-defect-reporting/references/playbook.md || A coordinate tap that works on a device is diagnosis only, not the fix.
G12 || shaft-locator-design/references/playbook.md || Mobile identity still follows the three locator tiers; coordinates are not a tier.
G13 || shaft-mobile-actions/SKILL.md || On iOS XCUITest, hittable=false means the tap did not land.
G14 || shaft-mcp/references/playbook.md || mobile_tap_coordinates is not a success path.
G15 || shaft-cli/references/playbook.md || Inside a shaft-cli session, mobile taps still use a locator, not viewport coordinates.
G16 || shaft-failure-analysis/references/playbook.md || Do not replace a silent no-op with a coordinate workaround.
G17 || shaft-web-actions/references/playbook.md || An action that returns without an exception is not proof the control fired.
G18 || shaft-recording-codegen/references/playbook.md || Replay-proven code is required before a mobile tap snippet is treated as done.
G19 || shaft-developer/references/routing.md || Never copy session auth tokens from a bug report into logs, commits, or skills.
G20 || shaft-developer/references/routing.md || When the first unique locator tier matches, stop; do not add a coordinate fallback.
SHAFT CLI Command Reference
shaft-cli is a command-line interface to the same shaft-mcp tool set. When a
shaft-cli launcher is installed, prefer it over raw MCP tool calls — a plain
shell command costs fewer agent tokens than an MCP schema load. Fall back to
the shaft-mcp:<tool> MCP calls in shaft-mcp-tools.md when shaft-cli is not
installed or a command fails. For stateful work (recordings, live browser or
mobile drivers) start a shaft-cli session first or stay on a persistent MCP
client — a recording must never belong to a one-shot process (see Sessions
below).
Install: pass --install-shaft-cli to the shaft-mcp installer (see
https://shafthq.github.io/docs/agentic/cli). The launcher lands under the
platform application-data root, e.g.
%LOCALAPPDATA%\ShaftHQ\shaft-cli\versions\<version>\shaft-cli.cmd on Windows.
Tool mapping
Every tool in shaft-mcp-tools.md is callable by its exact name, unprefixed:
shaft-cli call <TOOL> [key=value ...] [--args '<json>'] [--json] [--stdio-ok]
key=valuevalues are coerced (true/false→ boolean, numbers → numeric,{...}/[...]→ JSON, otherwise string);key=valueoverrides--argskeys.--jsonprints the raw JSON-RPC result instead of rendered text.- Exit codes: 0 success; 1 tool or transport error (or
--jsonwithisError); 2 unknown alias action.
Sessions and state
One-shot commands spawn an ephemeral shaft-mcp child per call, so state (browser, driver, capture) does not survive between them. For stateful tools start a session first:
shaft-cli session start # persistent daemon; state survives across commands
shaft-cli session status
shaft-cli session stop
Stateful commands fail fast without a session unless --stdio-ok is passed.
Stateless tools (guide search, doctor analysis) need no session.
Discovery
shaft-cli tools [--json] lists every tool with its description at runtime.
shaft-cli tools --cached [--json] reads the bundled generated copy of the
canonical tool-index.json with no MCP server or one-shot child process. Its
names, descriptions, parameter schemas, and curated metadata exactly match the
shaft-mcp version shipped with that CLI; use the live command only when the
running server may be a different version.
Codegen (deterministic, no MCP session)
shaft-cli codegen [--session <path>] [any CaptureCli generate flag]
generates SHAFT test code directly from a capture recording file — no MCP
server, no session, no --stdio-ok (Decision 6: single-session actions like
codegen prefer the deterministic CLI path over MCP). --session defaults to
the newest *.json file under ./recordings when omitted, mirroring
capture_generate_replay's zero-arg default, so a bare shaft-cli codegen
works right after a capture start/capture stop pair. Every other
CaptureCli generate flag (--backend webdriver|playwright, --output-dir,
--package, --class-name, --overwrite, --replay, ...) passes through
unchanged; exit code 2 means no --session was given and none could be
inferred.
Local infrastructure setup (direct, no MCP session)
shaft-cli setup exposes the release-coupled setup catalog and lifecycle. It
defaults to the read-only EXTERNAL mode. Managed mutations require a plan
file plus its exact printed digest; a stale or edited plan is rejected before
the cache or durable-data roots are created.
shaft-cli setup catalog|profiles [--json]
shaft-cli setup doctor|status|verify --profile REPORTING [--json]
shaft-cli setup plan --profile REPORTING --mode MANAGED --output <absolute-plan.json> [policy options] [--json]
shaft-cli setup install|apply|update --plan <absolute-plan.json> --approve <sha256:digest> [--accept-license <id>] [policy options] [--json]
shaft-cli setup start|stop --profile <profile>
shaft-cli setup logs --profile REPORTING
The first managed provider installs verified, SHAFT-owned portable Node and
Allure 3 artifacts for the REPORTING profile. start and stop return
unsupported for profiles without an owned service; they never adopt or stop
an unknown process. Exit codes are 0 for ready/success, 2 for invalid input or
approval, 3 for missing/degraded readiness, 4 for unsupported providers, and
5 for execution or integrity failures. --cache-root and --data-root are
paired advanced overrides and must both be absolute.
Plan and install must receive the same policy options because those values are
bound into the approved digest: --offline, --auto-start,
--prefer-system-tools=true|false, --reuse-owned-processes=true|false,
--startup-timeout <ISO-8601 duration>, and
--shutdown-timeout <ISO-8601 duration>.
The equivalent Java surface is SHAFT.Infrastructure. Its no-argument
read-only methods use SHAFT.Properties.infrastructure; install and start
require the exact SetupPlan and SetupApproval. Defaults remain
EXTERNAL, offline=false, and autoStart=false. An explicit remote
execution address wins for web, Grid, and mobile endpoint profiles.
Curated aliases
Shortcuts over call, same options:
shaft-cli browser navigate|screenshot|dom|url(session required)shaft-cli element click|type|hover(session required)shaft-cli capture start|stop|status|code|step-delete|step-reorder(session required)shaft-cli guide search(stateless)shaft-cli doctor analyze|suggest|local-ai-status(stateless)
Sponsors
Details
- Marketplace
- ShaftHQ/SHAFT_ENGINE
- Source
- shaft-skills
- Author
- ShaftHQ
- Category
- Testing
- License
- MIT
- Keywords
- shaft, test-automation, selenium, appium, rest-assured, java, testng, junit