Skills and MCP tools for walkerOS - build sources, destinations, transformers, and flows.

GitHub stars
351

Install

/plugin install walkeros@elbwalker

Run it inside Claude Code after adding the marketplace with /plugin marketplace add elbwalker/walkerOS.

Contents

Discovered from source

Skills 22

Create a CMP Source

A CMP source is a specialized walkerOS source that listens to a consent management platform's events and translates consent states into elb('walker consent', state) calls.

Every CMP source follows the same skeleton with only 5-6 decision points that vary per CMP. This skill turns "build a new CMP source" into a structured fill-in-the-blanks workflow.

Prerequisites

Before starting, read these skills:

Supporting files

This skill includes reference files you can copy:

  • examples/ - Generic CMP consent examples to adapt for your CMP
    • inputs.ts - Consent input scenarios (full, partial, minimal, revocation)
    • outputs.ts - Expected walkerOS consent states after mapping
    • env.ts - Mock environment factories (createMockElbFn, createMockWindow, mockEnv)
  • templates/cmp/ - Complete CMP source implementation template
    • index.ts - Source implementation with detection paths + handleConsent
    • types.ts - Type definitions (Settings, CMP API, Types bundle)
    • test-utils.ts - Test utilities (MockWindow, createMockElb, createCmpSource)
    • index.test.ts - Test suite skeleton (8 describe blocks)

Note: These are generic templates for the skill. The actual CMP package's examples go in packages/web/sources/cmps/[name]/src/examples/.

Canonical template

CookieFirst at packages/web/sources/cmps/cookiefirst/ is the canonical template. Copy its structure for every new CMP source.

Note: The CookieFirst package.json predates this skill and lacks the walkerOS-source keyword. New CMP sources MUST include it (the CookiePro and Cookiebot packages do).

Process overview

1. Research     -> Fill in the CMP research template
2. Examples     -> Create input/output examples (full, partial, minimal, revocation, edge cases)
3. Mapping      -> Define category map with sensible defaults
4. Scaffold     -> Copy from CookieFirst template
5. Convention   -> walkerOS.json, buildDev
6. Test         -> Write tests FIRST (TDD): 25-32 tests across 8-9 describe blocks
7. Implement    -> Wire up detection paths + handleConsent + destroy
8. Document     -> README + docs page, registered with the other CMP sources

Phase 1: Research the CMP

Goal: Fill in every field of this template before writing any code.

CMP research template

Fill in ALL of these fields for the target CMP:

FieldDescriptionExample (CookieFirst)
CMP nameOfficial product nameCookieFirst
Global window objectwindow.X shape and key propertieswindow.CookieFirst.consent (boolean map)
SDK eventsEvent names, CustomEvent detail structurecf_init (Event), cf_consent (CustomEvent with consent detail)
Callbacks/hooksFunction wrapping patternsNone
Category namingHuman-readable vs opaque IDsHuman-readable (necessary, functional, performance, advertising)
Consent access patternIs consent read from a property (window.CMP.consent) or an API method (CMP.getConsent())?Property: window.CookieFirst.consent
Explicit consent detectionAPI or mechanism to check if user actively choseconsent === null means no explicit choice
"Already loaded" detectionHow to detect CMP loaded before sourcewindow.CookieFirst.consent exists and is non-null
Official docs URLLink to CMP's developer/API documentationCookieFirst Public API docs
npm packages / TS typesAvailable type packagesNone (define own types)
Timing constraintsDoes the source need to load before/after the CMP? Any require config needed?No require: consent sources should init immediately
Event registration mechanismHow does the CMP register event listeners? addEventListener, callback assignment, SDK method?addEventListener (standard DOM events)
Cleanup/unsubscribe mechanismHow to remove listeners on destroy? removeEventListener, nullify callback, SDK unsubscribe method?removeEventListener (standard DOM cleanup)
SDK readiness patternHow does the CMP signal its SDK is ready? DOM event, callback array, global flag, Promise?cf_init DOM event

Detection paths

Every CMP source needs up to 3 detection paths:

PathPurposeQuestions to answer
Already loadedCMP loaded before sourceIs there a global object to check? What state does it expose? Is consent read from a property or an API method?
Init listenerCMP loads after sourceWhat event/callback fires on SDK init? Is it a DOM event, callback assignment, or SDK readiness array?
Change listenerUser updates consentWhat event fires on consent change? Is it the same as init?

Event registration patterns

Not all CMPs use addEventListener. Fill in "Event registration mechanism" in the research template to determine which pattern applies:

PatternCMPsRegistrationCleanup
DOM eventsCookieFirst, CookiePro, CookiebotaddEventListener(name, handler)removeEventListener(name, handler)
Callback assignmentCookiePro (OptanonWrapper)window.OptanonWrapper = handlerwindow.OptanonWrapper = original
SDK readiness arrayDidomi (didomiOnReady)window.didomiOnReady.push(handler)No unsubscribe (fires once)
SDK methodDidomi (on)CMP.on('consent.changed', handler)Vendor-specific (check docs)

This affects the source skeleton (Phase 7), MockWindow shape (Phase 6), and destroy implementation.

Decision matrix

Fill in this matrix for your CMP. Reference implementations for comparison:

DecisionCookieFirstUsercentricsCookieProCookiebot
Already loaded?window.CookieFirst.consentGetter read at init (V3 __ucCmp.getConsentDetails(), V2 UC_UI.getServicesBaseInfo())window.OneTrust + window.OptanonActiveGroupswindow.Cookiebot with hasResponse
Init listenercf_init eventUC_UI_INITIALIZED event, then the getterOptanonWrapper callback (self-unwrap)None separate: the change events re-fire on every load with a stored choice
Change listenercf_consent eventUC_UI_CMP_EVENT decisions (ACCEPT_ALL, DENY_ALL, SAVE), then the getterOneTrustGroupsUpdated eventCookiebotOnAccept / CookiebotOnDecline events
Consent shapeBoolean map { category: bool }Category states (V3 categories, V2 services grouped by categorySlug)Comma-separated IDs ,C0001,C0003,Boolean map plus method and stamp
Category namingHuman-readableAdmin-configuredOpaque IDs (C0001-C0005)Fixed (necessary, preferences, statistics, marketing)
Explicit checkconsent === nullOfficial consent metadata (V3 consent.type; V2 an EXPLICIT entry in service consent history)IsAlertBoxClosed()consent.method === 'explicit'
Default categoryMapPopulated (human names to walkerOS)Empty (pass-through)Populated (opaque IDs need mapping)Populated, necessary left unmapped
Number of change events1 (cf_consent)1 (UC_UI_CMP_EVENT, which also fires for UI events that are no decision)2 (OptanonWrapper + OneTrustGroupsUpdated)2 (accept, decline), plus CookieConsentOn* aliases that are not used
Consent layersSingle (categories only)Single (categories; services are read only to group them)Single (categories only)Single (categories only)
Consent accessProperty (window.CookieFirst.consent)API method (the getter; the event carries no consent)Property (window.OptanonActiveGroups)Property (window.Cookiebot.consent)
Event registrationaddEventListeneraddEventListenerCallback assignment + addEventListeneraddEventListener

Gate: Research complete

  • All fields in the research template filled
  • Detection paths identified (which of the 3 apply)
  • Decision matrix row completed for the new CMP
  • Official docs URL captured

Phase 2: Create examples

Goal: Define realistic consent data BEFORE writing implementation.

Required example inputs

Create at minimum these scenarios in src/examples/inputs.ts. See inputs.ts for the generic template.

ExamplePurposeDescription
fullConsentAll categories acceptedUser clicked "Accept All"
partialConsentSome categories acceptedUser customized consent
minimalConsentOnly essential/necessaryUser clicked "Deny All" or similar
implicitConsentPage-load defaults (if applicable)CMP loaded but user hasn't chosen. Note: behavior varies by CMP: some grant nothing by default, others grant functional/necessary. Research the specific CMP's default consent state.
noConsentNo consent data availableCMP hasn't loaded yet
revocationInputConsent withdrawalUser goes from full to partial

Add CMP-specific edge cases:

  • Case sensitivity: Uppercase/lowercase variants of consent fields
  • Service-level consent: If CMP supports individual service booleans
  • Custom categories: Non-standard category IDs
  • Empty/malformed data: Edge cases for the consent shape

Required example outputs

Create expected walkerOS consent states in src/examples/outputs.ts. See outputs.ts for the generic template.

Key rule: denied groups MUST have explicit false, not be omitted (see mandatory check #1).

Example env

Create mock factories in src/examples/env.ts. See env.ts for the generic template.

Step examples

src/examples/step.ts turns the inputs into Flow.StepExample entries: the CMP input as in, a trigger that says how to deliver it, and the consent command the source issues as out ([['elb', 'walker consent', state]]). src/examples/trigger.ts exports the createTrigger that feeds them (see create-source, Phase 3). The docs page, walkeros push --simulate source.<id> and the step example tests read them; packages/web/sources/cmps/usercentrics/src/examples/step.ts builds its examples from its inputs.ts and outputs.ts.

Export via dev.ts

// src/dev.ts
export * as examples from './examples';
export * as schemas from './schemas';

schemas carries the settings schema (categoryMap, explicitOnly, ...): buildDev() fails the build without it.

Gate: Examples valid

  • All example files compile
  • Can trace: each input -> expected output for each example
  • Edge cases included (case sensitivity, empty data, revocation)

Phase 3: Define category mapping

Goal: Decide default categoryMap and document mapping rationale.

When to use a populated default map

Use a populated default when the CMP uses opaque or non-standard category names that are meaningless without mapping:

// CookiePro: opaque IDs require mapping
export const DEFAULT_CATEGORY_MAP: Record<string, string> = {
  C0001: 'necessary', // Strictly Necessary, grants no consent group (check 11)
  C0002: 'analytics', // Performance
  C0003: 'functional', // Functional
  C0004: 'marketing', // Targeting
  C0005: 'social', // Social Media
};

When to use an empty default map

Use an empty default when the CMP uses human-readable, admin-configured category names that can pass through as-is:

// Usercentrics: admin-configured names pass through
const settings: Settings = {
  categoryMap: config?.settings?.categoryMap ?? {},
};

Merging behavior

Custom entries merge with (and override) defaults:

const mergedCategoryMap = {
  ...DEFAULT_CATEGORY_MAP,
  ...(config?.settings?.categoryMap ?? {}),
};

Any deny denies for many-to-one mappings

When multiple CMP categories map to the same walkerOS group, the group is true only if every one of them is true: any deny denies. All CMP sources follow this rule. For a presence list (CookiePro), an inactive ID is a denial. Keep the default map one category per group, so the rule only matters for a user's own map.

// Any deny denies: once false, stays false
state[mapped] = state[mapped] === undefined ? value : state[mapped] && value;

Dual-layer consent (categories + services/vendors)

Some CMPs expose consent at multiple layers (e.g., Didomi: purposes + vendors). When the decision matrix shows "Consent layers: Dual," decide how to handle:

Option A: Map primary layer only (recommended for most cases). Use the category/purpose layer and ignore the service/vendor layer. This matches walkerOS's category-level WalkerOS.Consent model directly.

Option B: Expose both layers via settings. Add a setting like consentLayer: 'categories' | 'services' and map whichever the user chooses. Use this when the CMP's service-level consent is meaningfully different from its category-level consent.

Document the chosen approach in the README under "How it works."

Gate: Mapping defined

  • Default categoryMap chosen (populated or empty, with rationale)
  • Any-deny-denies rule for many-to-one mappings documented
  • Merge behavior with user overrides documented
  • Dual-layer strategy decided (if applicable per decision matrix)

Phase 4: Scaffold

Goal: Create package structure mirroring CookieFirst.

Directory structure

packages/web/sources/cmps/[name]/
├── package.json
├── tsconfig.json
├── tsup.config.ts
├── jest.config.mjs
├── README.md
└── src/
    ├── index.ts                    # Main source export
    ├── dev.ts                      # Dev exports (examples, schemas)
    ├── schemas/
    │   └── index.ts                # Settings schema
    ├── types/
    │   └── index.ts                # Types, Settings, CMP API interface, declare global
    ├── examples/
    │   ├── index.ts                # Re-exports
    │   ├── inputs.ts               # CMP consent input examples
    │   ├── outputs.ts              # Expected walkerOS consent outputs
    │   ├── step.ts                 # Step examples built from inputs and outputs
    │   ├── trigger.ts              # createTrigger
    │   └── env.ts                  # Mock factories
    └── __tests__/
        ├── index.test.ts           # Full test suite
        └── test-utils.ts           # createMockElb, createMockWindow, createSource

package.json template

[version] is the version every @walkeros/* package currently has (see packages/core/package.json); they are released together.

{
  "name": "@walkeros/web-source-cmp-[name]",
  "description": "[CMP Name] consent management source for walkerOS",
  "version": "[version]",
  "license": "MIT",
  "walkerOS": {
    "type": "source",
    "platform": "web",
    "docs": "https://www.walkeros.io/docs/sources/web/cmps/[name]"
  },
  "main": "./dist/index.js",
  "module": "./dist/index.mjs",
  "types": "./dist/index.d.ts",
  "exports": {
    ".": {
      "types": "./dist/index.d.ts",
      "import": "./dist/index.mjs",
      "require": "./dist/index.js"
    },
    "./examples": {
      "types": "./dist/examples/index.d.ts",
      "import": "./dist/examples/index.mjs",
      "require": "./dist/examples/index.js"
    },
    "./dev": {
      "types": "./dist/dev.d.ts",
      "import": "./dist/dev.mjs",
      "require": "./dist/dev.js"
    },
    "./walkerOS.json": "./dist/walkerOS.json"
  },
  "files": ["dist/**", "CHANGELOG.md"],
  "scripts": {
    "build": "tsup --silent",
    "clean": "rm -rf .turbo && rm -rf dist",
    "dev": "jest --watchAll --colors",
    "lint": "eslint \"**/*.ts*\"",
    "test": "jest",
    "typecheck": "tsc --noEmit",
    "update": "npx npm-check-updates -u && npm update"
  },
  "dependencies": {
    "@walkeros/core": "[version]",
    "@walkeros/collector": "[version]"
  },
  "repository": {
    "url": "git+https://github.com/elbwalker/walkerOS.git",
    "directory": "packages/web/sources/cmps/[name]"
  },
  "author": "elbwalker <hello@elbwalker.com>",
  "keywords": [
    "walker",
    "walkerOS",
    "walkerOS-source",
    "source",
    "web",
    "[name]",
    "consent",
    "cmp"
  ]
}

Config files

Copy these exactly from CookieFirst, updating only the globalName:

  • tsconfig.json - extends @walkeros/config/tsconfig/web.json
  • tsup.config.ts - uses buildModules, buildExamples, buildBrowser, buildES5 and buildDev
  • jest.config.mjs - extends @walkeros/config/jest/web.config

Phase 5: walkerOS.json convention

Add walkerOS field to package.json

{ "walkerOS": { "type": "source", "platform": "web" } }

Use buildDev() in tsup.config.ts

Use the standard buildDev() helper from @walkeros/config/tsup (consistent with create-source and create-destination skills):

import { buildDev } from '@walkeros/config/tsup';
// In defineConfig array:
buildDev(),

The CookieFirst tsup.config.ts already has it.

Hints (Optional)

If your CMP source has non-obvious behaviors or troubleshooting patterns, add hints. See walkeros-writing-documentation skill for full guidelines and walkeros-create-source skill for the hints pattern.

Gate: Convention met

  • walkerOS field in package.json with type and platform
  • Dev build configured in tsup.config.ts
  • Keywords include walkerOS and walkerOS-source

Phase 6: Test (TDD: write tests BEFORE implementation)

Goal: Write the full test suite first. Watch it fail. Then implement.

Test structure (8-9 describe blocks)

Describe blockTestsWhat it covers
initialization4-6No errors, correct type, default settings, custom settings, listener registration
explicit consent filtering3-4Explicit events processed, implicit ignored/processed based on setting, case-insensitive type
non-consent event filtering2Non-consent events ignored, events without detail ignored
category mapping5-8Full/partial/minimal consent, custom mapping, unmapped categories, any deny denies
[CMP-specific parsing]2-4CMP-specific consent format parsing (service-level, string parsing, etc.)
event handling3Consent changes, multiple changes, revocation
consent revocation2Full->partial, full->minimal (explicit false values verified)
cleanup2Destroy removes listeners, restores wrapped functions
no window environment1Handles missing window gracefully

Test utilities pattern (MockWindow)

See test-utils.ts for the complete template including MockWindow interface, createMockElb, createMockWindow, and createCmpSource factories.

For CMPs that use callback assignment or SDK methods instead of addEventListener (e.g., Didomi's onReady, CookiePro's OptanonWrapper), adapt MockWindow to expose those callbacks as testable properties.

Test template

Use the test template: index.test.ts

Gate: Tests fail for the right reason

  • Tests fail with "Cannot find module '../index'" (module doesn't exist yet)
  • Tests use example outputs for assertions (not hardcoded values)
  • Consent revocation test included (full grant -> revoke -> verify explicit false)

Phase 7: Implement

Goal: Wire up detection paths, handleConsent, and destroy. Make tests pass.

v4 source identity

CMP sources register their type literal via SourceMap in src/types.ts. CMPs are web by default - platform is optional in the augmentation:

import type { Source } from '@walkeros/core';

declare module '@walkeros/core' {
  interface SourceMap {
    // Replace with the CMP's package-level identifier.
    cookiefirst: {
      type: 'cookiefirst';
      platform?: 'web'; // optional: CMP runs on web by default
    };
  }
}

The source returns { type: 'cookiefirst', config, push, init, destroy } from Source.Init; the type literal must match the SourceMap key.

CMP sources do NOT set source.url or source.referrer. A CMP only knows about consent state, not the page that produced it; the browser, dataLayer and session sources set the page context of the events they push. Don't reach for window.location in CMP source code, leave those fields untouched.

Settings interface pattern

See types.ts for the complete type definitions including Settings, InitSettings, Types bundle, CMP API interfaces, and declare global window augmentation.

The core settings of a CMP source:

  • categoryMap?: Record<string, string>: maps CMP categories to walkerOS consent groups (every CMP source)
  • explicitOnly?: boolean: for a CMP that reports implicit or default consent before the visitor decides (Usercentrics, CookiePro, Cookiebot). Default true: the source reports only explicit choices, so a visitor the CMP never asks sends no consent; false also reports implicit or default consent. CookieFirst reports consent only after a choice (consent is null before), so its source has no such setting.
  • globalName?: string: only where the CMP lets a site rename its global; Cookiebot hard-codes window.Cookiebot, so its source has no such setting

Source skeleton

Every CMP source follows this skeleton with 5-6 decision points. See index.ts for the complete template.

Key implementation steps:

  1. Resolve window (env.window fallback to globalThis)
  2. Merge settings with defaults
  3. Track listener references for cleanup
  4. handleConsent function (explicitOnly, categoryMap with any deny denies, elb('walker consent', state))
  5. Detection path: Already loaded
  6. Detection path: Init listener
  7. Detection path: Change listener

The factory MUST be side-effect-free; do the detection paths in init(). Steps 5-7 attach listeners and perform the "already loaded" static consent read, which emits elb('walker consent', state). Put them inside an init() method on the returned instance, NOT in the factory body. The collector runs the factory in Pass 1 (before all sources are merged) and init() in Pass 2. Emitting consent from the factory races source merge order and can leave a later require:["consent"] source parked. Return { type, config, push, init, destroy }; init runs the static read + attaches listeners, destroy removes them. (The collector also matches require against current recorded state, so a consent read from init() still activates dependent steps regardless of order, but a side-effect-free factory is the contract.)

Note on init listeners: Some init listeners (like CookieFirst's cf_init) read consent from the global window object, not from event.detail. Others (like cf_consent) receive consent via event.detail. Check which pattern your CMP uses for each detection path.

Gate: All tests pass

  • npm run verify:touched -- web-source-cmp-<name> passes (L1: typecheck + lint + test; the argument is the package name after @walkeros/)
  • npm run build passes

Phase 8: Document

README structure

Keep it short and point to the docs page, like the existing CMP sources (sentence-case headings, imports in code examples):

<logo>

# @walkeros/web-source-cmp-[name]

[One line: what the source integrates and how it maps categories.]

Documentation | NPM Package | Source Code

## Installation

## Quick start (startFlow with imports)

## Documentation (link to the docs page)

## Contribute

## License

Create the docs page

Create the docs page at website/docs/sources/web/cmps/[cmp-name]/index.mdx, following the CookiePro page (website/docs/sources/web/cmps/cookiepro/index.mdx): installation with imports, the <Configuration> block fed by the package's walkerOS.json, how it works, the default category mapping, a custom mapping example and timing considerations. Then register the page:

  1. Add it to the CMPs category in website/sidebars.ts
  2. Add a row to the table in website/docs/sources/web/cmps/index.mdx, and name the source where that page and website/docs/guides/consent/index.mdx compare the CMP sources
  3. Add the package to website/package.json so the page can import its walkerOS.json

Mandatory checks

These are non-negotiable patterns every CMP source MUST follow. Violating any of these is a privacy compliance issue or a correctness bug.

1. Ensure every consent update includes explicit false for denied groups

The collector uses merge semantics (assign()), so omitting a key means "no change," NOT "denied." Every consent state passed to elb('walker consent', state) must include explicit false for denied groups, not just true for granted ones.

Boolean-map CMPs (CookieFirst, Usercentrics group-level): The CMP's consent object already contains explicit false values (e.g., { marketing: false, functional: true }). These flow through naturally via iteration, no extra code needed.

Presence-based CMPs (CookiePro): Only active groups are listed (e.g., ",C0001,C0003,"). Absence means denied. You MUST give EVERY mapped group an explicit value, true only when every ID mapped to it is active (any deny denies):

// Presence-based CMPs: walk the map, not the active IDs
const active = new Set(activeIds.map((id) => id.toLowerCase()));
Object.entries(normalizedMap).forEach(([id, group]) => {
  const granted = active.has(id);
  state[group] = state[group] === undefined ? granted : state[group] && granted;
});

2. Case-insensitive comparison for all CMP string fields

Use .toLowerCase(). CMP docs are inconsistent on casing.

// Explicit type check
if (settings.explicitOnly && detail.type?.toLowerCase() !== 'explicit') return;

// Category ID lookup
const mapped = normalizedMap[id.toLowerCase()];

3. Apply categoryMap consistently in ALL code paths

If there are multiple parsing branches (group-level vs service-level, or already-loaded vs event-listener), mapping MUST work identically in each.

4. Prevent dual-firing on consent events

Many CMPs fire multiple signals for the same consent action. If you don't guard against this, handleConsent fires twice per user action. Three known patterns:

The collector's exactly-once delivery does not solve this for you: it guarantees each subscriber sees a given state version once, but two upstream CMP signals (a callback plus a DOM event) are two distinct consent commands, so both are delivered legitimately. Coalescing duplicate upstream signals is the source's job, as below.

Pattern A: Callback + event (CookiePro) CMP fires both a callback (OptanonWrapper) and a DOM event (OneTrustGroupsUpdated) on the same action. Use the callback for init only and self-unwrap after first call:

actualWindow.OptanonWrapper = () => {
  if (originalWrapper) originalWrapper();
  handleConsent();
  actualWindow.OptanonWrapper = originalWrapper; // Self-unwrap
};

Pattern B: Change events that re-fire on load (Cookiebot) CMP fires one of several events per decision (CookiebotOnAccept, CookiebotOnDecline), and fires it again on every page load with a stored choice. Register the SAME handler for all of them, and listen to one name set only: Cookiebot also dispatches each event under a CookieConsentOn* alias. Because the static read at init and the load-time event report the same choice, drop a state identical to the last one the source sent.

Pattern C: One event bus for every UI event (Usercentrics) Usercentrics fires UC_UI_CMP_EVENT for decisions and for UI events such as a view change. Handle only the decision types (ACCEPT_ALL, DENY_ALL, SAVE) from the event detail, and read the consent from the getter (the event carries none). A view toggle then publishes nothing.

Research step: During Phase 1, fill in "Number of change events" in the decision matrix. If >1, determine which pattern applies and plan accordingly.

5. Document timing/race conditions in the README

What if the CMP loads before the source? What about explicitOnly: false? Each CMP has different timing behavior: document it explicitly under "Timing considerations."

For "previous-choice" detection (was this an active user decision or a first-visit default?), read the CMP's official consent metadata, the consent type or decision history (e.g. Usercentrics V3 consent.type, V2 the service consent history), not a per-pageload event type field. A per-pageload event type conflates first-visit defaults with a returning visitor's restored choice, so a returning visitor's prior consent would be dropped under explicitOnly.

6. Test consent revocation end-to-end

Full grant -> revoke -> verify denied (explicit false values). This is the most common source of bugs.

test('handles consent withdrawal (full to partial)', async () => {
  const mockWindow = createMockWindow(inputs.fullConsent);
  await createCmpSource(mockWindow, mockElb);

  // Initial: full consent, read at init
  expect(consentCalls[0].consent).toEqual(outputs.fullConsentMapped);

  // User revokes marketing and analytics
  mockWindow.__dispatchEvent('cmp_consent', inputs.revocationInput);
  expect(consentCalls[1].consent.marketing).toBe(false); // explicit false
  expect(consentCalls[1].consent.analytics).toBe(false); // explicit false
});

This is the revocation test of the template (index.test.ts); cmp_consent stands for your CMP's change event.

7. Use MockWindow interface in tests

Properly typed, not as unknown as casts scattered through tests. Define one MockWindow interface in test-utils.ts that extends the source's narrowed CmpWindow (the surface the source actually touches), not the global Window. Because Env.window is narrowed to CmpWindow (as the template and the Cookiebot source do), the mock literal satisfies it directly, with no as unknown as Window cast.

8. Store category keys in user-expected format

Normalize during init for case-insensitive lookup, but store the original keys in the config so users see what they configured.

// Store original casing in config
const mergedCategoryMap = { ...DEFAULT_CATEGORY_MAP, ...userMap };

// Build normalized lookup for internal use
const normalizedMap: Record<string, string> = {};
Object.entries(mergedCategoryMap).forEach(([key, value]) => {
  normalizedMap[key.toLowerCase()] = value;
});

9. walkerOS.json convention

Add "walkerOS": { "type": "source", "platform": "web" } to package.json.

10. README requirements

Keep the README short, as the existing CMP sources do: logo, one-line description, Documentation/NPM/Source Code links, installation, a quick start with imports, a link to the docs page, contribute and license sections, sentence case headings. Configuration, mapping, how it works and timing considerations belong on the docs page (see "Create the docs page").

11. Strictly necessary cookies grant no consent group

Strictly necessary cookies need no consent, and the CMP grants them with every choice. Never map such a category (necessary, C0001, an essential category) to a consent group like functional in the default categoryMap: as the only category of that group, it would grant the group with every choice, and every destination that requires the group would act on consent the visitor never gave. Report it under its own name (necessary) or leave it out, and cover it with a test: a choice that grants only strictly necessary cookies leaves every group except necessary false.


Validation checklist

Beyond understanding-development requirements (build, test, lint, no any):

  • All 11 mandatory checks pass
  • Research template fully filled
  • Decision matrix row complete
  • Examples include revocation and edge cases
  • Tests: 25-32 tests across 8-9 describe blocks
  • MockWindow interface in test-utils (not scattered casts)
  • Category mapping: any deny denies for many-to-one
  • Consent state always includes explicit false for denied groups
  • destroy() cleans up ALL listeners and restores wrapped functions
  • Detection never fails silently: a setting the source needs that is missing throws a StepError (code: 'config') from init, and init never returns early or returns false; a CMP that is simply absent is not an error (consent stays unknown)
  • README follows required structure
  • Docs page created and registered (Phase 8)

Known limitations

The skill's source skeleton and code templates are based on DOM-event CMPs (CookieFirst, Usercentrics, CookiePro). CMPs that deviate significantly from this pattern will require adaptation:

LimitationAffected CMPsWorkaround
Source skeleton assumes addEventListenerDidomi (SDK array/method)Use the "Event registration patterns" table in Phase 1 to identify the correct pattern, then adapt the skeleton's listener setup and destroy() accordingly.
Consent read via property, not API methodDidomi (getCurrentUserStatus())If consent is accessed via an API method call rather than a window property, wrap the call in handleConsent and adjust the "already loaded" detection path.
destroy() may not be possibleCMPs with SDK readiness arrays (fire-once, no unsubscribe)Document in README that the init callback cannot be removed. Only the change listener needs cleanup.
No vendor-level consent modelDidomi (purposes + vendors as separate consent layers)Use the "Dual-layer consent" guidance in Phase 3. walkerOS Consent is category-level; vendor-level consent requires flattening or a consentLayer setting.

These are research-phase decisions; the skill's phases, mandatory checks, and validation checklist still apply. The research template and decision matrix capture these variations so they are identified early.


Reference files

WhatWhere
Skill examplesexamples/
Skill templatestemplates/cmp/
Canonical templatepackages/web/sources/cmps/cookiefirst/
Source typespackages/core/src/types/source.ts
Consent guidewebsite/docs/guides/consent/
Usercentrics sourcepackages/web/sources/cmps/usercentrics/
CookiePro sourcepackages/web/sources/cmps/cookiepro/

Related skills

Create a New Destination

Prerequisites

Before starting, read these skills:

Choose Your Template

ComplexityTemplateWhen to Use
Simpleplausible/Single SDK call, minimal config
Complexgtag/Multiple services, sub-destinations
Servergcp/Server-side, batching, SDK init

Process Overview

1. Research     → Deeply understand vendor SDK, API, and event taxonomy
2. Classify     → Determine vendor taxonomy type and integration approach
3. Examples     → Define in/out pairs FIRST (start with the end result)
4. Mapping      → Define walkerOS → vendor transformation
5. Scaffold     → Copy template and configure
6. Convention   → Add walkerOS.json metadata and buildDev
7. Implement    → Build using examples as test fixtures
8. Test         → Verify against example variations
9. Document     → Write README

Phase 1: Research

Goal: Deeply understand the vendor SDK before writing any code. Research quality determines implementation quality.

1.1 Find and Install Official SDK

Always prefer the vendor's official SDK package over raw HTTP API calls. The SDK handles transport, batching, retries, and plugin ecosystems; don't reinvent these.

  • Install the SDK: npm install @vendor/sdk and read the actual source
  • Read TypeScript types: Import types from the SDK package directly. Never duplicate type definitions. This ensures IntelliSense completeness and consistency with SDK updates.
  • Understand the full API surface: List every public method, not just track(). Identity methods, property operations, group management, specialized event types (revenue, etc.)
# Search npm for official packages
npm search [vendor-name]
npm search @[vendor]

# Install and inspect actual types
npm install @vendor/analytics-browser
ls node_modules/@vendor/analytics-browser/lib/esm/

1.2 Understand SDK Architecture

  • Init options: What does init() accept? What can be configured?
  • Call ordering: Can you call methods before init()? Is there internal queuing? What are the race condition implications?
  • Plugin system: Does the SDK support plugins? How are they added?
  • Identity management: How does the SDK handle user/device/session IDs? Does it manage cookies/storage? Can this be disabled (since walkerOS manages identity)?
  • Consent: Does the SDK have a consent mode or just binary opt-out?

1.3 Identify All Event Methods

Go beyond just track(). Most SDKs have specialized methods:

Method CategoryExampleswalkerOS Handling
Event trackingtrack(), logEvent()Default push()
User propertiesidentify(), setUserProperties()mapping.settings
Revenue/purchaserevenue(), purchase()mapping.settings
Groups/accountssetGroup(), groupIdentify()mapping.settings
Identity setterssetUserId(), setDeviceId()settings.identify
Opt-outsetOptOut(), consent()on('consent')
Cleanupflush(), reset()destroy()

1.4 Check Existing Patterns

Review similar destinations in the codebase:

ls packages/web/destinations/

Gate: Research Complete

  • Official SDK installed and types inspected
  • All public methods listed with their purpose
  • Init options understood (what to disable, what to pass through)
  • Call ordering / race conditions understood
  • Identity, consent, and plugin patterns documented

Checkpoint: Research Review (Optional)

If working with human oversight, pause here to confirm:

  • SDK integration approach correct?
  • Event mapping makes sense for the use case?
  • Any vendor quirks or limitations to handle?

Phase 2: Classify Vendor Taxonomy

Goal: Understand what the vendor expects, which determines destination complexity.

2.1 Taxonomy Types

TypeDescriptionMapping NeededExample Vendors
Free-formAny event name accepted, no prescribed schemaMinimal: names pass throughMost modern analytics
RigidPrescribed event names unlock specific reportsEssential: must map to exact namesSome legacy analytics
Schema-basedSelf-describing events with formal schemasStructural: must build schema objectsData warehouse tools

2.2 Determine Integration Approach

ApproachWhen to usePattern
Vendor SDK as hostSDK has plugins, batching, retriesLoad SDK, disable what walkerOS replaces, use as transport
Script + command queueSDK loaded via script tagLoad script, use global function queue
HTTP APINo SDK available, or server-sideDirect HTTP calls via sendWeb() or fetch

Prefer the vendor SDK: it handles transport, retries, and plugin orchestration. HTTP API is a fallback when no SDK exists.

When using the vendor SDK:

  • Disable features walkerOS replaces (autocapture, identity storage) by default, but allow users to re-enable via settings
  • Pass through all SDK init options for full configurability
  • Import types from the SDK package directly

Gate: Classification Complete

  • Taxonomy type identified (free-form / rigid / schema-based)
  • Integration approach chosen (SDK / script / HTTP)
  • Know which SDK features walkerOS replaces vs. uses

Phase 3: Create Examples (BEFORE Implementation)

Mandatory. Examples are the test fixtures for Phase 8. Define expected in / out / mapping triples FIRST: start with the end result in mind. Without examples, you cannot test. Even for free-form vendors where no mapping is strictly "required," step examples serve as the single source of truth for tests, simulations, and documentation.

Authoritative pattern: See using-step-examples for the Three Type Zones and lifecycle. This skill reuses that contract; do not diverge.

3.1 Scaffold Directory Structure

mkdir -p packages/web/destinations/[name]/src/examples
mkdir -p packages/web/destinations/[name]/src/{schemas,types}

3.2 Required Files (4 files total)

The reference web destinations (meta, snowplow, plausible, piwikpro, api) use exactly four files in src/examples/. Match this structure: no events.ts, outputs.ts, or standalone mapping.ts.

FilePurpose
examples/env.tsMock environment for testing (no real network calls)
examples/step.tsFlow.StepExample entries with in / out / mapping? triples
examples/minimal.tsFlow.MinimalExample: only the settings it needs to start
examples/index.tsBarrel exports: env, step and minimal

The step.ts file embeds the input event, the mapping config, and the expected vendor output together in one Flow.StepExample.

The skill's examples/ folder holds these four files for the template destination (templates/simple/): copy them to src/examples/.

3.3 Typing Rules (strict)

No any. Every example value must be explicitly typed.

  • Inputs use WalkerOS.Event (via getEvent() from @walkeros/core); never hand-roll event literals.
  • Outputs use vendor SDK types imported from the official package whenever the vendor publishes them. Do not invent local output types for payloads the vendor already types (e.g. use Meta Pixel's fbq argument types, not a local FbqCall interface).
  • Step entries are typed Flow.StepExample from @walkeros/core.
  • Mock env is typed against the destination's local Env type from ../types. No as any, no untyped {}.
  • Vendor SDK types come from the SDK you installed in Phase 1: reuse them rather than duplicating shapes.

3.4 Code Template: examples/step.ts

The examples match the template destination (index.ts), which calls a vendor global window.vendorSdk('track', name, params):

import type { Flow } from '@walkeros/core';
import { getEvent } from '@walkeros/core';

// One step example per supported feature / setting.
// `in`  is a WalkerOS.Event (getEvent returns the same event on every run).
// `out` lists the vendor calls the destination makes, as [callable, ...args]
//       tuples written from the vendor's documented calls.
// `mapping` is the mapping rule under test (optional: omit it for the default
//       push).

// Set `title` + `description` for public examples; mark test-only fixtures
// with `public: false`. See the walkeros-using-step-examples skill.

export const init: Flow.StepExample = {
  title: 'Initialization',
  description:
    'The destination loads the vendor script and initializes the SDK with the API key.',
  in: { loadScript: true, settings: { apiKey: 'YOUR_API_KEY' } },
  out: [['window.vendorSdk', 'init', 'YOUR_API_KEY']],
};

export const purchase: Flow.StepExample = {
  title: 'Purchase',
  description:
    'A completed order is sent as a purchase with its id, value and currency.',
  in: getEvent('order complete'),
  mapping: {
    name: 'purchase',
    data: {
      map: {
        transaction_id: 'data.id',
        value: 'data.value',
        currency: { key: 'data.currency', value: 'EUR' },
      },
    },
  },
  out: [
    [
      'window.vendorSdk',
      'track',
      'purchase',
      { transaction_id: '0rd3r1d', value: 462, currency: 'EUR' },
    ],
  ],
};

export const pageView: Flow.StepExample = {
  title: 'Page view',
  description: 'A page view without a rule is sent under its walkerOS name.',
  in: getEvent('page view'),
  out: [['window.vendorSdk', 'track', 'page view', {}]],
};

A consent example sets command: 'consent', which routes in through elb('walker consent', in) instead of an event push (see consentRevoke below).

Init step example

A destination whose init makes vendor calls (script tags, SDK initializers, queue setup) can ship them as an examples.step.init entry, as plausible, meta and snowplow do.

  • in mirrors the real Destination.Config shape users copy-paste, typically { loadScript: true, settings: { /* vendor-specific */ } }. Whatever a user would configure in their flow goes here verbatim.
  • out is the ordered list of vendor calls the init() lifecycle produces. Each effect tuple follows the standard [callable, ...args] shape.

runExamples keeps init calls out of every event example's out and takes events only, so the init example runs in a test of its own: start a real collector, call destination.init(...) with an env that records the paths examples.env.simulation names, and compare the calls with init.out. The template index.test.ts does this, as packages/web/destinations/meta/src/__tests__/conformance.test.ts does.

Multi-tool packages (like gtag, which drives GA4, Google Ads, and GTM) ship per-tool init examples (examples.step.ga4Init, adsInit, gtmInit) instead of a single init, and their tests run each with its sub-tool's settings.

Failure, silent and consent examples

Besides the happy path, step.ts carries public: false examples that the conformance kit and runExamples (Phase 8) run on a real collector. The first and the last fit the template destination; the two in between show failures other destinations report:

// A silent rule sends nothing.
export const silentRule: Flow.StepExample = {
  public: false,
  description: 'A rule with silent: true sends no event.',
  in: getEvent('page view'),
  mapping: { silent: true },
  out: [],
};

// A destination that needs a user id reports an event without one.
export const missingInput: Flow.StepExample = {
  public: false,
  description: 'Without a user id the push fails with missing-input.',
  in: getEvent('page view', { user: { device: 'c00k13' } }),
  mapping: { settings: { identify: { map: { id: 'user.id' } } } },
  out: [],
  outcome: { status: 'failed', code: 'missing-input', retryable: false },
};

// An HTTP destination whose vendor answers 200 with an error in the body.
export const rejectedByVendor: Flow.StepExample = {
  public: false,
  description: 'The vendor answers 200 and reports the event as failed.',
  in: getEvent('page view'),
  respond: { status: 200, body: { events: [{ status: 'failed' }] } },
  outcome: { status: 'failed', code: 'transport', retryable: false },
};

// A consent revocation reaches the vendor.
export const consentRevoke: Flow.StepExample = {
  public: false,
  description:
    'For a destination with config.consent { marketing: true }, revoking marketing tells the vendor to revoke.',
  command: 'consent',
  in: { marketing: false },
  out: [['window.vendorSdk', 'consent', 'revoke']],
};

outcome is { status: 'failed' | 'queued', code, retryable? }; respond is what the vendor's boundary answers (status, body, or error: 'network' | 'timeout'). An example that expects no vendor call has out: []. A "one of these settings" rule (ga4, ads or gtm) is stated as anyOf in the settings schema, not only in init.

For destinations, the Three Type Zones collapse to:

  • in = walkerOS event (WalkerOS.Event)
  • out = vendor output (typed against vendor SDK)
  • mapping = rule under test (optional)

3.5 examples/index.ts (barrel)

export * as env from './env';
export * as step from './step';
export { minimal } from './minimal';

3.6 examples/env.ts

Mock the vendor SDK surface and any DOM touchpoints. Never reach real network, real cookies, or real globals. Type the exports against your local Env, and name the calls a simulation records in simulation:

import { mockScriptDocument } from '@walkeros/web-core/dev';
import type { Env } from '../types';

const noop = () => {};

/** Before init: no SDK global yet, `init` installs the stub. */
export const init: Env = {
  window: {},
  document: mockScriptDocument(),
};

/** After init: the SDK global the push calls. */
export const push: Env = {
  window: { vendorSdk: Object.assign(noop, { q: [] as IArguments[] }) },
  document: mockScriptDocument(),
};

/** The calls a simulation records. */
export const simulation = ['call:window.vendorSdk'];

3.7 examples/minimal.ts

The minimal example is the setup a reader copies: the docs page renders its install line, integrated code and flow.json under ## Installation from it (<Quickstart data={data} />), and the conformance test runs exactly those settings and requires one public example delivered.

import type { Flow } from '@walkeros/core';

/** The settings the vendor needs to deliver: the API key, with the vendor script loaded by the destination. */
export const minimal: Flow.MinimalExample = {
  settings: { apiKey: 'YOUR_API_KEY' },
  config: { loadScript: true },
};
  • settings: only the keys the settings schema requires, plus any that init refuses to start without under the example's config (Plausible's domain with loadScript: true), with placeholder values. Other keys stay in the step examples that need them; the kit fails a minimal example that sets one.
  • config: only { loadScript: true }, for a web destination that loads its vendor script only when asked to; the kit fails a script-loaded destination without it.
  • secrets (server destinations): each credential setting mapped to its secret name, <VENDOR>_<SETTING> (for example { accessToken: 'META_ACCESS_TOKEN' }). The generated code reads process.env.META_ACCESS_TOKEN and $secret.META_ACCESS_TOKEN; a web destination holds no secret.
  • A package with several exports keeps one per export, in the export's folder (src/firehose/examples/minimal.ts), listed in exportExamples.

A package that cannot express its setup in one minimal example (one package serving several docs pages, an install line with a driver choice) passes withoutMinimal: '<reason>' to conformance instead, and its page keeps a hand-written setup.

3.8 Test Fixture Contract (hard rule)

The examples authored here are the Phase 8 test fixtures. No parallel fixtures allowed.

  • Every event example runs through runExamples from @walkeros/collector/dev with claims (the standard events the destination maps): it runs each one on a real collector as walkerOS.json publishes it and compares its out and its outcome. Do not hand-roll that loop.
  • Tests must NOT contain hardcoded payloads, vendor configs, or expected outputs.
  • If a test needs a value that is not in examples.step, add it to step.ts first, then consume it from the test. Never inline test data.
  • The only per-test setup allowed is the settings a group of examples needs (e.g. enabling the right sub-tool).

See packages/web/destinations/meta/src/__tests__/conformance.test.ts for a canonical reference.

3.9 Export via dev.ts

export * as schemas from './schemas';
export * as examples from './examples';

Phase 3 Acceptance Checklist

  • src/examples/env.ts: mock env, no real network, typed against local Env
  • src/examples/step.ts: one Flow.StepExample per supported feature / setting, typed in / out / mapping?
  • src/examples/minimal.ts: Flow.MinimalExample with only the settings the schema requires or init refuses to start without (secrets for server credentials, config.loadScript for a script-loaded web destination)
  • src/examples/index.ts: barrel exports env, step and minimal
  • No standalone events.ts, outputs.ts, or mapping.ts files
  • All vendor SDK types imported from the official package, no any, no reinvented local output types
  • src/__tests__/conformance.test.ts runs the event examples through runExamples with claims, and a test runs the init example (if any)
  • Tests contain zero hardcoded payloads / vendor configs / expected outputs: everything flows from examples.step
  • npm run build passes: examples compile against published types
  • Each example traces: in → apply mapping → matches out

Phase 4: Define Mapping

Goal: Document transformation from walkerOS events to vendor format.

Mapping rules live inside each Flow.StepExample entry in step.ts, no separate mapping.ts file. Each step example embeds the exact mapping rule under test alongside its in event and expected out output.

Verify Mapping Logic

For each entry in step.ts, trace:

Input: examples.step.purchase.in     (WalkerOS.Event)
  ↓ Apply examples.step.purchase.mapping
  ↓ name transforms, data.map applied
Output: Should match examples.step.purchase.out

Gate: Mapping Verified

  • Step examples cover: page view + at least one conversion event
  • One step example per supported setting / sub-tool
  • Each mapping traces correctly from in to out

Phase 5: Scaffold

Template destination: packages/web/destinations/plausible/

cp -r packages/web/destinations/plausible packages/web/destinations/[name]
cd packages/web/destinations/[name]

# Update package.json: name, description, repository.directory

Directory structure:

packages/web/destinations/[name]/
├── src/
│   ├── index.ts           # Main destination (init + push)
│   ├── index.test.ts      # The init example and other boundary tests
│   ├── __tests__/         # conformance.test.ts: conformance + runExamples
│   ├── dev.ts             # Exports schemas and examples
│   ├── examples/          # env.ts, step.ts, minimal.ts, index.ts
│   ├── schemas/
│   └── types/
├── package.json
├── tsconfig.json
├── tsup.config.ts
├── jest.config.mjs
└── README.md

Transformer Chain Integration

Destinations can wire to transformer chains via before in the init config:

destinations: {
  myDestination: {
    code: destinationMyDestination,
    config: { settings: { /* ... */ } },
    before: 'redact'  // Events go through redactor before this destination
  }
}

Phase 6: walkerOS.json Convention

Every walkerOS package ships a walkerOS.json file for CDN-based schema discovery.

Add walkerOS field to package.json

{
  "walkerOS": { "type": "destination", "platform": "web" },
  "keywords": ["walkerOS", "walkerOS-destination", ...]
}

Use buildDev() in tsup.config.ts

Keep buildDev() in the defineConfig array (the template package's tsup.config.ts has it):

import { buildDev } from '@walkeros/config/tsup';
// In defineConfig array:
buildDev(),

It builds src/dev.ts and writes dist/walkerOS.json from the schemas, examples and hints it exports.

Hints (Optional)

If your destination has capabilities, behaviors, or troubleshooting patterns not obvious from schemas alone, add hints. See walkeros-writing-documentation skill for full guidelines.

Create src/hints.ts:

import type { Hint } from '@walkeros/core';

export const hints: Hint.Hints = {
  'auth-methods': {
    text: 'Supports X, Y, and Z auth methods. See settings schema for all options.',
    code: [{ lang: 'json', code: '{ "settings": { ... } }' }],
  },
};

Export from src/dev.ts:

export * as schemas from './schemas';
export * as examples from './examples';
export { hints } from './hints';

Guidelines:

  • Expand awareness: describe capabilities ("supports X, Y, Z"), don't prescribe one path
  • Reference schemas and examples, don't duplicate them
  • Verify every claim against actual implementation before publishing
  • Key naming: kebab-case, group with prefixes (auth-*, storage-*, troubleshoot-*)
  • Most destinations don't need hints; schemas and examples cover the common case

Gate: Convention Met

  • walkerOS field in package.json with type and platform
  • buildDev() in tsup.config.ts
  • Build generates dist/walkerOS.json
  • Keywords include walkerOS and walkerOS-destination

Phase 7: Implement

Now write code to produce the outputs defined in Phase 3.

Template Files

Use these templates as your starting point:

FilePurposeTemplate
types/index.tsType definitionstypes.ts
index.tsMain destinationindex.ts
index.test.tsInit example testindex.test.ts
__tests__/conformance.test.tsFailure contract, all examplesconformance.test.ts

Key Patterns

  1. Init receives context: Destructure config, env, logger, id from context

  2. Push receives context: Includes data (the mapped data), rule (the matching mapping rule), ingest

  3. Use getEnv<Env>(env): Never access window/document directly, and never cast them. Declare your SDK global (declare global { interface Window { vendorSdk?: ... } }) and narrow Env['window'] to the SDK shape in types.ts. The generic getEnv<Env>(env) then returns your narrowed env intersected with the real DOM globals, so window.vendorSdk and document.createElement are fully typed with zero casts. No as any, as unknown, as Window, as Document, @ts-ignore, or @ts-expect-error is allowed in production or test code, ever. If something will not type check, fix the types, do not cast.

  4. Return config from init: Allows updating config during initialization

  5. Optional destroy method: Implement if the destination holds resources (DB connections, SDK clients, timers) that need cleanup on shutdown. Call flush() or equivalent on the vendor SDK.

  6. Mapping-based settings (recommended): Use standard walkerOS mapping values (map, loop, key, value, condition) in mapping.settings.* for vendor-specific operations. Resolve via getMappingValue(event, value, { collector }) in push(), with the collector from the push context, and interpret the resolved object's keys as SDK method instructions. This keeps config agnostic and reuses the mapping engine.

  7. Consent two-layer: config.consent gates walkerOS event delivery. on('consent') controls vendor SDK internals (opt-out, pause capture, etc.). Both needed for complete consent compliance. on() may be async: the collector awaits it, bounded by config.timeout, and delivers the current state before the destination's first push; events pushed meanwhile answer queued with the reason held. A state handler that rejects or does not settle in that time writes an ERROR line (consent not applied) and holds the destination's events for one more config.timeout. The first push after that window retries a delivery whose handler rejected, and once a delivery succeeds the held events are delivered. Until one has succeeded, the held events and every later one fail with blocked (the push answers ok: false, the events go to the dead-letter queue, an ERROR Push failed line is written) and are not sent when the destination recovers; later pushes retry a rejected delivery once per config.timeout. A handler that timed out gets no new delivery until it settles, so one that never settles keeps the destination blocked. A walker run discards events that are still held, with no counter and no log line. Lifecycle handlers such as on('run') are awaited too, bounded by the same timeout, and never hold events. elb commands and startFlow resolve only after the destinations' on() handlers for that command have settled or timed out. See consent for the full sequence.

    • For step-example tests, use command: 'consent' on Flow.StepExample to invoke the on('consent') handler. Do not push consent data as an event.
  8. A failure is a throw (the failure contract). A push or init that returns normally counts as delivered, so:

    • Throw a StepError from @walkeros/core with a code (config for a missing or invalid setting, missing-input for an event the destination cannot send, transport for a failed request; add status for an HTTP status, retryable to override the default). Never return early, never return false from init, never log the failure yourself: the collector counts the push as failed, keeps a DLQ entry and writes one ERROR line per cause (repeats within a minute are counted, not logged again).
    • Await every call the vendor makes. A call that is not awaited fails after the push was already counted as delivered. The lint rules walkeros/await-send and walkeros/no-silent-return flag both.
    • A response body goes on the error's body, never into its message; messages and log lines never carry event data.
    • A value-level problem (one mapped field cannot be built) drops that field, sends the event and reports once with reportCause. Only a wholly unsendable event throws missing-input.
  9. Scripts through loadVendor (@walkeros/web-core). It injects the script once per page into the document env provides (the page's own by default) and reports vendor-not-loaded, vendor-not-ready and vendor-timeout. For tests and examples, mockScriptDocument() from @walkeros/web-core/dev provides a document. For a vendor that replays its own stub, call check() in push; for any other, await ready(). Pass attributes the tag needs (data-domain) through options.attributes.

  10. Honour rule.silent. A rule with silent: true sends no event of its own and runs only the side effects it maps (identify, group). Do not return early for it (walkeros/no-silent-return is an error): write push = async (event, context) => { if (!context.rule?.silent) await send(event, context); } and run the side effects around it. A silent rule whose side-effect mapping resolves to nothing sends nothing, counts as delivered and writes one DEBUG line (<Vendor> skipped <call>: <mappingKey> resolved to nothing, the event name, never the value). Every destination ships a silent step example (see Phase 3).

  11. Consent reaches the vendor. config.consent decides: in on('consent'), tell the vendor about a revocation and a grant when it has a switch (getGrantedConsent(config.consent, collector.consent)), and throw a consent-not-applied StepError when the vendor does not take it.

  12. Batching. A server SDK that batches implements pushBatch and ships a default config.batch in the destination's own config (the server segment destination: { size: 15, wait: 10_000 }); each entry has its own outcome, and config.batch: { size: 1 } is the per-event setting. See batch scheduling for size, wait and age.

Reserved fields

config.setup is reserved for the setup lifecycle (see Adding setup (optional)). Do not use the key setup for unrelated package metadata or hint keys. The framework wires this field to SetupFn via resolveSetup, and walkeros setup destination.<name> reads it. Repurposing the name will collide with that wiring.

Gate: Implementation Compiles

  • npm run build passes
  • npm run verify:touched -- <platform>-destination-<name> passes (L1: typecheck + lint + test; the argument is the package name after @walkeros/)

Adding setup (optional)

A destination package can implement an optional setup() function to provision external resources idempotently: BigQuery datasets and tables, Pub/Sub topics, SQLite tables, warehouse schemas, S3 buckets, webhook registrations on downstream platforms. Setup runs only when an operator explicitly types walkeros setup destination.<name>. The runtime never auto-invokes it from init(), push(), or destroy().

The framework provides the slot, the CLI command, and a resolveSetup helper. The package owns: what setup means, idempotency, error handling, return value.

For background on how setup fits the destination lifecycle, see understanding-destinations.

Types

// types/index.ts
import type {
  Credential,
  Destination as CoreDestination,
  ServiceAccount,
} from '@walkeros/core';

export interface Settings {
  /* runtime push settings */
}
export interface InitSettings {
  /* one-time init settings */
}
export interface Mapping {
  /* per-event mapping config */
}
export interface Env {
  /* injected platform deps (SDK clients, etc.) */
}

// The package's own setup options interface.
// Becomes the U slot of Types; surfaces as `config.setup: boolean | Setup` for users.
export interface Setup {
  // package-specific provisioning options
  // e.g. for BigQuery: location, partitioning, clustering, schema
  location?: string;
  partitioning?: { field: string; type: 'DAY' | 'HOUR' };
}

// Service-account / key-pair credentials. Becomes the trailing C slot of Types;
// surfaces as the optional top-level `config.credentials` (sibling of `settings`),
// set with a `$secret.` reference and validated per package.
export type Credentials = Credential<ServiceAccount>;

export type Types = CoreDestination.Types<
  Settings,
  Mapping,
  Env,
  InitSettings,
  Setup,
  Credentials
>;

Where credentials belong. Put service-account / key-pair credentials (the kind paired with $secret.NAME to inject a secret) on the strictly-typed top-level config.credentials, not inside settings. Read them from config.credentials, and fall back to Application Default Credentials when they are omitted. The raw settings.<sdk> passthrough (e.g. settings.bigquery) stays as the lower-level escape hatch for SDK-specific auth options.

Implementation

// setup.ts
import type { Destination as CoreDestination, SetupFn } from '@walkeros/core';
import { resolveSetup } from '@walkeros/core';
import type { Setup, Types } from './types';

const DEFAULT_SETUP: Setup = {
  location: 'EU',
};

export const setup: SetupFn<
  CoreDestination.Config<Types>,
  CoreDestination.Env<Types>
> = async ({ config, env, logger }) => {
  const options = resolveSetup(config.setup, DEFAULT_SETUP);
  if (!options) return; // config.setup is false or unset

  // Package-specific provisioning, idempotent.
  // Returning a structured object (e.g. { datasetCreated: true })
  // makes that data available to operators via `walkeros setup ... | jq`.
};

Wire it in your default export:

// index.ts
import type { Destination as CoreDestination } from '@walkeros/core';
import { setup } from './setup';
import type { Types } from './types';

export const destinationMine: CoreDestination.Instance<Types> = {
  type: 'my-destination',
  config: {},
  async push(event, context) {
    // Send the event (Phase 7)
  },
  setup,
};

export default destinationMine;

When to implement

Implement setup() when your destination needs first-time provisioning of external resources before events can flow: warehouse tables, Pub/Sub topics, S3 buckets, schema bindings, IAM roles, webhook registrations on downstream platforms. Skip it when your destination only consumes credentials or HTTP endpoints the user already provisioned.

Contract

  • Triggered only by walkeros setup <kind>.<name>. Never by runtime push, init, or destroy.
  • Idempotency is your responsibility. Re-running setup against a fully provisioned environment must be a safe no-op. Use try-create-catch-409 on REST APIs, IF NOT EXISTS on SQL, native idempotent operations where available. The framework does not retry, track state, or detect drift.
  • Return structured data from setup() when useful for operator scripting. The CLI emits non-undefined return values as JSON to stdout.
  • For packages where setup: true (boolean form) is meaningless because mandatory fields have no safe defaults (e.g., Kafka numPartitions, GitHub webhook webhookUrl), reject the boolean form with a clear runtime error listing required fields:
if (config.setup === true) {
  throw new Error(
    'kafka destination setup requires explicit options: ' +
      '{ topic, numPartitions, replicationFactor }. There is no safe default.',
  );
}

Phase 8: Test Against Examples

Tests verify implementation against the examples from Phase 3. If examples are incomplete, tests will be incomplete.

See testing-strategy for the shared env / dev-examples conventions this phase depends on.

Verify implementation produces expected outputs.

Test Templates

Use the test templates: conformance.test.ts (the conformance kit and every event example) and index.test.ts (the init example). Reference canonical implementation: packages/web/destinations/meta/src/__tests__/conformance.test.ts.

Conformance test

Every destination has a conformance test that runs it on a real collector with conformance('destination', destination, options) from @walkeros/collector/dev (template: conformance.test.ts). It checks the failure contract: a failing send is reported with a cause, a failure never leaks event data into a log line, a silent example sends nothing, a consent revocation reaches the vendor, init throws on a missing setting, and every event the destination cannot send has a declared outcome. It fails with every broken item at once.

conformance does not compare a failure example's outcome with what the push reported. The same test file therefore runs the event examples through runExamples with claims, which compares each example's out and outcome: a failure example with a wrong outcome fails the test and is not published under "When it fails".

It also runs the minimal example: pass minimal: examples.minimal (the template does), and the kit starts the destination with the settings its docs quickstart shows and requires one public example delivered. A package without src/examples/minimal.ts passes withoutMinimal: '<reason>' instead. A test with neither, with both, or with an option that disagrees with the file fails and says which option to pass, so a copied template without minimal fails.

The profile option says what the destination talks to: http (calls fetch), sdk-env (a server SDK injected through env), web-sdk (a web SDK started in init: sdk-env without the init check), vendor-global (a vendor script that defines a global), socket (a long-lived connection). The browser profiles (web-sdk, vendor-global) also take page from @walkeros/web-core/dev (import it after the @walkeros/collector/dev import, pass it right after profile); without it the test throws profile <name> runs on a page: pass page from @walkeros/web-core/dev. Server destinations pass no page. Never prove delivery or loss with a mocked collector: startFlow and mock only the network or the vendor global.

Key Test Patterns

  1. runExamples runs every event example on a real collector, with claims listing the standard events the destination maps. Do not write per-feature tests with hand-rolled payloads.
  2. The init example runs on its own: its in is a config, not an event.
  3. Zero hardcoded payloads: every input, vendor config, and expected output comes from examples.step or examples.env. If you need something new, add it to examples first.
  4. Clone env per test: clone(examples.env.push) so vendor mocks don't leak across tests.

Gate: Tests Pass

  • npm run verify:touched -- <platform>-destination-<name> passes (L1)
  • The event examples run through runExamples with claims
  • Tests contain no hardcoded payloads, vendor configs, or expected outputs
  • Every assertion reads from examples.step[...].out
  • The conformance test passes with the right profile and minimal: examples.minimal (or withoutMinimal: '<reason>')
  • examples.step has a silent example, a failure example (outcome, with a description) for each event the destination cannot send, and a consent revocation example when the vendor has a consent switch

Phase 9: Document

Follow the writing-documentation skill for:

  • README structure and templates
  • Example validation against apps/quickstart/
  • Quality checklist before publishing

Key requirements for destination documentation:

  • Event mapping table (walkerOS → vendor format)
  • ## Installation renders <Quickstart data={data} />: the install line, integrated code and flow.json generated from examples.minimal. No hand-written install or setup snippet there; the doc-lint fails one
  • <Configuration type="destination" data={data} /> renders settings, mapping, examples and "When it fails" (from the failure examples' description) from walkerOS.json

Validation Checklist

Beyond understanding-development requirements (build, test, lint, no any):

  • Uses getEnv<Env>(env) pattern (never direct window/document access, never as Window/as Document/as unknown casts in src or tests)
  • dev.ts exports schemas (with settings; z.object({}) when the step has none) and examples
  • Multi-export package (two or more walkerOS.exports): exportExamples and exportSchemas in dev.ts list exactly those exports, the default included, and every exportSchemas entry has settings; the build fails otherwise (see using-step-examples)
  • No silent path: a failure is a thrown StepError, every vendor call is awaited, init never returns false, scripts load through loadVendor, rule.silent is honoured
  • Examples match type signatures
  • Tests use examples for assertions (not hardcoded values)
  • walkerOS.json generated at build time
  • walkerOS field in package.json
  • Works with no env: one test runs the published default export from a pure JSON config (no injected env, no mocked SDK) against a local fake endpoint on 127.0.0.1, with the real vendor SDK and dummy credentials, and asserts what the fake received. Mocked-env tests cannot catch a missing env?.X ?? X fallback, which turns a bundled flow into a silent no-op

Reference Files

WhatWhere
Simple templatepackages/web/destinations/plausible/
Complex examplepackages/web/destinations/gtag/
Typespackages/web/core/src/types/destination.ts

Related Skills

Create a New Source

Prerequisites

Before starting, read these skills:

Source Types

A source's identity is split into two fields:

FieldMeaningExamples
source.typeThe kind of source (its role / mechanism)browser, dataLayer, cookiefirst, fetch
source.platformThe runtime that hosts the source (web, server, app, ...)web, server
PlatformInputExample types
webDOM events, dataLayerbrowser, dataLayer
serverHTTP requests, webhooksexpress, fetch, lambda, cloudfunction

Augmenting SourceMap

Every source registers its type literal and any source-specific source.* fields by augmenting SourceMap from @walkeros/core. This is how the collector and downstream consumers know about the new source kind without loosening the union to string.

Add this to the source's src/types.ts (or src/types/index.ts):

import type { Source, Elb } from '@walkeros/core';

declare module '@walkeros/core' {
  interface SourceMap {
    // Replace `mySource` with the source's package-level identifier.
    mySource: {
      type: 'mySource'; // matches the literal you return from Source.Init
      platform?: 'web'; // 'web' | 'server' | 'app' | ...
      // Add any extra fields the source surfaces in `event.source.*` here.
      // e.g. `version?: string;` is already on the base Source - only add
      // truly source-specific keys.
    };
  }
}

Reference implementations: packages/web/sources/browser/src/types/index.ts, packages/web/sources/demo/src/types.ts. Conflicting declarations cause compile errors on purpose, this surfaces naming collisions early.

Source Categories

CategoryPurposeExamplesKey Concern
TransformationConvert external format → walkerOS eventsdataLayer, fetchMapping accuracy
TransportReceive events from specific platformgcp, aws, expressPlatform integration

Choose Your Template

ComplexityTemplateWhen to Use
Simple transformationfetch/Generic HTTP handler, data conversion
Platform transportgcp/, aws/Cloud platform integration
Browser interceptiondataLayer/DOM events, array interception

Process Overview

1. Research     → Deeply understand external system, SDK, and data format
2. Classify     → Determine source type and integration approach
3. Examples     → Define in/out pairs FIRST (start with the end result)
4. Mapping      → Define input → walkerOS event transformation
5. Scaffold     → Copy template and configure
6. Convention   → Add walkerOS.json metadata and buildDev
7. Implement    → Build using examples as test fixtures
8. Test         → Verify against example variations
9. Document     → Write README

Phase 1: Research

Goal: Deeply understand the external system before writing any code. Research quality determines implementation quality.

1.1 Find and Install Official SDK

Always prefer the vendor's official SDK package over raw HTTP API calls. The SDK handles transport, data formatting, and platform specifics - don't reinvent these.

  • Install the SDK - npm install @vendor/sdk and read the actual source
  • Read TypeScript types - Import types from the SDK package directly. Never duplicate type definitions. This ensures IntelliSense completeness and consistency with SDK updates.
  • Understand the full API surface - List every public method and type export. What data structures does the platform provide? What request/event formats exist?
# Search npm for official packages
npm search [vendor-name]
npm search @[vendor]

# Install and inspect actual types
npm install @vendor/sdk
ls node_modules/@vendor/sdk/lib/esm/

1.2 Understand SDK Architecture

  • Init options - What does the SDK expose? How is the platform connection established?
  • Call ordering - When does data arrive? Is it pushed (webhooks, callbacks) or pulled (polling, intercepting)? What are the timing implications?
  • Data format - What does the raw event/request look like? Headers, body structure, query params, authentication tokens?
  • Identity signals - Does the external system provide user IDs, session IDs, device IDs? How are they delivered (headers, cookies, body fields)?
  • Consent - Does the platform have consent signals? How are they communicated?

1.3 Identify All Data Entry Points

Go beyond just the primary event payload. Most external systems provide multiple data channels:

Data ChannelExampleswalkerOS Handling
Event payloadRequest body, DOM event dataDefault push()
Headers/metadataAuth tokens, content-type, origincontext or user
Query paramsUTM parameters, tracking IDsdata or context
Platform contextCloud function metadata, Lambda ctxsource or custom
IdentityUser ID, session ID, device IDuser
Consent signalsOpt-in/out flags, consent stringconsent

1.4 Check Existing Patterns

Review similar sources in the codebase:

# List existing sources
ls packages/web/sources/
ls packages/server/sources/

# Reference implementations
# - dataLayer: DOM-based, array interception
# - express: HTTP middleware
# - fetch: Generic HTTP handler (simplest server pattern)
# - gcp: Cloud Functions specific

Gate: Research Complete

  • Official SDK installed and types inspected (or HTTP API documented if no SDK exists)
  • All data entry points listed with their format
  • Init options and call ordering understood
  • Identity and consent signal delivery documented
  • Raw event/request structure mapped to walkerOS event fields

Checkpoint: Research Review (Optional)

If working with human oversight, pause here to confirm:

  • SDK integration approach correct?
  • Data capture strategy makes sense for the use case?
  • Any platform quirks or limitations to handle?

Phase 2: Classify Source Type

Goal: Understand what the source captures and how it delivers data, which determines implementation complexity.

2.1 Source Categories

CategoryDescriptionMapping NeededExample Sources
TransformationConverts external event format to walkerOSEssential - must map fieldsdataLayer, fetch
TransportReceives events from a specific platformStructural - platform unwrapgcp, aws, express
InterceptionIntercepts existing data flowsVaries - depends on data formatdataLayer, CMP sources

2.2 Determine Integration Approach

ApproachWhen to usePattern
Platform SDK as hostSDK provides typed request/responseUse SDK types, wrap handler in walkerOS source
DOM interceptionCapture browser-side eventsListen to DOM events, intercept arrays/globals
HTTP handlerGeneric webhook/API receiverParse request, extract events, forward to collector
Callback/event listenerPlatform provides event emitterRegister listener, transform events, forward to collector

Prefer the vendor SDK - it provides typed request/response objects and handles platform specifics. Raw HTTP parsing is a fallback when no SDK exists.

When using the vendor SDK:

  • Import types from the SDK package directly
  • Use SDK request/response types for handler signatures
  • Let the SDK handle platform-specific parsing (body parsing, header extraction)

For an HTTP handler, forward the full DeepPartialEvent the body carries, renaming only the wire-level event-name key. Never project the body through a field allowlist: that strips source provenance (release, trace) and other legitimate fields. express and fetch are the reference pass-through implementations.

Gate: Classification Complete

  • Source category identified (transformation / transport / interception)
  • Integration approach chosen (SDK / DOM / HTTP / callback)
  • Know what the source captures and how it delivers data

Phase 3: Create Input Examples (BEFORE Implementation)

Mandatory. Examples are the test fixtures for Phase 8. Define expected trigger / in / out triples FIRST - start with the end result in mind. Without examples, you cannot test. Even for simple sources, step examples are the single source of truth for tests, simulations, and documentation.

Authoritative pattern: See using-step-examples for the Three Type Zones, createTrigger contract, and CI integration. This skill reuses that contract - do not diverge.

3.1 Scaffold Directory Structure

mkdir -p packages/<platform>/sources/[name]/src/examples
mkdir -p packages/<platform>/sources/[name]/src/{schemas,types}

3.2 Required Files (3-4 files)

Use this layout in src/examples/. Every example a test reads lives in step.ts; do not add inputs.ts, outputs.ts, requests.ts or a standalone mapping.ts.

FileRequired?Purpose
examples/step.tsyesFlow.StepExample entries with trigger / in / out triples
examples/trigger.tsyescreateTrigger implementation following Trigger.CreateFn
examples/index.tsyesBarrel exports: env (if present), step, createTrigger
examples/env.tsif neededMock env for platform deps (browser window/document, express, etc.)

env.ts is included whenever the source touches platform globals or injected deps (browser, dataLayer, demo, the Lambda and Cloud Function sources). Sources whose tests drive the collector entirely through trigger.ts (e.g. web/sources/session, server/sources/express) may omit it. When in doubt, include it.

Some existing packages also keep fixture files their own tests import (fetch: inputs.ts, requests.ts; dataLayer: events.ts, mapping.ts; the CMP sources: inputs.ts, outputs.ts). A new source does not add them: anything a test needs goes into step.ts.

3.3 Three Type Zones for Sources

Sources are the inverse of destinations in the Three Type Zones model:

ZoneSource semantics
triggerHow to simulate the invocation (HTTP method, DOM event type, cloud event)
inExternal trigger content - HTTP request, DOM HTML, SDK payload (NOT a walkerOS event)
outWhat the source hands the collector: ['elb', event] tuples (Flow.StepOut)

Where a destination does WalkerOS.Event → vendor output, a source does external content → ['elb', event]. Read using-step-examples before authoring entries.

3.4 Typing Rules (strict)

No any. Every example value must be explicitly typed.

  • trigger uses the local source trigger type or a platform-native type (e.g. 'load' | 'click' for DOM, HTTP method strings for server sources).
  • in uses the vendor / platform SDK types imported from the official package whenever available (Express Request, Fetch Request, API Gateway APIGatewayProxyEvent, Lambda Context, GCP CloudEvent, etc.). Do not invent local request types when the platform publishes them.
  • out lists ['elb', event] tuples; each event is a WalkerOS.DeepPartialEvent (what the source pushes, before the collector completes it).
  • Step entries are typed Flow.StepExample from @walkeros/core.
  • Mock env is typed against the source's local Env type from ../types.
  • createTrigger is typed as Trigger.CreateFn<unknown, Result> and narrows its content to the source's own Content with a type guard: a simulation passes any JSON, and the conformance kit's createTrigger option is a Trigger.CreateFn, which a CreateFn<Content, Result> does not satisfy.

3.5 Code Template - examples/step.ts

import type { Flow } from '@walkeros/core';

// One step example per captured trigger / input shape.
// `trigger` tells createTrigger how to simulate the invocation.
// `in` is the platform-specific content (HTTP request, DOM HTML, SDK payload),
//      typed against the platform SDK's published types where available.
// `out` lists what the source hands the collector, as ['elb', event] tuples.

// Set `title` + `description` for public examples; mark test-only fixtures
// with `public: false`. See
// [walkeros-using-step-examples](../walkeros-using-step-examples/SKILL.md).

// Browser example (the browser source's own): a page load.
export const pageView: Flow.StepExample = {
  title: 'Page view',
  description:
    'A page load trigger captures the current URL, title, and referrer as a walker page view event.',
  trigger: {
    type: 'load',
    options: {
      url: 'https://example.com/docs',
      title: 'Documentation',
      referrer: 'https://example.com/',
    },
  },
  in: '', // no external content: DOM-driven trigger
  out: [
    [
      'elb',
      {
        name: 'page view',
        data: {
          domain: 'example.com',
          title: 'Documentation',
          referrer: 'https://example.com/',
          id: '/docs',
        },
        context: {},
        globals: {},
        trigger: 'load',
        source: {
          type: 'browser',
          platform: 'web',
          url: 'https://example.com/docs',
          referrer: 'https://example.com/',
        },
      },
    ],
  ],
};

// Server example (the express source's own): an HTTP POST with a walker event.
export const postEvent: Flow.StepExample = {
  title: 'POST event',
  description:
    'An Express POST to /collect with a JSON body becomes a single walker elb event.',
  trigger: { type: 'POST' },
  in: {
    method: 'POST',
    path: '/collect',
    body: {
      name: 'page view',
      data: { title: 'Home', url: 'https://example.com/' },
    },
  },
  out: [
    [
      'elb',
      {
        name: 'page view',
        data: { title: 'Home', url: 'https://example.com/' },
      },
    ],
  ],
};

3.6 examples/index.ts (barrel)

export * as env from './env'; // omit if the source has no env.ts
export * as step from './step';
export { createTrigger } from './trigger';

3.7 examples/trigger.ts - createTrigger

Every source exports a createTrigger following the unified Trigger.CreateFn interface. It simulates real-world invocations from the outside: a real request, DOM event or SDK call, as a user's platform makes it.

import type { Trigger } from '@walkeros/core';
import { isObject } from '@walkeros/core';
import { startFlow } from '@walkeros/collector';

// The source's own shapes (here an HTTP request and its answer)
export interface Content {
  method: string;
  url: string;
}
export interface Result {
  status: number;
}
const isContent = (value: unknown): value is Content =>
  isObject(value) &&
  typeof value.method === 'string' &&
  typeof value.url === 'string';

export const createTrigger: Trigger.CreateFn<unknown, Result> = async (
  config,
) => {
  let flow: Trigger.FlowHandle | undefined;

  const trigger: Trigger.Fn<unknown, Result> =
    (type?: string) => async (content) => {
      if (!isContent(content)) throw new Error('content is not a request');
      if (!flow) {
        const result = await startFlow(config);
        flow = { collector: result.collector, elb: result.elb };
      }
      // Package-specific: make a real HTTP request, inject DOM, dispatch an
      // SDK call, and return the Result type this source declares.
      return /* ... */;
    };

  return {
    get flow() {
      return flow;
    },
    trigger,
  };
};

The server template's examples/trigger.ts is a complete one. Reference implementations:

  • Browser: packages/web/sources/browser/src/examples/trigger.ts - DOM injection + native event dispatch
  • Session: packages/web/sources/session/src/examples/trigger.ts - no env.ts, trigger drives collector directly
  • Express: packages/server/sources/express/src/examples/trigger.ts - real HTTP fetch() to running server
  • CMP (Usercentrics): packages/web/sources/cmps/usercentrics/src/examples/trigger.ts - dispatches CMP events, asserts on collector consent state
  • Fetch (function handler): packages/server/sources/fetch/src/examples/trigger.ts - accesses source instance via collector.sources, calls source.push() with platform-native Request
  • AWS Lambda: packages/server/sources/aws/src/lambda/examples/trigger.ts - constructs API Gateway event + Lambda context
  • GCP CloudFunction: packages/server/sources/gcp/src/cloudfunction/examples/trigger.ts - synthesizes mock req/res (matching GCP Functions Framework)

3.8 Test Fixture Contract (hard rule)

The examples authored here are the Phase 8 test fixtures. No parallel fixtures allowed.

  • src/__tests__/stepExamples.test.ts MUST iterate examples via it.each(Object.entries(examples.step)).
  • Tests must NOT contain hardcoded trigger payloads, HTTP requests, DOM HTML, or expected events.
  • If a test needs a value that is not in examples.step, add it to step.ts first, then consume it from the test.
  • Tests invoke examples.createTrigger(config) and dispatch each example's trigger.type + in content, asserting the collector receives out.

See the canonical source tests packages/web/sources/browser/src/__tests__/stepExamples.test.ts and packages/server/sources/express/src/__tests__/stepExamples.test.ts.

3.9 Export via dev.ts

export * as schemas from './schemas';
export * as examples from './examples';

Phase 3 Acceptance Checklist

  • src/examples/step.ts - one Flow.StepExample per captured trigger / input shape, typed trigger / in / out
  • src/examples/trigger.ts - exports createTrigger typed as Trigger.CreateFn<unknown, Result>, narrowing its content with a guard
  • src/examples/index.ts - barrel exports step, createTrigger, and env (when present)
  • src/examples/env.ts - included whenever the source touches platform globals or injected deps; typed against local Env; no real network
  • No standalone inputs.ts, outputs.ts, requests.ts, or mapping.ts files
  • All platform / SDK types imported from the official package - no any, no reinvented request / response shapes
  • src/__tests__/stepExamples.test.ts iterates examples.step via it.each(Object.entries(...))
  • Tests contain zero hardcoded payloads, requests, or expected events - everything flows from examples.step
  • Edge cases included (minimal input, invalid input)
  • npm run build passes - examples compile against published types
  • Each example traces: trigger + in → source push → matches out

Phase 4: Define Mapping

Goal: Document transformation from input format to walkerOS events.

Mapping lives inside each Flow.StepExample entry in step.ts - no separate mapping.ts file. Sources typically carry the mapping either in the source's own settings (see dataLayer for an example) or inline via the trigger → in → out relationship: the in content is the raw platform payload; the out lists the events the source pushes after its transformation, as ['elb', event] tuples.

Verify Mapping Logic

For each entry in step.ts, trace:

Input: examples.step.pageView.trigger + examples.step.pageView.in
  ↓ createTrigger dispatches the trigger
  ↓ Source receives platform content, runs its transformation
  ↓ Source calls env.push / collector.push
Output: Should match examples.step.pageView.out (['elb', event] tuples)

Gate: Mapping Verified

  • Step examples cover the main input event types
  • Each example name transforms to correct walkerOS event name
  • Each example traces correctly from (trigger, in) to out

Phase 5: Scaffold

Template sources:

  • Web: packages/web/sources/dataLayer/
  • Server: packages/server/sources/fetch/ (simplest pattern)
cp -r packages/server/sources/fetch packages/server/sources/[name]
cd packages/server/sources/[name]

# Update package.json: name, description, repository.directory

Directory structure:

packages/server/sources/[name]/
├── src/
│   ├── index.ts           # Main export
│   ├── __tests__/         # stepExamples.test.ts and the other tests
│   ├── dev.ts             # Exports schemas and examples
│   ├── examples/
│   ├── schemas/
│   └── types/
├── package.json
├── tsconfig.json
├── tsup.config.ts
├── jest.config.mjs
└── README.md

Transformer Chain Integration

Sources can wire to transformer chains via next in the init config:

sources: {
  mySource: {
    code: sourceMySource,
    config: { settings: { /* ... */ } },
    next: 'validate'  // Events go through validator before collector
  }
}

Phase 6: walkerOS.json Convention

Every walkerOS package ships a walkerOS.json file for CDN-based schema discovery.

Add walkerOS field to package.json

{
  "walkerOS": { "type": "source", "platform": "web" },
  "keywords": ["walkerOS", "walkerOS-source", ...]
}

Use buildDev() in tsup.config.ts

Keep buildDev() in the defineConfig array (the template package's tsup.config.ts has it):

import { buildDev } from '@walkeros/config/tsup';
// In defineConfig array:
buildDev(),

It builds src/dev.ts and writes dist/walkerOS.json from the schemas, examples and hints it exports.

Hints (Optional)

If your source has capabilities, behaviors, or troubleshooting patterns not obvious from schemas alone, add hints. See walkeros-writing-documentation skill for full guidelines.

Create src/hints.ts:

import type { Hint } from '@walkeros/core';

export const hints: Hint.Hints = {
  'capture-timing': {
    text: 'Describes when events are captured. See settings schema for options.',
    code: [{ lang: 'json', code: '{ "settings": { ... } }' }],
  },
};

Export from src/dev.ts:

export * as schemas from './schemas';
export * as examples from './examples';
export { hints } from './hints';

Guidelines:

  • Expand awareness - describe capabilities ("supports X, Y, Z"), don't prescribe one path
  • Reference schemas and examples, don't duplicate them
  • Verify every claim against actual implementation before publishing
  • Key naming: kebab-case, group with prefixes (auth-*, capture-*, troubleshoot-*)
  • Most sources don't need hints - schemas and examples cover the common case

Gate: Convention Met

  • walkerOS field in package.json with type and platform
  • buildDev() in tsup.config.ts
  • Build generates dist/walkerOS.json
  • Keywords include walkerOS and walkerOS-source

Runtime-only npm dependencies

A third-party npm dependency that cannot be ESM-bundled (uses __dirname, ships a .node binary, etc.) needs no declaration: server bundles are traced with nft, which copies every file the bundle reaches at runtime. Do not add a walkerOS.bundle field to package.json: the CLI ignores it and warns still declares walkerOS.bundle.<keys>; this is ignored. See walkeros-using-cli: server bundles use nft tracing.


Phase 7: Implement

Now write code to produce the outputs defined in Phase 3.

Template Files

FilePurposeTemplate
types/index.tsType definitionstypes.ts
schemas/index.tsZod settings schemaschemas.ts
index.tsMain sourceindex.ts
dev.tsDev entry (schemas, examples)dev.ts
examples/step.ts, trigger.ts, index.tsexamples/ (step.ts, trigger.ts, index.ts)
__tests__/stepExamples.test.tsStep example testsindex.test.ts
__tests__/conformance.test.tsConformance kit (profile source)conformance.test.ts

The server template is an HTTP handler that turns a platform's webhook body into walkerOS events and answers the producer from the push result.

Key Patterns

  1. Context destructuring: Extract config, env, logger, id from context
  2. Settings schema, defaults inline: the Zod schema in src/schemas describes the settings for walkeros validate, walkerOS.json and the docs; the source applies its defaults inline and does not parse settings at runtime (zod stays out of the runtime bundle)
  3. Forward to collector: Call env.push() to send events to the collector
  4. Platform deps via env: All platform dependencies (window, document, express, cors) must go through env with fallback to globals/imports: env.express ?? express. This enables testing without mocking globals.
  5. Failures are reported, not logged by the source: a source that cannot start throws a StepError (code: 'config') from its factory or init; it never returns early and never returns false. Input the collector rejects comes back as invalid in the push result and is counted as rejected by the collector. A failure on the source's own side that the push result does not carry (for example an elbLayer entry that fails) goes through reportCause from @walkeros/core (step source.<id>, named by its event, never its payload). Use logger for debug traces of routine operations.
  6. Tell the producer the truth: an HTTP source builds its answer from the push result with pushResultToOutcome (or acknowledge(result)): ack answers 200 with the source's success body, retry (a retryable failure) answers 500, or 207 for the failed events of a batch, reject (invalid input) answers 400. A failure a resend cannot fix is acknowledged, because redelivery would loop. A queue consumer acks on ack and reject and nacks only on retry. The body never contains a destination id or a step's error text.
  7. Return Source.Instance: Return { type, config, push } object
  8. Optional destroy method: Implement if the source holds resources (HTTP servers, timers, connections) that need cleanup on shutdown

Gate: Implementation Compiles

  • npm run build passes
  • npm run verify:touched -- <platform>-source-<name> passes (L1: typecheck + lint + test; the argument is the package name after @walkeros/)

Adding setup (optional)

A source package can implement an optional setup() function to provision external resources idempotently: Pub/Sub subscriptions, webhook registrations on upstream platforms, queue declarations, SQS queues, polling cursors, inbound API keys. Setup runs only when an operator explicitly types walkeros setup source.<name>. The runtime never auto-invokes it from init(), push, or destroy().

The framework provides the slot, the CLI command, and a resolveSetup helper. The package owns: what setup means, idempotency, error handling, return value.

Today the command cannot run a source's setup. walkeros setup reads the package's default export as an object with a setup method, while a source's default export is its init function, so walkeros setup source.<name> stops with Package <name> has no default export. The SQS source exports its setup from its module (packages/server/sources/aws/src/sqs/index.ts), and the Pub/Sub pull source returns setup on the instance its init returns; the command reaches neither, so provision their resources with the platform's own tooling (gcloud, the AWS CLI, your infrastructure code).

For background on how setup fits the source lifecycle, see understanding-sources.

Types

// types/index.ts
import type { Elb, Source as CoreSource } from '@walkeros/core';

export interface Settings {
  /* runtime push settings */
}
export interface InitSettings {
  /* one-time init settings */
}
export interface Mapping {
  /* event extraction mapping */
}
export type Push = Elb.Fn; // the source's push signature (an HTTP handler, ...)
export interface Env extends CoreSource.BaseEnv {
  /* injected platform deps (SDK clients, request handlers, etc.) */
}

// The package's own setup options interface.
// Becomes the U slot of Types; surfaces as `config.setup: boolean | Setup` for users.
export interface Setup {
  // package-specific provisioning options
  // e.g. for Pub/Sub source: subscriptionName, ackDeadlineSeconds, filter
  subscriptionName?: string;
  ackDeadlineSeconds?: number;
}

export type Types = CoreSource.Types<
  Settings,
  Mapping,
  Push,
  Env,
  InitSettings,
  Setup
>;

Implementation

// setup.ts
import type { Source as CoreSource, SetupFn } from '@walkeros/core';
import { resolveSetup } from '@walkeros/core';
import type { Setup, Types } from './types';

const DEFAULT_SETUP: Setup = {
  ackDeadlineSeconds: 60,
};

export const setup: SetupFn<
  CoreSource.Config<Types>,
  CoreSource.Env<Types>
> = async ({ config, env, logger }) => {
  const options = resolveSetup(config.setup, DEFAULT_SETUP);
  if (!options) return; // config.setup is false or unset

  // Package-specific provisioning, idempotent.
  // Returning a structured object (e.g. { subscriptionCreated: true })
  // makes that data available to operators via `walkeros setup ... | jq`.
};

Export it next to the source's init function, as the SQS source's module does:

// index.ts
import type { Source as CoreSource } from '@walkeros/core';
import type { Types } from './types';

export { setup } from './setup';

export const sourceMine: CoreSource.Init<Types> = async (context) => ({
  type: 'my-source',
  config: context.config,
  push: context.env.elb,
});

export default sourceMine;

When to implement

Implement setup() when your source needs first-time provisioning of upstream resources before events can be received: Pub/Sub subscriptions bound to a topic, webhook registrations on upstream SaaS platforms (Stripe, GitHub, Shopify), SQS queue declarations, message broker bindings, polling cursors. Skip it when your source only listens on an HTTP port the runtime already owns or intercepts data already present (DOM events, dataLayer pushes).

Contract

  • Triggered only by walkeros setup <kind>.<name>. Never by runtime push, init, or destroy.
  • Idempotency is your responsibility. Re-running setup against a fully provisioned environment must be a safe no-op. Use try-create-catch-409 on REST APIs, IF NOT EXISTS on SQL, native idempotent operations where available. The framework does not retry, track state, or detect drift.
  • Return structured data from setup() when useful for operator scripting. The CLI emits non-undefined return values as JSON to stdout.
  • For packages where setup: true (boolean form) is meaningless because mandatory fields have no safe defaults (e.g., GitHub webhook webhookUrl, Pub/Sub source topicName), reject the boolean form with a clear runtime error listing required fields:
if (config.setup === true) {
  throw new Error(
    'github-webhook source setup requires explicit options: ' +
      '{ webhookUrl, repo, events }. There is no safe default.',
  );
}

Phase 8: Test Against Examples

Tests verify implementation against the examples from Phase 3. If examples are incomplete, tests will be incomplete.

See testing-strategy for the shared env / dev-examples conventions this phase depends on.

Verify implementation produces expected outputs.

Test Template

Use the test template: index.test.ts, copied to src/__tests__/stepExamples.test.ts. Canonical references:

  • packages/web/sources/browser/src/__tests__/stepExamples.test.ts
  • packages/server/sources/express/src/__tests__/stepExamples.test.ts

Key Test Patterns

  1. it.each(Object.entries(examples.step)) is mandatory - one iteration per step example. Do not write per-feature tests with hand-rolled payloads.
  2. Drive via createTrigger - construct the trigger with startFlow config, then dispatch each example's trigger.type + in content.
  3. A typed context helper for a test that calls the source factory directly: a small function in the test file, as packages/server/sources/express/src/__tests__/index.test.ts defines createSourceContext.
  4. Zero hardcoded payloads - every trigger type, request body, DOM HTML, and expected event comes from examples.step or examples.env. If you need something new, add it to examples first.
  5. Test error paths - verify graceful error handling for invalid input (add an error example to examples.step if needed), and that a push that fails for a retryable reason is answered with a retry, a deterministic one with an ack.
  6. Conformance - run the source with conformance('source', source, options) from @walkeros/collector/dev (profile source, createTrigger for how to feed it); it runs on a real collector. A web source may pass page from @walkeros/web-core/dev; the kit requires it only for the browser profiles web-sdk and vendor-global.

Gate: Tests Pass

  • npm run verify:touched -- <platform>-source-<name> passes (L1)
  • Tests iterate via it.each(Object.entries(examples.step))
  • Tests contain no hardcoded payloads, requests, or expected events
  • Every assertion reads from examples.step[...].out
  • Invalid input handled gracefully (no crashes)

Phase 9: Document

Follow the writing-documentation skill for:

  • README structure and templates
  • Example validation against apps/quickstart/
  • Quality checklist before publishing

Key requirements for source documentation:

  • Input format table documenting expected fields
  • Event name mapping table (source format → walkerOS format)
  • Configuration options table
  • Working code example with imports
  • Installation instructions

Validation Checklist

Beyond understanding-development requirements (build, test, lint, no any):

  • dev.ts exports schemas (with settings; z.object({}) when the step has none) and examples
  • Multi-export package (two or more walkerOS.exports): exportExamples and exportSchemas in dev.ts list exactly those exports, the default included, and every exportSchemas entry has settings; the build fails otherwise (see using-step-examples)
  • Examples include edge cases (minimal, invalid input)
  • Invalid input returns gracefully (no crashes, clear error)
  • Tests use examples for assertions (not hardcoded values)
  • walkerOS.json generated at build time
  • walkerOS field in package.json

Reference Files

WhatWhere
Web templatepackages/web/sources/dataLayer/
Server templatepackages/server/sources/fetch/
Source typespackages/core/src/types/source.ts
Test eventspackages/core/src/eventGenerator.ts

Related Skills

Create a New Transformer

Prerequisites

Before starting, read these skills:

Transformer Categories

CategoryPurposeExample
ValidateCheck event structure/contentJSON Schema, required fields
EnrichAdd data to eventsServer timestamps, geo data
RedactRemove/mask sensitive dataStrip PII, anonymize IPs

Process Overview

1. Research     → Understand use case (validate/enrich/redact)
2. Examples     → Create event before/after examples FIRST
3. Scaffold     → Copy template, configure package.json
4. Convention   → Add walkerOS.json metadata and buildDev
5. Implement    → Build transformer with TDD
6. Test         → Verify against example transformations
7. Document     → Write README

Supporting Files

This skill includes reference files you can copy:

Copy them into a package as Phase 3 lays it out: index.ts to src/transformer.ts, types.ts to src/types.ts, schemas.ts to src/schemas/index.ts, index.test.ts to src/__tests__/stepExamples.test.ts, conformance.test.ts to src/__tests__/ and examples/ to src/examples/.


Phase 1: Research

Goal: Understand what the transformer needs to do.

1.1 Define Use Case

  • Category: Validate, Enrich, or Redact?
  • Input: What events will this process?
  • Output: What should change? What should be blocked?
  • Configuration: What settings does user need?

1.2 Check Existing Patterns

# Reference implementation
ls packages/server/transformers/fingerprint/

# Transformer types
cat packages/core/src/types/transformer.ts

Gate: Research Complete

  • Category identified (validate/enrich/redact)
  • Input/output transformation defined
  • Configuration options listed

Phase 2: Create Examples (BEFORE Implementation)

Goal: Define event transformations in dev entry FIRST.

2.1 Scaffold Directory Structure

mkdir -p packages/server/transformers/[name]/src/{examples,schemas}

2.2 Step Examples

Write one step example per case in src/examples/step.ts: events that pass through (modified or not), the expected output after processing, and events that are blocked (for validators). Each export is one Flow.StepExample:

// examples/step.ts
import type { Flow } from '@walkeros/core';

export const orderPasses: Flow.StepExample = {
  title: 'Order passes',
  description: 'An order event passes through unchanged.',
  in: { name: 'order complete', data: { id: 'ORD-123' } },
  out: [
    ['return', { event: { name: 'order complete', data: { id: 'ORD-123' } } }],
  ],
};

export const debugFiltered: Flow.StepExample = {
  public: false,
  description: 'A debug event is dropped.',
  in: { name: 'debug test', data: { message: 'noise' } },
  out: [['return', false]], // the transformer drops this event
};

in is a walkerOS event; out is the ['return', value] effect: the result the transformer returns ({ event }), or false for an event it drops. A failure example sets outcome instead (see Phase 6). Set title + description for public examples; mark test-only fixtures with public: false. src/examples/index.ts exports them with export * as step from './step';. See using-step-examples for the Three Type Zones.

2.3 Export via dev.ts

export * as schemas from './schemas';
export * as examples from './examples';

Gate: Examples Valid

  • All example files compile (npm run build)
  • Can trace: input event → expected output for each example

Phase 3: Scaffold

Template transformer: packages/server/transformers/fingerprint/ (a server transformer). A platform-neutral transformer lives in packages/transformers/ (for example validate, ga4).

cp -r packages/server/transformers/fingerprint packages/server/transformers/[name]
cd packages/server/transformers/[name]

# Update package.json: name, description, repository.directory

Directory structure:

packages/server/transformers/[name]/
├── src/
│   ├── index.ts           # Main export
│   ├── transformer.ts     # Transformer implementation
│   ├── types.ts           # Settings, Types interfaces
│   ├── dev.ts             # Exports schemas and examples
│   ├── examples/
│   │   ├── index.ts       # export * as step from './step'
│   │   └── step.ts        # Step examples
│   ├── schemas/
│   │   └── index.ts       # Zod schemas for settings
│   └── __tests__/         # stepExamples.test.ts, conformance.test.ts
├── package.json
├── tsconfig.json
├── tsup.config.ts
├── jest.config.mjs
└── README.md

Phase 4: walkerOS.json Convention

Every walkerOS package ships a walkerOS.json file for CDN-based schema discovery.

Add walkerOS field to package.json

{
  "walkerOS": {
    "type": "transformer"
  },
  "keywords": ["walkerOS", "walkerOS-transformer", ...]
}

Use buildDev() in tsup.config.ts

Keep buildDev() in the defineConfig array (the template package's tsup.config.ts has it):

import { buildDev } from '@walkeros/config/tsup';
// In defineConfig array:
buildDev(),

It builds src/dev.ts and writes dist/walkerOS.json from the schemas, examples and hints it exports.

Hints (Optional)

If your transformer has capabilities, behaviors, or troubleshooting patterns not obvious from schemas alone, add hints. See walkeros-writing-documentation skill for full guidelines.

Create src/hints.ts:

import type { Hint } from '@walkeros/core';

export const hints: Hint.Hints = {
  'validation-behavior': {
    text: 'Describes how validation works. See settings schema for options.',
    code: [{ lang: 'json', code: '{ "settings": { ... } }' }],
  },
};

Export from src/dev.ts:

export * as schemas from './schemas';
export * as examples from './examples';
export { hints } from './hints';

Guidelines:

  • Expand awareness: describe capabilities ("supports X, Y, Z"), don't prescribe one path
  • Reference schemas and examples, don't duplicate them
  • Verify every claim against actual implementation before publishing
  • Key naming: kebab-case, group with prefixes (validation-*, enrichment-*, troubleshoot-*)
  • Most transformers don't need hints; schemas and examples cover the common case

Gate: Convention Met

  • walkerOS field in package.json with type: "transformer"
  • buildDev() in tsup.config.ts
  • Build generates dist/walkerOS.json
  • Keywords include walkerOS and walkerOS-transformer

Runtime-only npm dependencies

A third-party npm dependency that cannot be ESM-bundled (uses __dirname, ships a .node binary, etc.) needs no declaration: server bundles are traced with nft, which copies every file the bundle reaches at runtime. Do not add a walkerOS.bundle field to package.json: the CLI ignores it and warns still declares walkerOS.bundle.<keys>; this is ignored. See walkeros-using-cli: server bundles use nft tracing.


Phase 5: Implement

Now write code to transform examples as expected.

5.1 Define Types

See templates/validation/types.ts for the pattern. Define Settings and Types interfaces.

5.2 Implement Transformer (Context Pattern)

Transformers use the context pattern - they receive a single context object containing config, env, logger, id, and collector.

See templates/validation/index.ts for a complete implementation example.

Key patterns:

  1. Context destructuring: Extract config, logger, id from init context
  2. Settings schema, defaults inline: the Zod schema (see templates/validation/schemas.ts) describes the settings for walkeros validate, walkerOS.json and the docs; the transformer applies its defaults inline and does not parse settings at runtime
  3. Push receives pushContext: The push function gets event + push context
  4. Return values: { event } (continue), void (passthrough), false (cancel on purpose: a deliberate drop, not a failure; the push answers ok: true), or an array of results to fork the event (see understanding-transformers)
  5. A failure is a throw: throw a StepError from @walkeros/core with a code. missing-input is the code for an event or an ingest that lacks the input the transformer needs (do not drop it silently); config for a missing setting. init throws instead of returning false or returning early, and a transformer never logs a failure itself: the collector turns the throw into ok: false with the transformer named and writes the one ERROR line. A transformer that throws a plain error fails with the code transformer.

5.3 Export

src/index.ts:

export { transformerRedact } from './transformer';
export type { Settings, Types } from './types';

Gate: Implementation Compiles

  • npm run build passes
  • npm run verify:touched -- <platform>-transformer-<name> passes (L1: typecheck + lint + test; the argument is the package name after @walkeros/, transformer-<name> for a platform-neutral one)

Phase 6: Test Against Examples

Verify implementation produces expected outputs.

See templates/validation/index.test.ts for a complete test suite showing:

  1. createTransformerContext() helper - a typed init context for unit tests, defined in the test file
  2. createPushContext() helper - the same for the push context
  3. Examples for test data - Don't hardcode test values
  4. Return value testing - Verify event, void, or false returns
  5. Conformance - conformance('transformer', transformer, options) from @walkeros/collector/dev (profile transformer) runs it on a real collector, see packages/server/transformers/fingerprint/src/__tests__/conformance.test.ts. A transformer needs no page. A transformer whose input is the ingest and not an event declares its failure with a missing-input example (an event as in, out: [], an outcome and a description).

Gate: Tests Pass

  • npm run verify:touched -- <platform>-transformer-<name> passes (L1)
  • Tests verify against example outputs

Phase 7: Document

Follow the writing-documentation skill for:

  • README structure and templates
  • Quality checklist before publishing

Key requirements for transformer documentation:

  • Use case description (validate/enrich/redact)
  • Configuration options table
  • Working code example with imports
  • Installation instructions
  • Link to website docs

Transformer-Specific Validation

Beyond understanding-development requirements (build, test, lint, no any):

  • dev.ts exports schemas (with settings; z.object({}) when the step has none) and examples
  • Multi-export package (two or more walkerOS.exports): exportExamples and exportSchemas in dev.ts list exactly those exports, the default included, and every exportSchemas entry has settings; the build fails otherwise (see using-step-examples)
  • Examples include before/after event pairs
  • Return values handle all cases (event, void, false)
  • A failure is a thrown StepError (missing-input for absent input), init throws instead of returning false, conformance test passes
  • Tests use examples for assertions (not hardcoded values)
  • walkerOS.json generated at build time
  • walkerOS field in package.json

Reference Files

WhatWhere
Templatepackages/server/transformers/fingerprint/
Typespackages/core/src/types/transformer.ts
Chaining logicpackages/collector/src/transformer.ts

Related Skills

Debugging walkerOS Events

Quick Diagnosis

SymptomLikely CauseCheck
No events at allSource not initializedConsole for errors, verify startFlow()
Events fire but destination silentMapping mismatchEvent name matches mapping?
Partial data missingPath doesn't existLog event structure, check nested paths
Consent blockingRequired consent not grantedCheck consent config, grant consent
Destination errorVendor API issueCheck network tab, vendor console

Debugging Strategies

1. Console Logging

Log all events at collector level:

import { startFlow } from '@walkeros/collector';

const { elb } = await startFlow({
  destinations: {
    debug: {
      code: {
        type: 'debug',
        config: {},
        push: async (event) => {
          console.log('[walkerOS Event]', {
            name: event.name,
            data: event.data,
            context: event.context,
            consent: event.consent,
            timestamp: event.timestamp,
          });
        },
      },
    },
    // ... other destinations
  },
});

2. Network Tab Inspection

For destinations that make HTTP calls:

  1. Open DevTools → Network tab
  2. Filter by destination domain (e.g., google-analytics.com, facebook.com)
  3. Trigger event
  4. Inspect request payload

What to look for:

  • Request being made at all?
  • Correct endpoint URL?
  • Payload structure matches vendor spec?

3. Vendor Debug Tools

VendorDebug Tool
GA4GA4 DebugView
MetaFacebook Pixel Helper
PlausiblePlausible Dashboard real-time

4. Simulate or Mock the Destination

To see what a destination's mapping produces without calling the vendor, simulate it: the vendor calls are recorded on a mock env instead of sent.

walkeros push flow.json -e event.json --simulate destination.ga4

config.mock ("mock": true in the destination's config) replaces only the destination's push call with that value: the mapping still runs, init still runs (it may load the vendor script and make its init calls), and the push answers with the mock value instead of calling the vendor.

Common Issues

Event Name Mismatch

Problem: Event fires but destination doesn't receive it.

// Event pushed
elb({ name: 'product view', data: { id: 'P123' } });

// Mapping expects a different name
const mapping = {
  Product: {
    // Wrong: capital P
    View: { name: 'view_item' }, // Wrong: capital V
  },
};

Fix: Event names are case-sensitive. Use exact match:

const mapping = {
  product: {
    view: { name: 'view_item' },
  },
};

Missing Nested Data

Problem: items array is empty in destination.

// Event structure
const event = {
  name: 'order complete',
  data: { value: 100 },
  nested: [{ entity: 'product', data: { id: 'P1' } }],
};

// Mapping tries wrong path
const rule = {
  data: {
    map: {
      items: {
        loop: [
          'data.items', // Wrong: nested is at root, not in data
          { map: { id: 'data.id' } },
        ],
      },
    },
  },
};

Fix: Use correct path to nested array:

const items = {
  loop: [
    'nested', // Correct: root-level nested
    { map: { item_id: 'data.id' } },
  ],
};

Consent Blocking Events

Problem: Events not reaching destination.

Check 1: Does destination require consent?

// Destination config
const config = {
  consent: { marketing: true }, // Requires marketing consent
};

Check 2: Is consent granted?

// Check current consent state
console.log(event.consent);

// Grant consent
elb('walker consent', { marketing: true });

Simulated Destination Recorded Nothing: Pending vs Consent Skip

walkeros push flow.json -e event.json --simulate destination.X prints why a destination sent nothing, from the collector's own records (--json and MCP: skipped on the result):

Line (skipped)MeaningFix in simulate
pending: waits for consent (require) ({ reason: 'pending', require: ['consent'] })require: ["consent"]: never started, no consent state exists--consent '{"functional":true}' (MCP state.consent)
skipped: consent (requires marketing; granted functional) ({ reason: 'consent', required, granted })Started, but config.consent denied the event (collector + event consent)Grant the key in --consent or the event's consent
mapping: <key>, no calls, skipped: consent (...)That rule's own consent denied the event (on top of config.consent)Grant the key in --consent or the event's consent
mapping: none (skipped before mapping), no skipped lineNot consent: an event without name, a filter, a before chain stop, or an errorCheck the event has a name, the --verbose logs, the before chain and the mapping rules

require checks that consent state is present at all; config.consent checks each event's keys. --consent is the collector's starting state: any value clears the consent require, and its granted keys feed the consent check. A require other than consent (e.g. user) cannot be seeded by simulate.

An event without name sends nothing and the simulation still answers success: true with mapping: none (skipped before mapping) and no calls, while a real push of it answers ok: false with Event name is required.

Does the CMP Source Send Consent?

Simulate the CMP source with its own example (in plus trigger as { content, trigger }). The consent command it issues is recorded as an elb call:

success: true
  source.usercentrics
    call elb("walker consent",{"essential":true,"functional":true,"marketing":true})
    no events

No call elb("walker consent",...) means the CMP source published nothing for that input (for example explicitOnly with only implicit defaults), so destinations waiting for consent stay pending in the real flow. --consent does not help here: it is the collector's starting state and is never recorded as the source's call. For a consent-gated source (the session source with settings.consent), no events without --consent is expected; pass --consent '{"functional":true}' (MCP state.consent) to see what it emits.

startFlow Throws Cannot read properties of undefined (reading 'config')

A destinations entry without code (for example { push, config } written directly) makes startFlow throw this TypeError, which names neither the destination nor code. Put the destination instance under code, as in "Console Logging" above.

Vendor SDK Not Loaded

Problem: the push fails with vendor-not-loaded (Vendor script failed to load), vendor-not-ready (Vendor not ready: its global is missing) or vendor-timeout, or the vendor receives nothing although the push answers done.

Cause: the vendor script was blocked, did not load in time, or was never injected: a web destination that loads its vendor script (gtag, Meta, Plausible and others) injects it only with config.loadScript: true. Without it nothing is injected, and gtag's calls wait in dataLayer while the push answers done.

Fix: set "loadScript": true in the destination's config, or keep the vendor's own tag on the page. A destination package loads its script through loadVendor from @walkeros/web-core, which reports the codes above instead of losing events (see create-destination, Phase 7, "Scripts through loadVendor").

Function Mapping Errors

Problem: the log shows ERROR mapping fn failed { step: 'mapping.product view', code: 'mapping', ... error: "Cannot read properties of undefined (reading 'price')" }, and the destination receives the event without that field.

{
  "data": {
    "map": {
      "value": {
        "fn": "$code:(value, { event }) => event.data.product.price * 100"
      }
    }
  }
}

The function fails when the event has no data.product. A function that returns NaN instead (for example event.data.price * 100 on an event without a price) drops the field with no log line at all.

Fix: Read the event through optional chaining and a fallback, which avoids both:

{
  "data": {
    "map": {
      "value": {
        "fn": "$code:(value, { event }) => (event.data?.product?.price ?? 0) * 100"
      }
    }
  }
}

Debugging Checklist

When events aren't working:

  1. Console errors? Check browser console for exceptions
  2. Event pushed? Add debug destination to log all events
  3. Mapping matched? Verify entity/action names exactly match
  4. Data paths correct? Log full event structure, verify paths exist
  5. Consent granted? Check consent requirements and state
  6. SDK loaded? Verify vendor script loaded before push
  7. Network request? Check DevTools network tab
  8. Vendor receiving? Use vendor debug tools

Testing in Isolation

Run one destination on a real collector and record its vendor calls through env instead of loading the vendor script:

import { startFlow } from '@walkeros/collector';
import { getEvent } from '@walkeros/core';
import { destinationGtag } from '@walkeros/web-destination-gtag';

// Record every gtag() call instead of loading gtag.js
const calls: unknown[][] = [];
const { elb } = await startFlow({
  destinations: {
    ga4: {
      code: destinationGtag,
      config: {
        settings: { ga4: { measurementId: 'G-XXXXXXXXXX' } },
        mapping: {
          product: {
            view: { name: 'view_item', data: { map: { value: 'data.price' } } },
          },
        },
      },
      env: {
        window: {
          dataLayer: [],
          gtag: (...args: unknown[]) => calls.push(args),
        },
      },
    },
  },
});

const result = await elb(getEvent('product view'));
console.log(result.ok, calls);
// true [['config', 'G-XXXXXXXXXX', {}], ['event', 'view_item', { value: 420, send_to: 'G-XXXXXXXXXX' }]]

The CLI does the same for a flow file: walkeros push flow.json -e event.json --simulate destination.ga4.

MCP Tool Debugging

When using walkerOS MCP tools, check _hints.warnings in tool responses for diagnostic information:

  • flow_simulate warns when 0 destinations exist or none received the event; a destination that sent nothing carries skipped (pending on its require, or a consent skip) and the warning says which. Pass state.consent to start a destination that waits for consent. step is required (e.g. "destination.gtag"). Source steps take a { content, trigger? } event where content is { name, data }; sources, including @walkeros/source-demo, can be simulated this way. A source result shows the source's own walker commands as elb calls (no verbose needed) and its summary counts them (Source captured 0 events and 1 command); state.consent applies to source steps too.
  • flow_bundle with stats: false still writes the bundle, but its result reads success: false and Bundle produced no output
  • flow_examples warns when no examples are found in the config
  • package_search browses the published step packages whose walkerOS metadata it could read. When the npm search fails, the catalog is empty and a warning says so; a second warning counts the npm packages without readable walkerOS metadata, which includes every package that is not a step, so it appears on every browse
  • Error responses carry error and, where the tool knows a recovery, a hint

From a terminal, walkeros diagnostics reports the CLI version, the app URL (and whether it came from WALKEROS_APP_URL, the config file or the default), and whether the app answers its health route.

Related Skills

Mapping Configuration Recipes

Prerequisites

Read understanding-mapping first for core concepts.

Quick Reference

I want to...Use this pattern
Rename event{ name: 'new_name' }
Extract nested value'data.nested.value'
Set static value{ value: 'USD' }
Transform value{ fn: (value, { event }) => transform(event) }
Build object{ map: { key: 'source' } }
Process array{ loop: ['source', { map: {...} }] }
Gate by consent{ key: 'data.email', consent: { marketing: true } }
Gate one event type by consentrule consent: { marketing: true } (on top of config.consent)
First defined value (fallback)[{ key: 'data.sku' }, { key: 'data.id' }]
Add a field to a default rule{ extend: { data: { map: { x: 'path' } } } }
Drop a field from output{ remove: ['field.path'] }

Common Recipes

GA4 / gtag

Product view → view_item:

product: {
  view: {
    name: 'view_item',
    data: {
      map: {
        currency: { key: 'data.currency', value: 'EUR' },
        value: 'data.price',
        items: {
          loop: [
            'this', // the product is the event itself
            {
              map: {
                item_id: 'data.id',
                item_name: 'data.name',
                item_category: 'data.category',
                price: 'data.price',
                quantity: { key: 'data.quantity', value: 1 },
              },
            },
          ],
        },
      },
    },
  },
}

Order complete → purchase:

order: {
  complete: {
    name: 'purchase',
    data: {
      map: {
        transaction_id: 'data.id',
        value: 'data.value',
        currency: { key: 'data.currency', value: 'EUR' },
        items: {
          loop: [
            'nested',
            {
              // only the products, not other nested entities
              condition: (entity) =>
                typeof entity === 'object' &&
                entity !== null &&
                'entity' in entity &&
                entity.entity === 'product',
              map: {
                item_id: 'data.id',
                item_name: 'data.name',
                price: 'data.price',
                quantity: { key: 'data.quantity', value: 1 },
              },
            },
          ],
        },
      },
    },
  },
}

Meta Pixel

Product view → ViewContent:

product: {
  view: {
    name: 'ViewContent',
    data: {
      map: {
        content_ids: { set: ['data.id'] },
        content_type: { value: 'product' },
        content_name: 'data.name',
        value: 'data.price',
        currency: { key: 'data.currency', value: 'EUR' },
      },
    },
  },
}

Order complete → Purchase:

order: {
  complete: {
    name: 'Purchase',
    data: {
      map: {
        content_ids: {
          loop: [
            'nested',
            {
              condition: (entity) =>
                typeof entity === 'object' &&
                entity !== null &&
                'entity' in entity &&
                entity.entity === 'product',
              key: 'data.id',
            },
          ],
        },
        content_type: { value: 'product' },
        value: 'data.value',
        currency: { key: 'data.currency', value: 'EUR' },
        num_items: {
          fn: (value, { event }) =>
            event.nested?.filter((entity) => entity?.entity === 'product')
              .length ?? 0,
        },
      },
    },
  },
}

Custom API Destination

Transform to REST API format:

'*': {
  '*': {
    // A computed name goes in policy: a rule's name is a fixed string
    policy: {
      name: { fn: (value, { event }) => `${event.entity}_${event.action}` }, // page_view
    },
    data: {
      map: {
        eventName: 'name',
        eventData: 'data',
        userId: 'user.id',
        sessionId: 'user.session',
        timestamp: 'timestamp',
        metadata: {
          map: {
            consent: 'consent',
            globals: 'globals',
          },
        },
      },
    },
  },
}

Conditional Mapping

Different mapping based on event data:

order: {
  complete: [
    // High-value orders get extra tracking
    {
      condition: (value, { event }) =>
        typeof event.data?.value === 'number' && event.data.value > 500,
      name: 'high_value_purchase',
      data: {
        map: {
          value: 'data.value',
          priority: { value: 'high' },
          notify: { value: true },
        },
      },
    },
    // Standard orders
    {
      name: 'purchase',
      data: { map: { value: 'data.value' } },
    },
  ],
}

Patching Package-Shipped Rules (extend / remove)

@walkeros/transformer-ga4 ships default mapping rules in its settings.mapping. Normally a user rule at the same key replaces the default in full. Use extend or remove to patch instead.

Two layers, two keywords:

  • extend runs at init (config layer): deep-merges a partial rule onto the shipped default. A null value clears an inherited field. Only a package that ships defaults merges it; on any other rule walkeros validate and the bundle reject it (MAPPING_NO_OP).
  • remove runs after evaluation (output layer): strips dotted paths from the final data payload. Applied last, always wins. It also works in a destination's own rules.

A rule with neither keyword replaces the default in full.

{
  "settings": {
    "mapping": {
      "purchase": {
        "extend": {
          "data": { "map": { "affiliation": "params.ep.affiliation" } }
        },
        "remove": ["currency"]
      }
    }
  }
}

This keeps all fields the transformer ships for purchase, adds affiliation, and strips currency from the output.

For the full reference and the two-layer model, see Mapping.Rule docs.

Value[] Fallback Chains

At any value position, an array of values is a fallback chain: the first entry that resolves to a defined value wins.

{
  "item_id": [
    { "key": "data.sku" },
    { "key": "data.id" },
    { "value": "unknown" }
  ]
}

Use this to try multiple source fields before falling back to a constant.

Consent-Gated Fields

Only include PII if consent granted:

user: {
  login: {
    name: 'login',
    data: {
      map: {
        method: 'data.method',
        // Only include email if marketing consent
        email: {
          key: 'user.email',
          consent: { marketing: true },
        },
        // Only include user ID if functional consent
        userId: {
          key: 'user.id',
          consent: { functional: true },
        },
      },
    },
  },
}

Wildcard Patterns

Catch-all for unmatched events:

// Any product action (a computed name goes in policy)
product: {
  '*': {
    policy: { name: { fn: (value, { event }) => `product_${event.action}` } },
    data: 'data',
  },
}

// A click on any entity without its own rule for it: with the product rule
// above, a product click matches product '*' instead
'*': {
  click: {
    name: 'element_click',
    data: {
      map: {
        element_type: 'entity',
        element_id: 'data.id',
      },
    },
  },
}

Source-Side Mapping

A source's mapping uses the same entity-action rules on the events the source pushes, before they reach the collector, for example to rename, edit (policy) or drop them. Turning a raw request format into events is the source's own job (see create-source).

// In a source's config
{
  mapping: {
    product: { click: { name: 'product view' } }, // rename
    test: { '*': { ignore: true } }, // drop every test event
  },
}

Debugging Tips

  1. Event not mapping? Check entity/action match exactly (case-sensitive)
  2. Data missing? Verify source path exists: 'data.nested.field'
  3. Function errors? Add null checks: e.data?.price ?? 0
  4. Array empty? Confirm nested array exists and has items

Related Skills

Reference:

Calling walkerOS MCP Tools from Code

When to Use This Skill

You are an AI agent running inside a code-execution host, not issuing tool calls one at a time through the model. Examples:

  • Claude Code in code-execution / "Code Mode"
  • a Cloudflare Worker or other serverless sandbox
  • an app's server-side sandbox that binds the walkerOS MCP transport

The pattern: bind the MCP transport once, call only the tools you need from code, filter the verbose structuredContent payload in code, and return only the distilled answer to the model. This suppresses tokens that the model never needs to see (full catalogs, per-destination simulation dumps) and avoids a chain of one-by-one tool calls.

This is the "Code Execution with MCP" / "Code Mode" pattern: treat MCP tools as functions you call in a sandbox, not as round-trips through the model context.

The Seven Pure Tools

These tools of the local walkerOS MCP server (@walkeros/mcp) are pure and composable: same input, same output, no auth, no side effects in any account. Safe to call freely from code.

Tool (fully-qualified)PurposeVerbose output worth filtering
walkeros-flow:flow_validateValidate a flow or step config; return issuessmall
walkeros-flow:flow_simulateRun a flow against an event, mock destination callsyes (per-destination calls)
walkeros-flow:flow_bundleBundle a flow config to a JS artifactmedium (bundle/meta)
walkeros-flow:flow_examplesFetch canonical flow/step examplesyes (many examples)
walkeros-flow:flow_loadLoad a flow config, or start a new onemedium (the whole config)
walkeros-flow:package_searchFind packages by type/platform/keywordyes (catalog)
walkeros-flow:package_getRead a package's schemas, hints, examplesyes (full schemas)

To test an event against a flow, use walkeros-flow:flow_simulate: it runs the event through the flow with destination calls mocked, so nothing leaves the sandbox. Do not use walkeros-flow:flow_push for this (see below).

Always use the fully-qualified <server>:<tool> name (Agent Skills best-practice) so the call is unambiguous when multiple MCP servers are bound. walkeros-flow is the name the walkerOS plugin and the documented MCP config give the local server; the hosted walkerOS MCP server is bound as walkeros.

Out of scope for this pattern

walkeros-flow:flow_push sends a real event to real destinations (real API calls to live endpoints), so calling it "freely" would produce real, duplicated sends; use walkeros-flow:flow_simulate to test without sending. The tools for what the walkerOS app stores (projects, saved flows, deployments, secrets, Observe) live on the hosted walkerOS MCP server and act with the person's authorization there. Do not drive these tools from this code-execution filtering pattern; they belong in an interactive, authorized session.

The Recommended Pattern

  1. Bind the transport once. Connect to the walkerOS MCP server at the top of your script, reuse the client for every call.
  2. Call only the tools you need. Don't enumerate; pick the minimal set.
  3. Read structuredContent. Every tool returns an MCP envelope; the typed payload lives in structuredContent (the content text block is for humans).
  4. Filter in code. Reduce verbose payloads to the few fields that answer the question.
  5. Return the distilled answer. Hand the model one line, not the full dump.
// Pseudo-code in a code-execution host. `mcp` is the bound walkerOS client:
// call the raw tools and read `structuredContent` yourself.

const flow = await readFile('flow.json', 'utf8');

// 1. Validate first; bail with one line if the config is broken. Branch on
//    `code`, never on `message`. `strict: true` turns warnings and skipped
//    checks (`details.skipped`) into failures: `valid` is false, as CLI --strict.
const v = await mcp.call('walkeros-flow:flow_validate', {
  type: 'flow',
  input: flow,
  strict: true,
});
const { valid, errors, warnings, details } = v.structuredContent;
if (!valid) {
  const codes = [...errors, ...warnings].map((issue) => issue.code);
  const skipped = (details.skipped ?? []).map((skip) => skip.code);
  return `invalid flow: ${[...codes, ...skipped].join(', ')}`;
}

// 2. Simulate one event per destination: `step` names one target step
//    ("destination.<name>"), and each answer is verbose (every vendor call).
const destinations = Object.keys(
  JSON.parse(flow).flows.default.destinations ?? {},
);
const failed = [];
for (const name of destinations) {
  const sim = await mcp.call('walkeros-flow:flow_simulate', {
    configPath: 'flow.json',
    step: `destination.${name}`,
    event: { name: 'product add', data: { id: 'ers', price: 420 } },
  });
  // 3. Filter in code: keep only destinations that received nothing. The
  //    model never sees the per-destination call dump, only the verdict.
  const received = Object.values(sim.structuredContent.destinations ?? {}).some(
    (d) => d.received,
  );
  if (!received) failed.push(name);
}

// 4. Return one line.
return failed.length
  ? `simulate failed for: ${failed.join(', ')}`
  : 'simulate ok for all destinations';

The same shape applies to walkeros-flow:package_search / walkeros-flow:package_get (filter a catalog or a schema down to the one field the user asked about) and walkeros-flow:flow_examples (pick the one matching example).

Responsibility Boundary

PII handling, secret redaction, and sandbox isolation are the execution host's responsibility, not walkerOS's: walkerOS MCP tools are pure functions over the config and event you pass them.

Related Skills

walkerOS Testing Strategy

Overview

walkerOS uses a layered testing approach with built-in patterns for mocking and documentation sync. This skill ensures tests are reliable, efficient, and maintainable.

Core principle: Test real behavior using the env pattern, link to dev examples, verify before claiming complete.

The Rules

Rule 1: Use env for Mocking, Not Jest

walkerOS has a built-in dependency injection pattern via env in context. This is lighter than Jest mocks, enables documentation generation, and keeps tests in sync with examples.

Wrong:

jest.mock('../ga4', () => ({ initGA4: jest.fn() }));
expect(initGA4).toHaveBeenCalledWith(...);

Right:

import { examples } from '../dev';
import { mockEnv } from '@walkeros/core';

const calls: Array<{ path: string[]; args: unknown[] }> = [];
const testEnv = mockEnv(examples.env.push, (path, args) => {
  calls.push({ path, args });
});

await destination.push(event, { ...context, env: testEnv });
expect(calls).toContainEqual({
  path: ['window', 'gtag'],
  args: ['event', 'page_view', { page_title: 'Home' }],
});

Rule 2: Link Tests to dev Examples

The dev.ts export provides examples.env (the mock env and its simulation paths) and examples.step (the step examples), plus examples.minimal for a destination and examples.createTrigger for a source. Using these in tests ensures documentation stays in sync.

Step Examples on a Real Collector

A destination's step examples run through runExamples, which starts a real collector for each one and compares its out and outcome:

import type { StandardEventName } from '@walkeros/core';
import { runExamples } from '@walkeros/collector/dev';
import { page } from '@walkeros/web-core/dev';
import { destinationMeta } from '..';
import { examples } from '../dev';

// The standard events the destination maps, each with a public example
const claims: StandardEventName[] = [
  'page view',
  'product view',
  'product add',
  'order complete',
];

// The init example takes a config as `in`, not an event
const { init, ...events } = examples.step;

it('runs the step examples', async () => {
  await runExamples('destination', destinationMeta, events, {
    settings: examples.minimal.settings,
    env: { push: examples.env.push, simulation: examples.env.simulation },
    page,
    claims,
  });
});

(packages/web/destinations/meta/src/__tests__/conformance.test.ts runs this test.)

A source's step examples run one test per example with it.each(Object.entries(examples.step)): the test feeds each example's trigger and in through examples.createTrigger and compares what the source hands the collector with out (see packages/server/sources/express/src/__tests__/stepExamples.test.ts).

See using-step-examples for the full lifecycle including the Three Type Zones and naming conventions.

Rule 3: Test Real Behavior, Not Mock Behavior

If you're asserting that a mock was called, you're testing the mock works, not the code.

Red flags:

  • expect(mockFn).toHaveBeenCalled() without verifying the mock produces real effects
  • Assertions on *-mock test IDs
  • Tests that pass when mock is present, fail when removed

Fix: Test what the code actually does. If external APIs must be mocked, verify the real API would receive correct data.

Rule 3b: Prove Delivery, Loss and Failures on a Real Collector

A test or repro that claims an event was delivered, lost or reported as failed starts a real collector with startFlow and mocks only the network or the vendor global (through env), never collector.push. A suite that mocks the collector cannot see loss: a destination that returns early, swallows an error or forgets an await passes it. Assert on what a user sees at default settings: the push result (ok: false, the destination under failed with its reason), the status counters, or the ERROR line the collector wrote.

Failure paths are examples too. A step package's failure modes (a missing required setting, a failing transport, a vendor script that is not loaded yet) are public: false step examples with an outcome (and respond for a vendor answer), not only the happy path. A silent rule and a consent revocation get examples as well.

Use the conformance kit. conformance(kind, step, options) from @walkeros/collector/dev runs a destination, source, transformer or store on a real collector against its step examples and checks the failure contract in one call. The profile states what the step talks to: for destinations http, sdk-env, web-sdk (a web SDK started in init), vendor-global or socket; transformer and source for those kinds. A web step passes page from @walkeros/web-core/dev, the browser it runs on; the browser profiles fail without it. A destination also passes minimal: examples.minimal (from src/examples/minimal.ts, only the settings the schema requires or init refuses to start without): the kit starts the destination with the settings its docs quickstart shows and requires one public example delivered. A package that cannot have a minimal example passes withoutMinimal: '<reason>' instead; with neither, or with an option that disagrees with the file, the test fails. runExamples runs the examples alone.

Examples are the tests. A destination's public examples (one per standard event it maps, from getEvent(name)) are asserted by runExamples with claims; do not restate an example's input and assertion in a second test. Every example runs as published: each function as source text, revived outside its module, the way walkerOS.json ships it (the runner reads that text from the code the test run compiled, not from the built dist/walkerOS.json). A condition that calls an imported helper therefore fails the test, as it would fail in a user's flow. Keep the tests for boundaries, failures and ordering. An example's out is a literal taken from the vendor's documented request, never the output of the code under test, otherwise the test passes by construction. See using-step-examples.

Rule 4: Test First, Watch It Fail

If you didn't see the test fail, you don't know it tests the right thing.

Process:

  1. Write failing test
  2. Verify it fails for expected reason (missing feature, not typo)
  3. Write minimal code to pass
  4. Verify it passes
  5. Refactor if needed

Red flags:

  • Test passes immediately when written
  • Can't explain why test failed
  • "I'll add tests later"

Rule 5: No Test-Only Methods in Production Code

Production classes shouldn't have methods only tests use.

Wrong:

class Session {
  destroy() {
    /* only used in tests */
  }
}

Right:

// In test-utils/
export function cleanupSession(session: Session) { ... }

Rule 6: Verify Before Claiming Complete

"Should pass now" is not verification.

Process:

  1. Run the actual test command
  2. Read the output
  3. Confirm pass/fail count
  4. Only then claim status

When to Use Each Test Type

TypeWhen to AddExample
IntegrationNew usage pattern, new external API interaction, new data flow pathCollector → Destination → gtag()
UnitCombinatorics, edge cases, pure function logicMapping variations, core utilities
ContractBoundary validationDestination output matches vendor API, source input validation

Guideline: Integration tests prove things work when stuck together. Unit tests efficiently cover variations. Contract tests catch API drift.

Simulation Testing

Simulation testing uses the CLI push command with --simulate flags. The collector does not export a simulate() function: simulation is a CLI concern that maps to mock/disabled config properties at runtime.

CLI usage:

# Simulate a destination (mocks its push, captures API calls)
walkeros push flow.json -e '{"name":"page view"}' --simulate destination.ga4

# Simulate a source (captures events it pushes, disables all destinations)
walkeros push flow.json -e '{"content":"","trigger":{"type":"load"}}' --simulate source.browser

# Mock a destination with a specific return value
walkeros push flow.json -e event.json --mock destination.ga4='{"status":"ok"}'

The event needs a name: a destination simulation of an event without one sends nothing and still answers success: true (mapping: none (skipped before mapping), no calls), while a real push of it answers ok: false with Event name is required.

Programmatic usage:

import { simulateDestination, simulateSource } from '@walkeros/cli';

// Simulate a destination: returns one Simulation.Result
const dest = await simulateDestination(
  'flow.json',
  { name: 'page view' },
  { destinationId: 'ga4', ingest: { userAgent: 'Mozilla/5.0' } },
);
// dest.calls = recorded vendor calls ({ fn, args, ts })
// dest.mappingKey = matched rule, e.g. "page view"

// Simulate a source: input is { content, trigger? }
const src = await simulateSource(
  'flow.json',
  { content: { name: 'page view' } },
  { sourceId: 'browser' },
);
// src.events = events captured from source env.push

simulateTransformer and simulateCollector follow the same shape. The CLI push --simulate path collects these results into PushResult.simulations and prints them with formatPushResult; programmatic push() does not simulate.

Key points:

  • --simulate destination.X starts only the target destination (no source, no other destination); the flow's transformers still start
  • --simulate source.X starts only that source and captures its events at the collector boundary; they reach no destination. A source package declaring examples.env.simulation (SQS, Pub/Sub pull) runs on its mock client, calls recorded
  • Simulate options: --consent (starting collector consent), --command (destination runs a collector command instead of a push), --page-url (web source page), --ingest (request context). Programmatic: simulateDestination(config, event, { destinationId, consent, command }), simulateSource(config, input, { sourceId, pageUrl })
  • Destination /dev env.push is auto-loaded to provide mock globals (fake window.gtag, a mock SDK client, a mock sendServer), resolved per export (exportExamples). Without one, simulate refuses a package destination before the flow starts instead of calling the real vendor (an inline code step has no package and runs as given). A flow store's mock env is injected when its package ships one; a store without one runs for real.
  • Returns PushResult with simulations (one Simulation.Result per simulated step: events, calls, mappingKey?, skipped?, error?; skipped says why a destination sent nothing: pending on its require, or a consent skip)
  • Trace-mode limit: simulate injects the mock env before init, so a client built in init is recorded. Runtime trace mode wraps the env per push only (collector/src/destination.ts), so calls through a client built in init are not recorded there.
  • The mockEnv() and env pattern examples above remain correct for unit testing individual step functions directly

Package-Specific Approaches

PackageApproach
coreUnit tests only - pure functions, no env needed
collectorIntegration tests critical - input/output consistency is paramount
browser sourceMaintain walker algorithm coverage
web destinationsIntegration tests per unique pattern + unit tests for mappings, use env pattern
server destinationsSame as web destinations
cli/dockerIntegration tests for spawn behavior, explore dev pattern to reduce duplication
sourcesContract tests for input validation, integration tests for event capture

The env Pattern Deep Dive

How env Works

Each destination/source defines an env type that specifies external dependencies:

// Destination-specific env type
export interface Env extends DestinationWeb.Env {
  window: {
    gtag: Gtag.Gtag;
    dataLayer: unknown[];
  };
  document: {
    createElement: (tagName: string) => HTMLElement;
    head: { appendChild: (node: unknown) => void };
  };
}

mockEnv() Function

The mockEnv() function from @walkeros/core creates a Proxy that intercepts all function calls:

import { mockEnv } from '@walkeros/core';

const calls: Array<{ path: string[]; args: unknown[] }> = [];
const testEnv = mockEnv(examples.env.push, (path, args) => {
  calls.push({ path, args });
  // Optionally return a value
});

// Now use testEnv in your destination context
await destination.push(event, { ...context, env: testEnv });

// Assert on captured calls
expect(calls).toContainEqual({
  path: ['window', 'gtag'],
  args: ['event', 'purchase', expect.objectContaining({ value: 99.99 })],
});

Rules for a package's env

  • init keeps an injected dependency. Every external dependency (SDK client, constructor or module, fetch, sendServer, auth client) is reached through env, and init creates one only when none is injected: env?.x ?? (await createX(...)). Overwriting an injected client means tests and simulate hit the real vendor.
  • Every mock env key is read. Each key a package's examples.env.push declares must be read as env.<key> somewhere in the package's src; otherwise the mock is dead and the real module runs (jest may pass only through __mocks__).
  • simulation names the request call and is pinned with expectSimulationResolves(examples.env) from @walkeros/core/dev (see using-step-examples).

dev.ts Structure

Each package with external dependencies should have:

// src/dev.ts
export * as schemas from './schemas';
export * as examples from './examples';

// src/examples/index.ts (a destination)
export * as env from './env';
export * as step from './step'; // Step examples { in, out }
export { minimal } from './minimal';

Testing Sources with Injected env

Sources accept platform dependencies via env. Mock window, document, or library imports by passing them through env instead of mocking globals.

Type the mock against the source's own Env type, not against the global Window. A source that narrows Env.window to only the members it touches, as the Cookiebot source does with CookiebotWindow, takes a mock of that shape directly with no as unknown as Window cast:

import type { Elb } from '@walkeros/core';
import { createMockLogger } from '@walkeros/core';
import type { CookiebotWindow, Types } from '../types';

// Instead of mocking the global window:
const mockWindow: CookiebotWindow = {
  Cookiebot: {
    consent: { marketing: true, method: 'explicit' },
    hasResponse: true,
  },
  addEventListener: jest.fn(),
  removeEventListener: jest.fn(),
};

const elb: Elb.Fn = jest.fn();
const env: Types['env'] = {
  push: elb,
  command: elb,
  elb,
  window: mockWindow,
  logger: createMockLogger(),
};

This pattern avoids global state pollution between tests and enables simulation in non-browser environments. Most sources type Env.window as the full Window & typeof globalThis; the mock stays cast-free only when the source narrows it to the members it uses (the same declare-global + narrowed-Env approach destinations use, see create-destination, Phase 7, Key Patterns 3). Server sources take their platform modules the same way (env.express ?? express).

Red Flags - Stop and Fix

  • Using jest.mock() for internal modules when env pattern is available
  • Tests that don't import from ../dev
  • Assertions only checking mock call counts
  • Proving delivery or loss with a mocked collector.push instead of startFlow
  • A destination without a conformance test, a silent example or a failure example, or a failure example without a description
  • Tests with extensive mock setup (>50% of test is setup)
  • Test-only methods added to production classes
  • Claiming tests pass without running them

Commands

Verification tiers (run from the repository root):

# L1, the default during a task: typecheck, lint, test for the touched package
npm run verify:touched -- core
npm run verify:touched -- web-destination-gtag

# L2, at plan completion: only packages affected since origin/main
git fetch origin main --depth=1
npm run verify:affected

# L3, before pushing or marking PR-ready: critical path + affected
npm run test:smoke

# Single test file (still useful while iterating)
cd packages/<name> && npm run test -- path/to/file.test.ts

# Watch mode (single package)
cd packages/<name> && npm run test -- --watch

Avoid bare npm run test at root inside per-task steps. That is L4 (full suite, 10-15 min) and is reserved for plan completion when the plan touched shared infra, or for explicit user request.

Reading a Step Back from the Collector

When a test calls a step's raw push directly through the collector bag (collector.sources.X.push, collector.destinations.X.push, etc.), use the typed accessors from @walkeros/core instead of casting:

import { Source } from '@walkeros/core';

const src = Source.getSource<TestSourceTypes>(collector, 'testSource');
await src.push({ method: 'GET', path: '/api/data' });

Available helpers: Source.getSource, Destination.getDestination, Transformer.getTransformer, Store.getStore. Each accepts an optional type parameter to recover the per-step generic that the bag's index signature erases on read. Each throws <Kind> not found: <id> when the id is unknown.

Do not write collector.sources.X.push as any or collector.sources.X.push as (rawData: ...) => Promise<...>. The accessor exists exactly to remove that boundary cast.

Shared Destination Test Harness

The conformance kit (conformance and runExamples from @walkeros/collector/dev) is the shared harness for destination tests: the failure contract, missing settings, the minimal example and every step example.

When writing a new destination test:

  • Keep destination-specific behavior (vendor-specific batch shapes, ordering, edge cases the examples cannot state) in the test file.
  • Mirror the shape of an existing destination of the same family (web vs server), for example packages/web/destinations/meta/src/__tests__/ or packages/server/destinations/pinterest/src/__tests__/.
  • Do NOT invent a new private harness inside the destination. If the kit does not cover a case, flag it.

Related Skills

Reference:

Understanding walkerOS Destinations

Overview

Destinations receive processed events from the collector and deliver them to third-party tools (analytics, marketing, data warehouses).

Core principle: Destinations transform and deliver. They don't capture or process: that's sources and collector.

Destination Interface

See packages/core/src/types/destination.ts for canonical interface.

MethodPurposeRequired
init(context)Load scripts, authenticateOptional
push(event, context)Transform and send eventRequired
pushBatch(batch, context)Batch processingOptional
destroy(context)Cleanup on shutdownOptional
configSettings, mapping, consentRequired

destroy?: DestroyFn: optional cleanup method. Called during command('shutdown'). Use to close DB connections, flush buffers, or release SDK clients. Receives { id, config, env, logger }.

The env Pattern

Destinations use dependency injection via env for external APIs. This enables testing without mocking.

// Destination defines its env type
export interface Env extends DestinationWeb.Env {
  window: {
    gtag: Gtag.Gtag;
    dataLayer: unknown[];
  };
}

// Destination uses env, not globals
async function push(event, context) {
  const { env } = context;
  env.window.gtag('event', mappedName, mappedData);
}

Testing with env

REQUIRED SKILL: See testing-strategy for full testing patterns.

import { mockEnv } from '@walkeros/core';
import { examples } from '../dev';

const calls: Array<{ path: string[]; args: unknown[] }> = [];
const testEnv = mockEnv(examples.env.push, (path, args) => {
  calls.push({ path, args });
});

await destination.push(event, { ...context, env: testEnv });

expect(calls).toContainEqual({
  path: ['window', 'gtag'],
  args: ['event', 'purchase', expect.any(Object)],
});

The env as observability seam

The env pattern is also the interception seam for trace-level vendor-call capture. At trace level the collector may attach an observe recorder under the observe key of the env, and @walkeros/web-core's getEnv wraps the callables the destination declared as observable, recording each call. A destination that reaches its vendor through getEnv(env) is automatically observable; one that reaches a global directly, bypassing env, is not. This is one more reason to route every external call through env.

Rules that keep the seam intact:

  • init keeps an injected env. A client built in init (SDK client, auth client, writer) is taken from env when present and created only otherwise (env?.BigQuery ?? BigQuery). Every key the mock env declares must be read as env.<key> in src.
  • simulation names the request call (sendServer, call:PubSub.topic.publishMessage, call:JSONWriter.appendRows), not a constructor. The one recorder is observeEnv in @walkeros/core; it navigates constructors, factories and Promises, and records only the leaf.
  • Trace-mode limit. CLI and MCP simulate inject the mock env before init, so a client built in init is recorded. Runtime trace mode wraps the env per push only, so calls through a client built in init are not recorded there.
  • Client IP and user agent. Server conversion-API destinations (Meta, TikTok, Snapchat, Pinterest, Bing, Reddit, Criteo, X, Data Manager, Piwik PRO) expose ip and userAgent settings (Mapping.Value | false), defaulting to ['ingest.ip', 'event.user.ip'] and ['ingest.userAgent', 'event.user.userAgent'], resolved with getMappingValue(createMappingRoot(ingest, event), setting, { collector }). false disables, an explicit mapped vendor field wins, each package carries its own few lines (no shared helper).

Destination Config

config: {
  settings: { /* destination-specific */ },
  mapping: { /* event transformation rules */ },
  data: { /* global data mapping */ },
  consent: { /* required consent states */ },
  policy: { /* processing rules */ },
  queue: boolean,  // when added later, receive the run's earlier events (default true)
  timeout: number, // delivery timeout in ms (default 10000)
  queueMax: number,  // consent-queued events cap (default 1000)
  dlqMax: number,    // dead-letter queue cap (default 100)
}

Buffer bounds

Each destination keeps two internal buffers: queuePush (consent-denied events) and dlq (failed pushes). Both are size-bounded with FIFO drop-oldest eviction. Defaults: queueMax: 1000, dlqMax: 100. Set either on a destination's config to override per destination. Operators read drop counts from collector.status.dropped[stepId('destination', id)]?.queue (consent-denied evictions) and ?.dlq (DLQ evictions). Build the key with stepId() from @walkeros/core. Point-in-time sizes stay on collector.status.destinations[id].queuePushSize / dlqSize.

queuePush drains back into the destination once consent allows delivery. The DLQ is never replayed: an event that reaches it is gone once it is evicted.

Delivery timeout

config.timeout bounds one delivery. A push or pushBatch that has not settled within it is abandoned, its events are routed to the DLQ and failed is incremented. The default is 10000 ms, applied whenever timeout is 0 or unset, so one slow destination never wedges the collector push.

A destination's own client timeout has to fit inside it, retry delays included. Many SDK defaults are longer (30 s in several HTTP clients). With a longer one, the collector stops waiting at 10 s, dead-letters the batch and counts it lost, then the request lands anyway a few seconds later. Those events are both counted lost and actually written. Derive the client's per-attempt timeout from config.timeout minus what the retry delays can take, split across the attempts, so the last attempt finishes before the race does.

The same bound applies at shutdown, where the collector races each pending batch flush against a fixed 5000 ms step timeout before destroying the destination. A destination whose retries need longer than that loses them on shutdown.

Outcomes and failure reasons

A push ends in one place per destination in the PushResult: done, queued (reason consent, batch or held) or failed. A failed entry carries a reason { code, message, retryable, status? } that the collector derives from the thrown value: a StepError keeps its code; a TypeError, RangeError or ReferenceError is runtime (not retryable), except Node's fetch failed with a cause.code; any other throw is transport. A failed browser fetch is a TypeError too, so throw network failures as a StepError (sendWeb and sendServer do). Codes the collector assigns itself: timeout, not-ready (a destination whose init could not start or returned false), breaker-open, blocked (held past one retry window), queue-full, mapping (a rule condition threw), transformer. Codes a destination throws: config, missing-input, vendor-not-loaded, vendor-not-ready, vendor-timeout, transport. retryable decides whether a server source asks the producer to resend (acknowledge(result) from @walkeros/core: ack, retry or reject); it never reads message text.

Unmet consent is never a failure. A destination waiting for consent answers queued with the reason consent; a server flow built by the CLI skips a consent-denied event instead (consentDenied: 'skip', counted in status.destinations[id].skipped, a debug line). A rule with ignore: true is a skip as well, not a delivery.

Failed means thrown. A destination reports a failure only by throwing a StepError; it never returns early, never returns false from init, never logs the failure itself, and awaits every call it makes. The collector writes one ERROR line per (step, code, status) and minute. Every destination honours rule.silent (no event of its own, side effects still run) and ships a silent example.

The conformance kit. conformance('destination', destination, options) from @walkeros/collector/dev runs a destination on a real collector against its step examples and fails with every broken item of the contract at once. Pick the profile by what the destination talks to: http, sdk-env, web-sdk (a web SDK started in init), vendor-global, socket. A web destination also passes page from @walkeros/web-core/dev. Every destination test passes minimal, the package's examples.minimal published from src/examples/minimal.ts (a package with several exports passes exportName and exportExamples, one src/<export>/examples/minimal.ts each), or withoutMinimal: '<reason>' when the package publishes none. The kit composes the minimal example into the quickstart the docs page shows and requires it delivered. A test with neither option fails, and so does one whose option disagrees with the published file. See create-destination.

Batch scheduling

Set config.batch on a destination (with pushBatch implemented) to batch all of its events into one shared buffer. No '* *' wildcard mapping rule is needed. The configuration shape is batch?: number | { wait?, size?, age? } at both the destination-config layer and the mapping-rule layer.

pushBatch is the gate. The collector engages batching only when destination.pushBatch exists. Set config.batch on a destination that lacks it and every event is sent one at a time; the collector logs one ERROR at registration naming the setting, and walkeros validate and the bundle fail (BATCH_WITHOUT_PUSH_BATCH) for inline code. When adding batching to a destination, implementing pushBatch is the change; config.batch alone does not batch. A batched event answers queued with the reason batch, so a server source acknowledges the producer at queue time; a failed flush is visible (ERROR line, failed counter, DLQ) but is not redelivered. walker flush flushes every pending batch.

  • wait (ms): debounce window. The timer resets on every push. A bare number, batch: 1000, is shorthand for { wait: 1000 }.
  • size: hard count cap. Default 1000. Flushes immediately when reached.
  • age (ms): hard age cap since the first entry of the current window. Default 30000. Prevents debounce starvation under sustained load.

A mapping rule's own batch splits that entity-action into its own buffer and overrides config.batch per field (rule ?? config ?? default). To batch only specific events, omit config.batch and set batch on those rules. Pending batches flush on shutdown.

Per-event metadata

Batch.entries[] carries per-event { event, ingest?, respond?, rule?, data? }. Destinations that need per-event request IDs or HTTP responses (BigQuery, mParticle, HubSpot) should read entries instead of assuming all events in a batch share one ingest. batch.events and batch.data are derived views of entries.

Failure handling

pushBatch has three return outcomes:

  • Resolves void: the whole batch succeeded; every entry counts as delivered.
  • Throws or rejects: the whole batch failed. Every entry is routed to the destination's dlq and status.destinations[id].failed is incremented by the batch size.
  • Resolves a BatchOutcome: partial failure. { failed: [{ index, error? }] } names the entries that did not make it, by index into batch.entries. Only those are DLQ'd; the rest count as delivered. Use this when the vendor reports per-row results (BigQuery's rowErrors is the reference case), so a later DLQ retry does not rewrite rows that already landed.

A BatchOutcome is for sinks that report per row, and for nothing else. A sink that succeeds or fails as a whole (one SQL INSERT, one file write, one publish) must never report an all-failed outcome in place of throwing. Only a throw can count as a failure on the breaker (a retryable one, such as a network error, a timeout or a 429 or 5xx answer), and only a flush with at least one delivered entry records a success, so an outcome where nothing succeeded records neither: a total outage passes the breaker unseen and it never opens. A whole-batch sink has exactly two answers: resolve void or throw.

Batching, the delivery timeout, the DLQ and the breaker are the collector's; per-call retry is the vendor SDK's (BigQuery, Kafka, HubSpot and the AWS SDK each have their own backoff semantics). Where a destination relies on that SDK retry, it keeps the attempts and their delays inside config.timeout, and makes a repeat safe through an idempotency key or a deduplication token computed once and reused across attempts. Without that, a retry of a delivery the vendor had already accepted duplicates data. Counters (count, out) are bumped only after a successful flush.

Three rules every destination follows. First, no retry loops, backoff or DLQ code of its own: each request is sent once through the vendor SDK, whose built-in retry stays at its defaults, because a standard retry belongs in the collector, not in each package. Second, a server destination's init is offline: it validates the config and builds clients, but makes no network call, loads no credentials and creates no remote resources; connections (Kafka, BigQuery, S3, Sheets, SQLite) open at the first use, memoized and retried after a failure, and provisioning belongs to setup. Third, every failure, thrown or reported in a BatchOutcome, is a StepError that names the cause (and, for a configuration error, the fix) and carries code (the vendor error name or a package code), status (the HTTP status when the vendor answered) and retryable. Take retryable from the vendor SDK's own public classifier where one exists, else from documented codes or statuses, and keep the SDK error reachable as cause. @walkeros/server-destination-aws is the reference implementation.

Operators also see status.destinations[id].inFlightBatch: the number of events buffered but not yet delivered.

Out-of-band errors from an EventEmitter SDK

tryCatchAsync only catches errors on the awaited push/pushBatch path. If a destination owns an SDK object that is an EventEmitter (a BigQuery StreamConnection, a Redis client, a Kafka producer), that object can emit 'error' on a detached tick with no awaiter. An EventEmitter that emits 'error' with zero listeners throws synchronously and crashes the process.

A destination that owns such an object MUST attach an 'error' listener and route the error through context.reportError, which is available on every step context:

  • context.reportError(err) (no event): a connection-level error between pushes. Logs (redacted) and bumps status.connectionErrors[stepId]. Does not count as a failed event.
  • context.reportError(err, event): a specific event was lost. Routes it to the DLQ and bumps failed, exactly like an in-band push failure.

reportError is guarded and must never be called in a way that throws back into the emitter tick. The Pub/Sub pull source (sources/gcp/src/pubsub/pull) is the reference for attaching an 'error' listener. The runner's process guards are a backstop, but the listener is what gives clean DLQ routing, attribution, and redaction. Optionally pair this with the breaker config so a persistently broken transport stops retrying every event.

Require vs Consent

Two separate mechanisms control when destinations receive events:

MechanismPurposeScopeEffect
requireDelay initializationWhole destinationDestination stays in pending until all required events fire (e.g., walker consent)
consentFilter eventsPer-eventEvents without matching consent are queued (skipped in a server flow built by the CLI)

Require gates the destination lifecycle. A destination with require: ["consent"] does not exist in the collector until consent state is present: a "walker consent" event, or starting consent (startFlow's consent, which walkeros push --simulate sets with --consent). Until then, events are queued internally. require takes effect only in the step definition's config (destinations.<id>.config in flow.json, { code, config } in startFlow); a require in the destination code's own config is ignored today, and the destination receives events at once.

Consent gates individual event delivery. A destination with consent: { marketing: true } only receives events where the collector's consent state (or the event's own consent field) includes { marketing: true }.

State on flush (consent queue): When consent is granted, the consent-queued events are delivered in their original order. Each one gets the current consent, its user is merged with the collector's (the event's own values win), and it keeps the globals it was captured with. Events that still lack the required consent stay queued. See How queued events are processed. Events replayed from the run's buffer, to a require-pending destination when it activates or to a destination added later, behave differently today: they carry the user set after they were captured, and only an event that brought its own user keeps its values.

Both can be combined:

{
  "config": {
    "require": ["consent"],
    "consent": { "marketing": true }
  }
}

This means: don't initialize until consent fires, then only accept events with marketing consent. The state that satisfies require reaches on() before the destination's queued events are pushed.

Simulation impact: a destination whose require is not met is reported as skipped with the reason pending by flow_simulate and walkeros push --simulate. Pass the starting consent to simulate it: state.consent in flow_simulate, --consent on the CLI.

Policy

Policy shapes the event before its data is mapped; the rule was selected earlier, as the event entered the destination. Defined at config level (applies to all events) or rule level (applies to specific events):

{
  "config": {
    "policy": {
      "user_data.email": {
        "key": "user.email",
        "consent": { "marketing": true }
      }
    }
  }
}

Policy supports consent-gated field injection: fields are only added when the required consent is present in the event.

Destination Paths

TypePathExamples
Webpackages/web/destinations/gtag, meta, api, piwikpro, plausible
Serverpackages/server/destinations/aws, gcp, meta

Template Destination

Use as starting point: packages/web/destinations/plausible/

Transformer Wiring

Destinations can wire to their own transformer chain via the before property. It runs for this destination only, after the collector chain (collector.next, shared by every destination); a stop in it skips only this destination:

destinations: {
  gtag: {
    code: destinationGtag,
    before: 'redact'  // First transformer to run before this destination
  }
}

The transformer chain runs after collector enrichment, before the destination receives events. Each destination can have its own chain. See understanding-transformers for chain details.

Consent: Two-Layer Pattern

Destinations that integrate vendor SDKs typically need two consent layers:

Layer 1: config.consent gates walkerOS event delivery. If consent is not granted, events don't reach the destination. This is the primary barrier.

Layer 2: on('consent') controls vendor SDK internals. Even when walkerOS stops sending events, the vendor SDK may still run its own behaviors (DOM capture, polling, fetching configs). Use on('consent') to pause/resume these.

import { getGrantedConsent } from '@walkeros/core';

on(type, { config, collector }) {
  if (type !== 'consent' || !config.consent) return;
  // The rule of the delivery gate: one granted required key is enough.
  // Derive from config.consent, don't hardcode consent names.
  const granted =
    getGrantedConsent(config.consent, collector.consent) !== false;
  vendorSdk.setOptOut(!granted);
}

Complete consent compliance needs both layers: config.consent prevents data flow, on('consent') prevents vendor SDK side effects.

on('consent') runs before the first push on every activation path (require, walker destination at runtime, state passed to walker run, state set before run) and is delivered exactly once per state change. It may be async: the collector awaits it, bounded by config.timeout, and events pushed meanwhile answer queued with the reason held. A handler that rejects or does not settle in that time writes an ERROR line (consent not applied) and holds the destination's events for one more config.timeout. A push after that window retries a failed delivery; once a delivery succeeds, the held events are delivered. If none has succeeded by the first push after the window, the held events and every later event with the required consent fail with the reason blocked until one does: the push answers ok: false, they are counted in failed and kept in the DLQ, an ERROR Push failed line is written, and they are not sent when the destination recovers. A handler that never settles keeps the destination blocked. Events still held at the next walker run are discarded. Lifecycle handlers such as on('run') are awaited too, bounded by the same timeout, and never hold events. elb commands and startFlow resolve only after the destinations' on() handlers for that command have settled or timed out. See Gate a destination with consent.

Response Delegation (env.respond)

Destinations can customize HTTP responses by calling context.env.respond?.({ body, status?, headers? }). This is useful for destinations that need to signal success/failure back to the HTTP caller. First call wins (idempotent). The respond function is optional, only present when the source provides one.

Setup (optional)

Destinations can implement an optional setup() lifecycle to provision external resources, for example a BigQuery dataset and table, a Pub/Sub topic, or a warehouse schema. Setup is never invoked by the runtime, push, init, or deploy. It runs only when an operator explicitly types walkeros setup destination.<name>.

The signature is (ctx: LifecycleContext<Config<T>, Env<T>>) => Promise<unknown>, where LifecycleContext carries { id, config, env, logger }. Idempotency is the package's responsibility: the framework adds no opinion. Use resolveSetup(ctx.config.setup, DEFAULTS) from @walkeros/core to normalize the boolean | object shape into a concrete options object.

See walkeros-create-destination, walkeros-understanding-sources, walkeros-understanding-stores, and the walkeros setup CLI documentation for the authoring template and operator workflow.

Related Skills

Source Files:

Package READMEs:

Documentation:

Understanding walkerOS Development

Overview

walkerOS follows extreme programming principles with strict conventions. Read this skill before writing any code.

Core principle: DRY, KISS, YAGNI. Test first. Verify before claiming complete.

Commands

CommandPurpose
npm installInstall all dependencies
npm run devWatch mode for all packages
npm run buildBuild all packages
npm run formatPrettier formatting

Verification cheatsheet:

  • While iterating (L1, touched package): npm run verify:touched -- <pkg>
  • At plan completion (L2, affected since origin/main): npm run verify:affected

XP Principles (Non-Negotiable)

PrincipleIn Practice
DRYUse @walkeros/core utilities, don't reimplement
KISSMinimal code to solve the problem
YAGNIOnly implement what's requested
TDDTest first, watch it fail, then implement
No anyNever use any in production code (tests are exception)

Folder Structure

packages/
├── core/           # Platform-agnostic types, utilities, schemas
├── collector/      # Central event processing engine
├── config/         # Shared config (eslint, jest, tsconfig, tsup)
├── cli/            # walkeros CLI (validate, bundle, push, setup)
├── runner/         # Runs prebuilt flow bundles
├── mcps/           # MCP servers (mcp, source-browser)
├── transformers/   # Neutral transformers: validate, ga4, demo
├── web/
│   ├── core/       # Web-specific utilities
│   ├── sources/    # browser, dataLayer, session, cmps/
│   └── destinations/ # gtag, meta, api, piwikpro, plausible, ...
└── server/
    ├── core/       # Server-specific utilities
    ├── sources/    # express, fetch, aws, gcp
    ├── destinations/ # aws, gcp, meta, ...
    ├── transformers/ # bot, fingerprint, file
    └── stores/     # fs, s3, gcs, sheets

apps/
├── walkerjs/       # Zero-config browser file for GTM
├── quickstart/     # Code examples (source of truth for patterns)
├── explorer/       # React components for the docs and exploration
├── cli/            # The walkeros npm package (CLI entry)
├── storybook-addon/ # walkerOS Storybook addon
├── scripts/        # Repository scripts (docs and skills checks)
└── demos/          # Demo applications

Core Package Usage

Always import from @walkeros/core:

// Types
import type { WalkerOS } from '@walkeros/core';

// Utilities
import {
  getEvent,
  createEvent, // Event creation
  getMappingEvent,
  getMappingValue, // Transformations
  isString,
  isObject,
  isDefined, // Type checking
  assign,
  clone, // Object operations
  tryCatch,
  tryCatchAsync, // Error handling
} from '@walkeros/core';

Config package for shared tooling:

  • ESLint config: @walkeros/config/eslint
  • Jest config: @walkeros/config/jest
  • TSConfig presets: @walkeros/config/tsconfig/{base,web,node}.json
  • Tsup config: @walkeros/config/tsup

Editing core Config types

Core component configs live in two places:

  • TS interface: packages/core/src/types/{destination,source,transformer,store,collector}.ts
  • Zod schema: packages/core/src/schemas/{destination,source,transformer,store,collector}.ts

Both are hand-written and mirror each other. TS stays authoritative because Zod's inferencer collapses recursive types (Routes, MatchExpression, Value) to unknown. Zod drives runtime validation, JSON Schema emission, and website Configuration reference tables.

When adding, renaming, or removing a Config field, update BOTH files. A compile-time drift guard at packages/core/src/schemas/__tests__/config-drift.test-d.ts fails tsc if the key sets diverge. The guard checks keys only; value types may differ (recursion, generic slots). Run npm run verify:touched -- core to verify.

Error visibility contract

Top-level boundaries in the collector (createPush in push.ts, createCommand in command.ts) wrap their inner pipeline in tryCatchAsync. The onError callback MUST do two things:

  • log one structured ERROR line with the context an operator needs to find the cause, and no payload: the event name for push, the command name for command, never the event, the ingest or the command's data, which can carry personal data,
  • increment collector.status.failed.

An empty onError is a defect: it swallows the exception, returns { ok: false } silently, and leaves the operator blind. Use packages/collector/src/push.ts and packages/collector/src/command.ts as the canonical pattern.

Two categories of caught error:

  1. Internal walkerOS pipeline failures (push, command, a throwing rule condition in push.ts, source factory / init / queueOn flush in source.ts, transformer init in transformer.ts, destination init in destination.ts): log AND status.failed++.
  2. User-supplied callbacks (a mapping value's condition / fn / validate, on subscriptions in on.ts): log only. status.failed stays a pipeline-health signal; user-code visibility goes via logs.

For invariant violations or operator-initiated aborts that must crash the host process, throw FatalError (exported from @walkeros/core). FatalError bypasses every boundary catch in both categories so a supervisor can terminate cleanly. Standard Error is absorbed, logged, and (for category 1) counted.

The log message verb identifies the site: 'mapping condition failed', 'source factory failed', 'transformer init failed', 'on callback failed', etc. Operators grep for the verb. No kind field is required except in on.ts, where seven sites share one verb and disambiguate via a typed kind field on the structured payload.

Platform boundaries

Every package is one platform: web, server, or neutral (usable on both, like @walkeros/core and @walkeros/collector). Neutral code ships inside walker.js and runs on a server too, so it uses neither platform's APIs.

  • Where it is declared. A step package sets walkerOS.platform in its package.json: ["web"], ["server"], or both for neutral. Every other package and published app is in the declared table of packages/config/scripts/check-platform.mjs. A package declared nowhere fails the check.
  • Presets. base.json is neutral (lib: es2020), web.json adds dom, node.json is es2022. All set types to node and jest, so no other @types package can bring the DOM back in. A package tsconfig may add only an allowlisted ambient vendor type (gtag.js, facebook-pixel).
  • What each may use. Web: browser globals, no Node globals or builtins (Buffer, process, fs, node:*). Server: Node, no browser globals. Neutral: neither, and no globalThis, no jsdom import and no /// <reference lib or types> in any file, tests included. ESLint holds these lines where TypeScript cannot: @walkeros/config/eslint/web, /node (its nodePlatform block) and /neutral.
  • API floor. Neutral and web code may use ES2020 APIs at most; ES2021 and later (for example the cause argument of new Error) fail typecheck, so errors assign cause themselves.
  • Needing the other platform. A neutral package that needs something only the web has defines a small interface in @walkeros/core (a port), web-core implements it, and the web caller passes it in explicitly. Examples: ObserveSlot (pageObserveSlot() from @walkeros/web-core) and Flow.StepPage (page from @walkeros/web-core/dev). Node-only code of core lives in @walkeros/core/node.
  • Check. npm run check:platform from the walkerOS root fails for any drift in a preset, lib, types, eslint config or declaration. CI runs it on every run, outside the turbo cache.

Testing

REQUIRED SKILL: Use testing-strategy for detailed testing patterns.

Quick reference:

  • Use env pattern for mocking (not Jest mocks)
  • Import from dev.ts for examples
  • Test first, watch it fail
  • Verify before claiming complete

Related Skills

Source Files:

Understanding walkerOS Events

Overview

walkerOS events are self-describing, stateless, vendor-agnostic data structures. They capture user interactions in a standardized format that can be transformed for any destination.

Core principle: Events describe what happened, not where it goes. They are stateless, self-describing and industry-agnostic.

Entity-Action Naming (Critical)

STRICT REQUIREMENT: All events use "entity action" format with space separation.

// Correct
'page view';
'product add';
'order complete';
'button click';

// Wrong
'page_view'; // underscore
'pageview'; // no separator
'purchase'; // no entity
'add_to_cart'; // wrong format

Parsing: const [entity, action] = event.split(' ')

  • Entity: Noun (page, product, user, order, button)
  • Action: Verb (view, add, complete, click, login)

Event Properties

See packages/core/src/types/walkeros.ts for canonical types (Event interface, plus event helpers in event.ts).

PropertyTypePurposeExample
namestring"entity action" format"product view"
dataobjectEntity-specific properties{ id: "P123", price: 99 }
contextobjectState/environment info (optional){ stage: ["checkout", 1] }
globalsobjectGlobal properties{ language: "en" }
userobjectUser identification{ id: "user123" }
nestedarrayRelated entities (optional)[{ entity: "product", data: {...} }]
consentobjectConsent states{ marketing: true }
idstringW3C span_id, 16 lowercase hex chars, generated by collector"0123456789abcdef"
timestampnumberAuto-generated Unix ms1647261462000
entitystringParsed from name"product"
actionstringParsed from name"view"
source.typestringSource kind (browser, dataLayer, session, ...)"browser"
source.platformstringRuntime platform (web, server)"web"
source.schemastringEvent-model version, set by the collector when no source"4"
source.tracestringRun-scoped W3C trace_id, shared by every event of a run"0123...cdef" (32 hex)
source.countnumberPer-run emission sequence (1, 2, 3, ...)1
source.releaseobjectPer-flow config release map, accumulates across crossings{ web: "3", server: "5" }

data Property

Entity-specific properties. Schema-free but consistent within entity type.

// product entity
data: { id: "P123", name: "Laptop", price: 999, currency: "USD" }

// page entity
data: { title: "Home", path: "/", referrer: "https://..." }

context Property

Hierarchical state information. Format: { name: [value, order] }. Optional.

context: {
  stage: ["checkout", 1],    // checkout stage, first step
  test: ["variant-A", 0],    // A/B test variant
  group: ["premium", 2]      // user segment
}

globals Property

Properties the collector merges into every event it creates.

globals: {
  language: "en",
  currency: "USD",
  environment: "production"
}

nested Property

Related entities captured together. Optional.

// Order with line items
nested: [
  { entity: 'product', data: { id: 'P1', quantity: 2 } },
  { entity: 'product', data: { id: 'P2', quantity: 1 } },
];

user Property

User identification across sessions.

user: {
  id: "user123",        // Your user ID
  device: "device456",  // Device ID
  session: "sess789",   // Session ID
  hash: "a1b2c3...",    // Hashed identifier (for example a hashed email)
  optout: false,        // User opted out of tracking (boolean)
  segment: "vip"        // Custom keys are allowed, each holds a property value
}

The declared fields keep their types: email must be a valid address, so a hashed email goes to hash, never to email. Any other key is a custom user attribute with a property value and passes the validate transformer's format: true check.

source Property

Where the event came from. The collector enriches the event with source data based on which source emitted it.

source: {
  type: 'browser',         // source kind (browser, dataLayer, session, ...)
  platform: 'web',         // runtime: 'web' or 'server'
  version: '3',            // external source-emitter version (the collector does not set it)
  schema: '4',             // event-model version (the collector sets it on an event without a source)
  trace: '0123...cdef',    // run-scoped W3C trace_id, groups all events of a run
  count: 1,                // per-run emission sequence (1, 2, 3, ...)
  release: { web: '3' },   // per-flow config release map, accumulates web to server
  url: 'https://...',      // page URL (set by the browser, dataLayer and session sources)
  referrer: '...',         // page referrer (same sources)
  tool: 'cli',             // tool that produced the event (optional)
  command: 'simulate',     // command name (optional)
}

Steps may add their own keys to source (the validate transformer writes valid, GA4 decoding adds pageLoadId and hitSequence); each holds a property value.

The browser, dataLayer and session sources set source.url and source.referrer from the page. CMP sources emit walker consent commands, not events.

source.trace is the run grouping key: the collector mints a fresh run-scoped trace_id on each run and stamps it (plus a per-run source.count) on every event when absent. It is preserved unchanged when an event is forwarded from web to server, so the whole pipeline shares one trace.

source.release records which config handled the event, keyed by flow name. Each flow the event passes through adds its own entry (that flow's config release), so an event captured on the web and processed on the server carries both flow releases, and a warehouse row shows exactly which config version touched it. This is distinct from source.version, the source package version set only by external source emitters; the collector does not stamp it.

Upgrading v3 events or configs: see Migrating from v3 to v4. Nested entities name their entity in entity, where v3 used type: nested: [{ entity, data }].

Design Principles

Statelessness

Events are immutable snapshots. They don't reference previous events or maintain state.

Self-Describing

Events contain all context needed to understand them, without external lookups.

Vendor-Agnostic

Events use generic concepts (product, order) not vendor-specific (GA4 item, FB content).

Transformation to vendor formats happens in mapping, not in event creation.

Creating Events

elb takes a partial event: name is required, every other field is optional. The collector adds id, timestamp, timing, entity, action and source, and merges its own globals, user and consent key by key (the event's values win).

import { startFlow } from '@walkeros/collector';

const { elb } = await startFlow({
  /* destinations */
});

const result = await elb({
  name: 'product add',
  data: { id: 'P123', price: 99 },
  context: { stage: ['cart', 1] },
  globals: { currency: 'USD' },
});

// A failed destination is in result.failed (with its reason); a rejected
// event reaches no destination and sets result.error, e.g. 'Event name is invalid'
if (!result.ok) console.error(result.failed ?? result.error);

Every event uses this partial event form. walker commands keep their string name: elb('walker consent', { marketing: true }).

Related Skills

Source Files:

Documentation:

Understanding walkerOS Flow

Overview

walkerOS follows a Source → Collector → Destination(s) architecture for composable, modular event processing.

Core principle: Separation of concerns. Each component has one job. Components are composable and replaceable.

The Flow Pattern

[Source.before] → Sources → [Source.next] → Collector → [Collector.next] → [Dest.before] → Destinations → [Dest.next]
(Preprocessing)   (Capture)  (Source chain)  (Processing) (Collector chain) (Dest. chain)   (Delivery)     (Post-push)

Consent-exempt:                                                                                   Post-consent:
- Decode           - Validation     - Event creation  - Validation     - Redaction      - Push     - Audit logging
- Validate format  - Enrichment     - Enrichment      - Bot scoring    - Per-vendor     - Send     - Response parsing
- Authenticate     - Redaction      - Consent check   - Drop for all   - Drop for one   - Store    - Webhooks

collector.next runs once per event before the destination fan-out, so its work (and a stop) applies to every destination. destination.before runs per destination.

Key Concepts

Composability

A Flow combines components. You can:

  • Use multiple sources feeding one collector
  • Route events to multiple destinations
  • Swap components without changing others

The Flow Type

See packages/core/src/types/flow.ts for the canonical interface.

// Conceptual structure (see source for full type)
interface Flow {
  stores?: Record<string, Store>;
  sources?: Record<string, Source>;
  transformers?: Record<string, Transformer>;
  destinations?: Record<string, Destination>;
  collector?: Collector.InitConfig;
}

Universal Push Interface

All components communicate via push functions:

ComponentPush SignaturePurpose
Sourcepush(input) → eventsCapture external data
Collectorpush(event) → PushResultProcess and route
Destinationpush(event, context) → response?Transform and deliver

A destination's return value is the response a destination.next chain reads at ingest._response.

The collector's elb() takes a partial event (elb({ name: 'page view', data })) or a walker command; collector.push takes event objects only. startFlow returns the primary source's push as elb, or the collector's elb when no source is configured. The browser source's push takes the same partial event; a server source's push is its request handler.

startFlow Helper

See packages/collector/src/flow.ts for the startFlow function.

import { startFlow } from '@walkeros/collector';

const { collector, elb } = await startFlow({
  stores: {
    /* key-value storage, init first, destroy last */
  },
  sources: {
    /* ... */
  },
  transformers: {
    /* ... */
  },
  destinations: {
    /* ... */
  },
});

Ingest: Mutable Pipeline Context

walkerOS uses a two-layer data model:

  • Event: strict schema, structured analytics data (name, data, context, etc.)
  • Ingest: free-form mutable context that flows alongside events

Any step can read and write arbitrary keys on ingest. The runtime manages _meta:

  • _meta.hops: increments per step (safety valve at 256)
  • _meta.path: ordered list of step IDs visited (path[0] = source ID)
// In a transformer
push: async (event, context) => {
  context.ingest.botScore = 0.95; // Write freely
  context.ingest.geo = { country: 'DE' };
  console.log(context.ingest._meta.path); // ['express', 'validate', 'enrich']
  return { event };
};

Ingest is cloned per destination to prevent cross-contamination in parallel processing. After a destination push, the response is available at ingest._response.

Stores

Three pipeline components (Source / Transformer / Destination) plus Stores as passive infrastructure. Stores are not a fourth pipeline stage: they are key-value storage that other components consume via env. They have no push, no next, no before; they sit alongside the pipeline rather than inside it.

  • Referenced via $store.storeId in env values (bundled mode) or by passing the same store definition object used in stores (integrated mode)
  • Init first, destroy last: stores are available before any source, transformer, or destination starts, and outlive them on shutdown
  • No chains: stores don't participate in the event pipeline. Components access them through their env.
  • Implementations: @walkeros/server-store-fs (async, filesystem), @walkeros/server-store-s3 (async, S3-compatible), @walkeros/server-store-gcs, @walkeros/server-store-sheets, @walkeros/server-store-api. The collector ships a built-in in-memory cache tier; enable it on any store via Flow.Store.cache instead of declaring a separate memory store.
{
  "stores": {
    "assets": {
      "package": "@walkeros/server-store-fs",
      "config": { "file": true, "settings": { "basePath": "./public" } }
    }
  },
  "transformers": {
    "file": {
      "package": "@walkeros/server-transformer-file",
      "config": { "settings": { "prefix": "/static" } },
      "env": { "store": "$store.assets" }
    }
  }
}

See walkeros-understanding-stores for the full store interface and lifecycle.

Separation of Concerns

ConcernHandled ByNOT Handled By
Event captureSourcesCollector, Destinations
Event structureEvent modelComponents
Consent checkingCollectorSources, Destinations
TransformationMapping systemRaw push calls
DeliveryDestinationsSources, Collector

Step-Level Primitives

Next to its package wiring, every step (source, transformer, destination) carries cache and state, and a transformer also mapping. A source's or destination's mapping and consent sit in its config.

Event shapes are not a step-level primitive. They live in the top-level contract block (a sibling of flows) as named JSON Schemas, and enforcement is an explicit @walkeros/transformer-validate step that references a contract via $contract.<name>. See Website: Contract for the contract shape and Website: Validate for runtime enforcement.

How a step references its implementation

Every step picks exactly one of three forms to point at its implementation:

  • package: "<npm-package>" alone loads the package's default export. The common case for sources and destinations.
  • package: "<npm-package>" plus import: "<exportName>" loads a specific named export from that package. Use when a package ships multiple named exports or has no default export.
  • code: { push, type?, init? } is inline implementation, no package wiring. Useful for one-off custom logic in TypeScript flows.

There is no string form of code. A bare step with no package, import, or code is a valid no-op for all four step kinds (handy as a chain-only or mapping-only transformer).

Transformer Chains

Transformers run at three points around the collector, configured via source.next, collector.next, and destination.before (plus source.before, transformer.before, and destination.next):

Source Chain (source.next)

Runs after source captures event, before collector processing:

Bundled mode (flow.json):

{
  "sources": {
    "browser": {
      "package": "@walkeros/web-source-browser",
      "next": "enrich"
    }
  },
  "transformers": {
    "enrich": {
      "code": {
        "push": "$code:(event) => ({ event: { ...event, data: { ...event.data, enrichedAt: Date.now() } } })"
      },
      "next": "redact"
    },
    "redact": {
      "mapping": { "policy": { "user.email": { "value": "[redacted]" } } }
    }
  }
}

enrich is inline code and redact a mapping-only step; any transformer package fits the same slots.

Integrated mode (TypeScript):

sources: {
  browser: {
    code: sourceBrowser,
    next: 'enrich'
  }
},
transformers: {
  enrich: {
    code: transformerEnrich,
    next: 'redact'
  },
  redact: {
    code: transformerRedact
  }
}

In both modes next sits on the step entry (config.next works too).

Transformer before chain

Each transformer can have its own before chain that runs before its push function:

{
  "transformers": {
    "fingerprint": {
      "before": "bot",
      "package": "@walkeros/server-transformer-fingerprint",
      "config": { "settings": { "salt": "$secret.FINGERPRINT_SALT" } }
    },
    "bot": {
      "package": "@walkeros/server-transformer-bot"
    }
  }
}

Collector Chain (collector.next)

Runs once per event after the collector completed it, before the destination fan-out. Every destination receives its output; a stop here reaches no destination:

{
  "collector": {
    "next": [
      {
        "match": {
          "key": "event.name",
          "operator": "eq",
          "value": "product impression"
        },
        "stop": true
      },
      "bot",
      "validate"
    ]
  }
}

In TypeScript it is the top-level next of startFlow({ next, ... }).

Destination Chain (destination.before)

Runs per destination, after the collector chain, before that destination receives the event. A stop here skips only this destination:

Bundled mode (flow.json):

{
  "destinations": {
    "gtag": {
      "package": "@walkeros/web-destination-gtag",
      "config": { "settings": { "ga4": { "measurementId": "G-DEMO123456" } } },
      "before": "redact"
    }
  },
  "transformers": {
    "redact": {
      "mapping": { "policy": { "user.email": { "value": "[redacted]" } } }
    }
  }
}

Integrated mode (TypeScript):

destinations: {
  gtag: {
    code: destinationGtag,
    before: 'redact'
  }
},
transformers: {
  redact: {
    code: transformerRedact
  }
}

Post-push chain (destination.next)

Runs after destination push completes. The push response is available at context.ingest._response:

Bundled mode (flow.json):

{
  "destinations": {
    "api": {
      "package": "@walkeros/server-destination-api",
      "config": { "settings": { "url": "https://collect.example.com/events" } },
      "next": "auditLog"
    }
  },
  "transformers": {
    "auditLog": {
      "code": {
        "push": "$code:(event, context) => { context.logger.info('delivered', { id: event.id, response: context.ingest._response }); }"
      }
    }
  }
}

Integrated mode (TypeScript):

destinations: {
  api: {
    code: destinationApi,
    next: 'auditLog'
  }
},
transformers: {
  auditLog: {
    code: {
      push: (event, context) => {
        context.logger.info('delivered', {
          id: event.id,
          response: context.ingest._response,
        });
      },
    },
  },
}

auditLog is inline code: there is no audit transformer package.

Chain Resolution

  • source.before → consent-exempt preprocessing chain
  • source.next → source chain, before the collector
  • transformer.before → pre-transform enrichment chain
  • transformer.next (flow.json) or transformer.config.next (runtime) → links transformers
  • collector.next → collector chain, once per event, for all destinations
  • destination.before → destination chain, for that destination only
  • destination.next → post-push processing chain

Cross-Flow References ($flow)

When a single flow.json defines multiple flows, any flow can pull values from another flow's config block via $flow.<name>(.<path>)?. The most common case is wiring a web flow's API destination to a server flow's deployed URL, so the two stay in sync without duplication.

{
  "version": 4,
  "flows": {
    "server": {
      "config": {
        "platform": "server",
        "url": "https://collect.example.com"
      },
      "sources": {
        "http": { "package": "@walkeros/server-source-express" }
      }
    },
    "web": {
      "config": { "platform": "web" },
      "destinations": {
        "api": {
          "package": "@walkeros/web-destination-api",
          "config": { "settings": { "url": "$flow.server.url" } }
        }
      }
    }
  }
}

An unknown flow, a missing key or an empty value is an error in walkeros validate (UNRESOLVED_REFERENCE) and in bundle, so a build never ships with an empty cross-flow value.

Step Examples

Each step in a flow (source, transformer, destination) can ship step examples: structured { in, out } pairs that define expected input/output behavior.

The Three Type Zones

Steps sit at boundaries between arbitrary formats and walkerOS events:

  • Source: arbitrary in (HTTP request, DOM event) → walkerOS event out
  • Transformer: walkerOS event in → walkerOS event out (or false)
  • Destination: walkerOS event in → arbitrary out (vendor API call)

See using-step-examples for the full ASCII diagram and detailed explanation.

Example: Step with Examples in Flow Config

{
  "destinations": {
    "gtag": {
      "package": "@walkeros/web-destination-gtag",
      "config": { "settings": { "ga4": { "measurementId": "G-DEMO123456" } } },
      "examples": {
        "purchase": {
          "in": {
            "name": "order complete",
            "data": { "id": "ORD-123", "value": 149.97 }
          },
          "mapping": {
            "name": "purchase",
            "data": {
              "map": { "transaction_id": "data.id", "value": "data.value" }
            }
          },
          "out": [
            [
              "window.gtag",
              "event",
              "purchase",
              {
                "transaction_id": "ORD-123",
                "value": 149.97,
                "send_to": "G-DEMO123456"
              }
            ]
          ]
        }
      }
    }
  }
}

Step examples enable it.each testing and cross-step checks in walkeros validate. See using-step-examples for the complete lifecycle.

Flow Graph Connection Rules

This section defines which components can connect to which, and how chains are resolved at runtime. Use it as the canonical reference for building flow graphs, validating configurations, and rendering UI visualizations.

Every chain and route shape below has a tested instance in packages/cli/examples/flow-complete.json. Look them up by feature id in the manifest (flowCompleteFeatures from @walkeros/cli/examples, each with a JSON Pointer and a note), taught in the guide chapter chains-routing (packages/cli/examples/flow-complete.md):

ShapeFeature ids
Source chainssource-before, source-next
Collector chaincollector-next, web-collector-next
Destination chainsdestination-before, pubsub-before
one, conditional next, sequenceroute-one, route-next-match, route-sequence
stop with and without matchroute-stop, event-filter, stop-unconditional
Code-less path stepcode-free-hop
and, or, notmatch-and, match-or, match-not

many, transformer.before and destination.next have no genuine use in that file; the route fixture routeCases in @walkeros/core/dev covers them.

Valid connection matrix

FromToVia FieldValid?
SourceTransformersource.beforeYes (consent-exempt)
SourceTransformersource.nextYes (source chain)
SourceCollector(implicit, no next)Yes
SourceSource-No
SourceDestination-No
TransformerTransformertransformer.beforeYes (pre-transform)
TransformerTransformertransformer.nextYes (chain continues)
TransformerCollector(implicit, source chain ends)Yes
TransformerDestination(implicit, chain ends)Yes
CollectorDestination(implicit, no next/before)Yes
CollectorTransformercollector.nextYes (all destinations)
CollectorTransformerdestination.beforeYes (one destination)
DestinationTransformerdestination.nextYes (post-push)
CollectorSource-No

Source chains (source.next)

  • Entry: source.next: "transformerId" or source.next: ["t1", "t2"]
  • Exit: chain ends, event reaches collector
  • Multiple sources can reference the same transformer (fan-in)
  • No next = source connects directly to collector
  • Resolved per event after the source's state, so it can route on a loaded value

Collector chain (collector.next)

  • Entry: collector.next (flow.json) or top-level next in startFlow
  • Runs once per completed event, before the destination fan-out
  • Exit: every finished copy is delivered to the destinations; a stop or a transformer returning false drops the event for all destinations
  • Consent replay, late destinations, and pre-run replay receive its output; it never runs twice for one event
  • No next = collector connects directly to the destinations

Destination chains (destination.before)

  • Entry: destination.before: "transformerId" or destination.before: ["t1", "t2"]
  • Exit: chain ends, event reaches destination; a stop skips only this destination
  • Multiple destinations can share the same transformer
  • No before = the collector (or its chain) connects directly to destination
  • Per-destination filtering belongs here, not in collector.next

Chain resolution (one model for every chain field)

One runner executes every position (runTransformerChain in packages/collector/src/transformer.ts), on top of the pure continuation stack startChain / advanceChain in packages/core/src/chain.ts. Routes are resolved per hop by getNextSteps(spec, root) in packages/core/src/route.ts, which returns NextSteps (ids plus an optional continuation, a stop, or forks) and needs the root { ingest, event }.

  • String start: runs the transformer, then its own next, recursively
  • Array start: the array is the backbone. A member's own next (static or conditional) is inserted right after that member, then the array continues. ["bot", "validate", "session"] with bot.next = "foo" runs bot, foo, validate, session. Insertion is depth-first.
  • Lazy per hop: a route is evaluated only when the event reaches it, against { ingest, event } as the previous step left them. A sequence ["a", { "match": ..., "stop": true }, "b"] evaluates the stop after a ran.
  • Repeated steps: a step listed twice runs twice (no dedup, no warning)
  • Cycles: a member next that leads back to a step already on its insertion path is skipped; each copy is capped at 256 steps
  • Unknown transformer id: logged as an ERROR line and skipped, the chain continues; walkeros validate reports UNKNOWN_ROUTE_TARGET as an error
  • Result { next }: replaces the member's own next for that event, then the array continues

For tooling without an event, getRouteGraph(spec, transformers?) enumerates every branch of the same compiled form (both sides of each gate, every one and many entry, stops marked), so canvases and validators draw exactly what can run.

Conditional routing (one operator)

The next and before properties accept a Route (string | Route[] | RouteConfig). A RouteConfig is a disjoint union: each config sets at most one of next (gated link), one (first-match dispatch), many (all-match fan-out), or stop (drop), never more than one. Every match reads { ingest, event }. The one operator picks the first entry whose match succeeds:

"next": {
  "one": [
    { "match": { "key": "ingest.path", "operator": "prefix", "value": "/api" }, "next": "apiHandler" },
    { "next": "default" }
  ]
}
  • one entries are evaluated in order, first match wins
  • Besides _meta, ingest holds only what the source's config.ingest maps and what earlier steps write into it: without { "map": { "path": "path" } } in that config, ingest.path is empty and the /api entry never matches
  • An entry without match always matches, use it as the fallback
  • No matching entry means the event passes through unchanged
  • Works on all chain positions: source.before, source.next, transformer.before, transformer.next, collector.next, destination.before, and destination.next
  • An array made only of route configs (no id) is an implicit one: first match wins. walkeros validate hints at it; write { "one": [...] } explicitly
  • Routes are compiled once (cached) and resolved per event by getNextSteps in packages/core/src/route.ts

All-match dispatch (many operator)

Use many when every matching entry should produce an independent copy of the event (audit-while-process, multi-decoder fan-out). Allowed in every chain field, at any depth. Each copy finishes the entire rest of the path on its own (rest of the array, every enclosing chain, collector, destinations); copies are never merged. Each copy gets its own event.id, derived deterministically from the parent id and branch position (deriveSpanId); trace stays shared. One matching entry continues in place without a copy. A transformer returning Result[] forks the same way.

"next": {
  "many": [
    { "match": { "key": "event.consent.analytics", "operator": "eq", "value": "true" }, "next": "ga4Pipeline" },
    { "next": "auditLog" }
  ]
}

A consent value is a boolean, and eq compares it as text, so a granted consent matches "true".

Dropping an event (stop)

{ "stop": true } ends the running copy (optionally gated by match; a failing match falls through). In a many, a stop ends only its own copy. Meaning per position:

PositionA resolved stop means
source.before, source.nextnever reaches the collector; push result { ok: true, dropped: true }
transformer.before, transformer.next, result { next }the copy ends; the enclosing position applies its meaning
collector.nextno destination receives the event (runs before the fan-out)
destination.beforethis destination skips it; others unaffected
destination.nextpost-push chain ends; delivery already happened

"Drop impressions everywhere" is a stop in collector.next; "drop impressions for vendor destinations only" is a stop in those destinations' before. A drop is observable as a skip with skipReason: 'dropped'. walkeros validate warns about entries after an unconditional stop (dead code).

Paths and pass-through steps (code-less transformer entries)

A path is the multi-step chain through a flow's transformers section. A pass-through step (short: pass) is a single step inside a path that declares no code and no package; the runtime synthesizes its push from the operative fields the step does declare.

Pass-through steps come in three variants:

  • Chain-only: only before and/or next set. A named hop that shares a chain across multiple call sites (avoids duplicating arrays).
  • Cache-only: only cache set. A dedup or short-circuit step. cache.stop: true in a source chain or collector.next halts the event for all destinations.
  • Mapping-only: only mapping: Mapping.Config set. A declarative event-to-event transform that mutates the event in-flight.
{
  "transformers": {
    "validateThenEnrich": {
      "before": ["validate", "enrich"]
    }
  },
  "destinations": {
    "gtag": {
      "package": "@walkeros/web-destination-gtag",
      "config": { "settings": { "ga4": { "measurementId": "G-DEMO123456" } } },
      "before": "validateThenEnrich"
    },
    "meta": {
      "package": "@walkeros/web-destination-meta",
      "config": { "settings": { "pixelId": "1234567890" } },
      "before": "validateThenEnrich"
    }
  }
}

Transformer step entries follow a closed schema: unknown top-level keys are validation errors (UNKNOWN_KEY). An entry with no field at all is a valid no-op.

See walkeros-understanding-transformers for full depth on the three variants, the closed-schema rule, and the dual semantic of mapping at the transformer position versus the destination position.

Transformer sharing

A single transformer can appear in source chains (source.next), the collector chain (collector.next), and destination chains (destination.before). The same transformer pool is shared; role depends on which chain references it. Work every destination needs belongs in collector.next, not repeated in each destination.before.

Deferred activation (require)

  • source.config.require: ["consent"]: the source's on() deliveries wait until consent state is recorded; its init runs at once
  • destination.config.require: ["user"]: destination deferred until "user" event fires
  • Multiple requirements: all must be fulfilled (each fires independently)

Mapping and consent gating

  • Source-level: source.config.mapping and source.config.consent: applied before the source chain; blocks event entirely
  • Destination-level: destination.config.consent is checked first, before the destination chain (destination.before): it skips only that destination and queues denied events. The queue sits after collector.next, so a replay of a queued event never re-runs collector.next; it does run destination.before. destination.config.mapping is applied after the destination chain, so a rename in the chain changes which rule matches

Canvas rendering rules (for UI graph visualization)

  • Shared transformers (pre+post): duplicate visually with a link indicator; editing one updates the other
  • Router fan-out: keep graph planar; trace edges individually
  • Orphan transformers (not in any chain): render grey/muted; gain color when connected
  • Diamond patterns (fan-in + fan-out): expected and valid
  • Overlapping destination.before chains: intentional (e.g., shared validator for monitoring)
  • Collector chain: draw collector.next between the collector and the destination fan-out, distinct from per-destination before chains; draw a stop as an end of its branch

Related Skills

Package READMEs:

Source Files:

Documentation:

Understanding walkerOS Mapping

Overview

Mapping transforms data at multiple points in the walkerOS flow:

  1. Source → Collector: Transform raw input (HTTP requests, dataLayer pushes) into walkerOS events
  2. Collector → Destination: Transform walkerOS events into vendor-specific formats

Core principle: Mapping is the universal transformation layer. Same strategies work everywhere in the flow.

Core Functions

See packages/core/src/mapping.ts for implementation.

FunctionPurpose
getMappingEvent(event, rules)Find mapping rule for an event
getMappingValue(value, data, { collector })Transform a value using mapping config
processEventMapping(event, config, collector)Unified processing for sources/destinations

processEventMapping Flow

1. Apply config.policy (modifies event)
2. Find matching rule via getMappingEvent()
3. Apply rule.policy (modifies event)
4. Transform config.data (global)
5. Check rule.ignore (short-circuits if true)
5b. Read rule.silent (informational - destination honors it)
6. Override event.name if rule.name
7. Transform rule.data (event-specific)
8. Add the include sections (rule.include ?? config.include; data wins)
9. Strip rule.remove paths from the produced data

Configuration Hierarchy

Mapping.Config (Top Level)

interface Config {
  consent?: Consent; // Required consent for ALL events
  data?: Value; // Global data transformation
  include?: string[]; // Event sections to flatten into context.data
  policy?: Policy; // Pre-processing for ALL events
  mapping?: Rules; // Event-specific rules
}
Mapping.Config.include

include lists event sections whose fields are added to the data the destination receives (PushContext.data), after the rule's own data ran. Each field becomes a flat key named <section>_<key>: nested objects are joined with _, arrays stay whole, a context entry contributes its label. Sections: data, globals, context, user, source, event (entity, action, id, timestamp, name, trigger, timing), or all for every one of them. On a key conflict the produced data (the config-level data merged with the rule's) wins; remove runs after it. The runtime helper is flattenIncludeSections (in packages/core/src/include.ts).

Use this to forward whole sections without mapping each field. include is also available on Mapping.Rule (rule-level replaces config-level). user includes every user field, email among them, unhashed.

{
  include: ['globals'],
  mapping: {
    page: {
      view: {
        // data: { page_title, globals_language, globals_currency, ... }
        data: { map: { page_title: 'data.title' } },
      },
    },
  },
}

Mapping.Rule (Per Event)

interface Rule {
  name?: string; // Override event name
  data?: Value; // Event-specific data transformation
  include?: string[]; // Event sections to flatten into context.data (rule-level)
  ignore?: boolean; // Skip this event entirely (no processing, no push)
  silent?: boolean; // Process settings side effects, skip destination default push
  policy?: Policy; // Event-specific pre-processing
  condition?: Function; // Match condition (for arrays)
  consent?: Consent; // Required in addition to config.consent for events this rule matches
  settings?: unknown; // Custom event configuration
  batch?: number | BatchOptions; // Own batch buffer (a number is the wait in ms)
  extend?: RulePatch; // Config-layer merge onto a package-shipped default rule
  remove?: string[]; // Output-layer: dotted paths stripped from the final payload
}
extend and remove (patching package-shipped rules)

@walkeros/transformer-ga4 ships its own default mapping rules. A user rule at the same key normally replaces the default in full. Two keywords change that:

  • extend (config layer): a partial rule deep-merged onto the package-shipped default at init, before any event is evaluated. A null value clears an inherited field. Lets you add or override one field while keeping the rest. It has an effect only in @walkeros/transformer-ga4's settings.mapping; in any other rule walkeros validate and the bundle reject it (MAPPING_NO_OP).
  • remove (output layer): dotted paths stripped from the produced payload after evaluation, applied last. Useful for dropping PII or vendor-reserved fields without rewriting the full rule. Works in a destination's rules and in @walkeros/transformer-ga4's settings.mapping.

A rule with neither keyword keeps the standard replace behavior.

See the website mapping docs for the authoritative reference and the GA4 example.

{
  "settings": {
    "mapping": {
      "purchase": {
        "extend": {
          "data": { "map": { "affiliation": "params.ep.affiliation" } }
        },
        "remove": ["currency"]
      }
    }
  }
}

Silent vs Ignore

Both flags control rule behavior but have different semantics:

  • ignore: true: the rule matched but nothing happens. No data transform, no destination call, no side effects. Use for suppression.
  • silent: true: the rule matched and the destination push() is called. settings.identify, settings.revenue, settings.group, etc. still run. Only the destination's default forwarding call (e.g. track(), capture(), event()) is suppressed. Use for "identify without an event" style flows.

If both flags are set on the same rule, ignore wins.

Common use case: a user login event that should call amplitude.identify() but should not create a separate track("user login") event in Amplitude.

Mapping.ValueConfig (Value Extraction)

interface ValueConfig {
  key?: string; // Extract from path
  value?: Primitive; // Static fallback value
  fn?: Function; // Compute the value from the source
  map?: Record; // Object transformation
  loop?: [path, config]; // Array transformation
  set?: Value[]; // Create array from values
  condition?: Function; // Conditional extraction
  consent?: Consent; // Consent-gated extraction
  validate?: Function; // Value validation
}

Exactly one producer runs per value, in this order: loop, map, set, key, fn. The rest never run, so a fn next to a key is ignored. value is the fallback when the producer yields nothing, including a loop over something that is not a list. To transform a field, use fn alone (it receives the source), or rewrite it with policy and read it with key.


Event Matching

Match events to transformation rules by entity and action.

const mapping = {
  // Exact match: "product view" → view_item
  product: {
    view: { name: 'view_item' },
    add: { name: 'add_to_cart' },
  },

  // Wildcard action: "foo *" → foo_interaction
  foo: {
    '*': { name: 'foo_interaction' },
  },

  // Wildcard entity: "* click" → generic_click
  '*': {
    click: { name: 'generic_click' },
  },
};

Conditional Mapping (Array)

In an array of rules, the first matching condition wins:

order: {
  complete: [
    {
      condition: (event) => event.data?.value > 100,
      name: 'high_value_purchase',
    },
    { name: 'purchase' }, // Fallback (no condition)
  ],
}

JSON with $code:

{
  "order": {
    "complete": [
      {
        "condition": "$code:(event) => event.data?.value > 100",
        "name": "high_value_purchase"
      },
      { "name": "purchase" }
    ]
  }
}

Value Mapping Strategies

Common patterns shown below. For detailed examples of all 12 strategies, see value-strategies.md.

Essential Patterns

// Key extraction (string shorthand)
'data.price'                              // → event.data.price

// Key with fallback
{ key: 'data.currency', value: 'USD' }    // Use USD if missing

// Static value
{ value: 'USD' }

// Function transform
{ fn: (event) => event.data.price * 100 } // Convert to cents

// Object map
{ map: { item_id: 'data.id', item_name: 'data.name' } }

// Array loop
{ loop: ['nested', { map: { item_id: 'data.id' } }] }

// Loop with "this" (single item as array)
{ loop: ['this', { map: { item_id: 'data.id' } }] }

// Set (create array)
{ set: ['data.id'] }                      // → ["SKU-123"]

// Fallback chain: Value[] at any value position (first defined value wins)
[{ key: 'data.sku' }, { key: 'data.id' }, { value: 'unknown' }]

// Consent-gated (the event's consent decides, in a loop too: a `consent`
// key on a loop item is plain data)
{ key: 'user.email', consent: { marketing: true } }

// Validate
{ key: 'data.email', validate: (v) => v.includes('@') }

Policy (Pre-Processing)

Policy shapes the event before its data is mapped. At a destination the rule was already selected when the event entered it; at a source or transformer, policy runs before matching. Use for:

  • Adding computed fields
  • Normalizing data structure
  • Consent-gated field injection

Config-Level Policy

Applied to all events:

config: {
  policy: {
    'user_data.external_id': 'user.id',
    'custom_data.server_processed': { value: true },
  },
  mapping: { /* ... */ }
}

Event-Level Policy

Applied after config policy, only for specific event:

mapping: {
  order: {
    complete: {
      policy: {
        'enriched.total_cents': {
          fn: (event) => Math.round(event.data.value * 100)
        }
      },
      name: 'purchase',
      data: { /* ... */ }
    }
  }
}

Policy with Consent

A policy entry that resolves to nothing removes the field, so a denied consent gate redacts it:

{
  "policy": {
    "user_data.em": {
      "key": "user.email",
      "consent": { "marketing": true }
    }
  }
}

Rule Features

Ignore Events

mapping: {
  test: { '*': { ignore: true } },  // Ignore all test events
}

Batch Processing

To batch all of a destination's events, set config.batch (no mapping rule needed):

config: {
  batch: { size: 5 },  // Flush after every 5 events (a bare number is the wait window in ms)
}

A rule-level batch batches only that entity-action into its own buffer and overrides config.batch per field:

mapping: {
  order: {
    complete: {
      batch: { wait: 1000 },  // Only order complete batches, with a 1s debounce
    }
  }
}

Custom Settings

mapping: {
  order: {
    complete: {
      name: 'purchase',
      settings: { priority: 'high', retryCount: 3 }
    }
  }
}

$code: Prefix (JSON Configs)

The $code: prefix enables JavaScript functions in JSON configurations:

{
  "fn": "$code:(event) => event.data.price * 100",
  "condition": "$code:(event) => event.data?.value > 100",
  "validate": "$code:(value) => value > 0"
}

Important: The $code: prefix is processed by the CLI bundler. It converts JSON strings to actual JavaScript functions during build.

Function Signatures

All mapping callbacks share (value, context). The second argument is a Mapping.Context object (see "Context object" below).

ContextSignature
fn(value, context) => result
condition (value)(value, context) => boolean
condition (rule)(event, context) => boolean
validate(value, context) => boolean
loop condition(value, context) => boolean

One-arg signatures like (value) => value.toUpperCase() continue to work, TypeScript ignores the unused second arg.

Context object

FieldTypeRequiredDescription
eventWalkerOS.DeepPartialEventyesThe root event being mapped
mappingMapping.Value | Mapping.RuleyesThe surrounding mapping config (or rule)
collectorCollector.InstanceyesActive collector, use for push and queue
loggerLogger.InstanceyesUse for info/warn/error/debug
consentWalkerOS.Consent (optional)noResolved consent at this evaluation point

Quick Reference

Value Extraction Cheatsheet

PatternResult
"data.id"Extract event.data.id
{ value: "USD" }Static "USD"
{ key: "x", value: "y" }Extract x, fallback to "y"
{ fn: (e) => ... }Custom function
{ map: {...} }Object transformation
{ loop: ["nested", {...}] }Array transformation
{ loop: ["this", {...}] }Single-item as array
{ set: ["a", "b"] }Create array [valA, valB]
[m1, m2, m3]Fallback chain
{ consent: {...} }Consent-gated
{ condition: fn }Conditional
{ validate: fn }Validated

Rule Features Cheatsheet

FeaturePurpose
nameOverride event name
dataTransform event data
ignoreSkip event entirely (no processing, no push)
silentRun settings side effects, skip default forwarding
policyPre-process event
conditionMatch condition (arrays)
consentRequired in addition to config.consent for the events it matches
settingsCustom configuration
batchOwn batch buffer for this event (destination needs pushBatch)
extendConfig-layer merge onto a package-shipped default (null clears a field)
removeOutput-layer: dotted paths stripped from the final payload

Config Features Cheatsheet

FeaturePurpose
consentRequired consent (all events)
dataGlobal data transformation
includeEvent sections added to data
policyGlobal pre-processing
mappingEvent-specific rules

Complete Examples

For full destination configuration examples (TypeScript + JSON), see complete-examples.md.


Mapping at the Transformer Position

Mapping.Config accepts the same shape at three positions in the flow, but the semantic differs by position:

PositionWhat the mapping produces
SourceA walkerOS event from raw input
TransformerA mutated walkerOS event that continues through the chain
DestinationA vendor-shaped payload that the destination consumes

When a transformer step declares only a mapping (no code, no package), the collector synthesizes a push that runs processEventMapping against each event. A config.mapping wins over the step-level mapping. Next to code or package, a mapping never runs. walkeros validate and the bundle reject both cases (MAPPING_NO_OP), and the collector logs one ERROR line for them when the transformer registers. It is the same keyword as the destination field, with a different semantic at this step position. See walkeros-understanding-transformers for the pass-through-step model.

Which fields apply at the transformer position

Only event-mutating fields run; every other field is a no-op that walkeros validate and the bundle reject (MAPPING_NO_OP) and the collector names in one ERROR line at registration (rules in array form included):

FieldTransformer position
policyApplies, pre-processes the event before rule matching
mapping[].conditionApplies, picks the rule (awaited in order, first match wins)
mapping[].policyApplies, per-event policy
mapping[].nameApplies, renames the event (mutation is observable downstream); in a destination's own before chain it does not change that destination's rule (ERROR rule-not-applied when it lands on another rule)
mapping[].ignoreApplies, drops the event from the chain entirely (no downstream step sees it)
consent, include, dataNo-op (no vendor payload, no destination to gate)
mapping[].consent, include, remove, batch, settings, extend, data, silentNo-op (destination concepts; a rule's consent also applies at a source)

Note the ignore: true semantic shift: at a destination it skips delivery to that destination only; at a transformer step it drops the event from the chain so no downstream step (transformer or destination) sees it.

{
  "transformers": {
    "redactPII": {
      "mapping": {
        "policy": {
          "user.email": { "value": "[redacted]" }
        },
        "mapping": {
          "test": {
            "*": { "ignore": true }
          },
          "order": {
            "complete": { "name": "purchase" }
          }
        }
      }
    }
  }
}

Where Mapping Lives

LocationPurpose
Source configTransform raw input → walkerOS events
Transformer step configMutate walkerOS events in-flight
Destination configTransform walkerOS events → vendor format
state.mappingShape a store read or write (paths relative to the value read)
packages/core/src/mapping.tsCore mapping functions
packages/core/src/types/mapping.tsType definitions
packages/cli/examples/flow-complete.jsonTested example, features by id

Related Skills

Source Files:

Detailed References:

Examples:

Understanding walkerOS Sources

Overview

Sources capture events from the external world (browser DOM, dataLayer, HTTP requests, cloud functions) and feed them to the collector.

Core principle: Sources capture. They don't process or deliver: that's collector and destinations.

Source Interface

See packages/core/src/types/source.ts for canonical interface.

Init Function (Context Pattern)

Sources use a context-based initialization pattern:

import type { Source } from '@walkeros/core';

export const sourceMySource: Source.Init<Types> = async (context) => {
  const { config = {}, env, logger, id } = context;
  // ...
};

Context contains:

PropertyTypePurpose
configSource.Config<T>Settings, mapping, options
envTypes['env']Environment (push, logger)
loggerLoggerLogging functions
idstringSource identifier
collectorCollector.InstanceReference to collector
withScope(scope, respond, body) => Promise<R>Bind ingest + respond to a single scope (server sources). Takes a normalized Source.Scope

Push Method

MethodPurpose
push(input)Receive external input, emit events

Init Method

init?: () => void | Promise<void>: optional eager-startup hook on the returned Source.Instance. The factory must be side-effect-free: build the instance and return it. The collector calls init() on every source eagerly after all factories register, regardless of config.require. Use init for the side effects: adopting a pre-init window queue (e.g., window.elbLayer), attaching DOM listeners, opening sockets, intercepting window.dataLayer. After init runs the collector flips Source.Config.init to true.

queueOn Buffer

queueOn?: Array<{ type: On.Types; data: unknown }>: optional buffer on the Source.Instance for lifecycle events delivered before the source is started (started ≡ config.init === true && !config.require?.length). The collector pushes { type, data } here when it would otherwise call source.on(type, data). Once the source becomes started, the collector replays each entry via source.on(...) and clears the queue.

Destroy Method

destroy?: DestroyFn: optional cleanup method. Called during command('shutdown'). Use to close HTTP servers, timers, or connections. Receives { id, config, env, logger }.

Source Lifecycle

The collector and every source agree on three lifecycle markers, all on Source.Config:

FieldPurpose
init?: booleanSet by the collector to true after Instance.init() resolves. Authors do not write to it. Reflects "init has run", not "is started".
require?: string[]Author-supplied timing hint. Lists collector state a source needs before its first on() delivery. Satisfied by the collector's current recorded state (level), not only by a future event, so order does not matter: the gate clears whether the required state was recorded before or after this source registered. It is not a correctness dependency: a source reacts to state correctly whether or not it declares require.
disabled?: booleanAccepted by walkeros validate, but the collector does not read it for sources: a source with disabled: true still registers, runs init and captures events. Remove the source from the flow to switch it off.

The flow is:

  1. Register: collector invokes the factory. Factory returns a fresh Source.Instance with no side effects.
  2. Init pass: collector calls Instance.init() on every registered source, then sets config.init = true.
  3. Lifecycle delivery: for each collector event (consent, user, session, run, …) the collector decrements every source's require in place AND reconciles every still-pending source/destination against current state, so a gate also clears from state that was already recorded. If a source is started, it calls source.on(type, data) directly. Otherwise it pushes { type, data } into Instance.queueOn.
  4. Replay: when a source becomes started (require empties), the collector replays its queueOn via source.on(...) and clears the queue.

require therefore gates the timing of on() delivery, not code execution and not correctness. Instance.init() always runs eagerly. There is no collector.pending.sources map: per-source state lives entirely on Source.Instance and Source.Config.

Exactly-once state delivery is a collector guarantee

State commands (consent, user, globals, custom) are recorded by the collector immediately, even before run, and delivered to each source's on handler exactly once per change. The collector tracks what each subscriber has already received, so re-running or re-registering never double-fires a state reaction. Sources should not hand-roll their own deduplication for state deliveries: the collector enforces exactly-once, and delivery is order-independent (it does not depend on source init order or on whether the state arrived before or after run).

This mirrors the destination model: Destination.Instance.init handles one-time bootstrap, Destination.Config.init is the collector-managed "init has run" flag, and Destination.Config.require gates event delivery the same way. Destinations receive the same exactly-once state delivery through their on() handler, and receive the current state before their first push. See walkeros-understanding-destinations.

Pre-run events are held and replayed by the collector

Events a source emits before the collector runs are held in a bounded FIFO buffer (collector.preRunQueue, capped by config.queueMax) and replayed at run, with the pipeline running once at replay time. A source therefore does not need bespoke on('run') deferral to avoid losing its startup events. Deferring emission to on('run') is fine and often clearer for ordering (the session source does this so session start lands in the post-run pipeline), not for loss protection.

The one case that needs an explicit run gate is a queue consumer (pubsub-pull, sqs): starting to consume means acknowledging messages to a broker, and an ack must never precede delivery. These sources start their subscription or poll loop in on('run') and never ack a message the pipeline did not accept (result.ok === false routes to the source's onPushError disposition, nack by default).

Push Signatures by Type

Source TypeSignatureExample
Cloud Functionpush(req, res) → Promise<void>HTTP handler
Browserpush(event) → Promise<PushResult>A partial event or a walker command
DataLayerpush is the collector's elbCaptures window.dataLayer.push instead

A source's push is the handler itself, with no wrapper:

// Direct deployment
http('handler', source.push);

Do not confuse Instance.push with env.push. They point opposite ways:

  • Instance.push is inward: the handler callers hand input to (an HTTP request, a window.elb(...) call, a synthetic test message). startFlow exports the primary source's Instance.push as its elb.
  • env.push is outward: where the source sends the events it produced.

A source can legitimately have both, and they need not be related: the browser source's Instance.push is a BrowserPush that accepts partial events and walker commands, while its events leave via env.push. The dataLayer and session sources keep Instance.push: env.elb, because callers do not feed them events at all: they capture from dataLayer.push and from session detection respectively.

When a test or integration code needs to invoke a source's push through the collector bag, collector.sources erases the per-source generic on read. Use Source.getSource<T>(collector, id) to recover the narrow signature without a cast. See the testing-strategy skill for the full pattern (symmetric helpers exist for destinations, transformers, stores).

Source Paths

TypePathExamples
Webpackages/web/sources/browser, dataLayer
Serverpackages/server/sources/gcp

Browser Source

The browser source captures events from DOM using data attributes.

<button data-elb="product" data-elb-product="id:P123;name:Laptop">
  <span data-elbaction="click">Add to Cart</span>
</button>

See packages/web/sources/browser/ for implementation.

Parallel sources

Several browser sources can run on one page. Trigger, element, and visibility state is per source, and the DOM is shared: any number of sources may scan the same node, each emitting into its own pipeline. Only two resources are single slots, window[elb] and the named elbLayer array, and each belongs to the first source that takes it. A source refused one of them logs an error and keeps capturing without it. Give each parallel flow its own elb and elbLayer name, or elb: false and elbLayer: false for an embedded source with no page-window footprint.

Full contract, recipes, and known limits: Multiple sources on one page.

DataLayer Source

Captures events from a GTM-style dataLayer array.

window.dataLayer.push({
  event: 'view_item',
  ecommerce: { items: [{ item_id: 'P123' }] },
});
// arrives as the event 'dataLayer view_item' (settings.prefix, default
// 'dataLayer'), the other keys as its data; a source mapping renames it

See packages/web/sources/dataLayer/ for implementation.

Server Sources

Handle HTTP requests in cloud functions. Server sources use the context pattern:

import type { Source } from '@walkeros/core';
import { createRespond, toEventList } from '@walkeros/core';

export const sourceCloudFunction: Source.Init<Types> = async (context) => {
  const { config = {} } = context;

  // Apply defaults inline: flow.json is developer-controlled, so no
  // runtime validation. Shape checks live in ./schemas and are used by
  // `walkeros validate` and dev tooling, never at runtime.
  const userSettings = config.settings || {};
  const settings = {
    ...userSettings,
    // example default: port: userSettings.port ?? 3000,
  };

  const push = async (req: Request, res: Response): Promise<void> => {
    const respond = createRespond((options) => {
      res.status(options.status ?? 200).json(options.body);
    });
    const scope = buildScope(req); // see Per-Scope Context below

    await context.withScope(scope, respond, async (env) => {
      for (const event of toEventList(scope.body)) await env.push(event);
    });
    respond({ body: { success: true } }); // a no-op if a step responded
  };

  return { type: 'cloudfunction', config: { ...config, settings }, push };
};

See packages/server/sources/gcp/ for implementation.

Forward the whole body. A server source hands the collector the full DeepPartialEvent it received, renaming only its wire-level event-name key. It must not project the body through a field allowlist, or it strips source provenance (release, trace) and other legitimate fields. express and fetch are the reference pass-through implementations.

Env Pattern (Dependency Injection)

Platform dependencies go through env with fallback to globals or direct imports. This enables testing and simulation without touching globals.

// Express source: env.express ?? express (import fallback)
const expressLib = env.express ?? express;
const app = expressLib();

// Web sources: env.window ?? window (global fallback)
const win = env.window ?? window;
const doc = env.document ?? document;

Every source's Env interface extends Source.BaseEnv with optional platform deps:

export interface Env extends Source.BaseEnv {
  window?: Window & typeof globalThis; // web sources
  document?: Document; // web sources
  express?: typeof express; // express source
  cors?: typeof cors; // express source
}

Tests inject mocks via env instead of mocking globals. See testing-strategy.

env.push for events, env.elb for commands

BaseEnv gives every source two outward exits, and they are not interchangeable:

ExitUse forWhat it is
env.pushEvents (event objects only)The collector's per-source pipeline (wrappedPush)
env.elbwalker * commandsThe raw collector adapter

Emit events through env.push. That is what makes the source's own next/before chains, mapping (including ignore and consent), cache, state, per-source Ingest, and collector.status.sources.<id> apply. A source that emits events through env.elb bypasses all of it, so those config fields become silently dead: accepted, never executed.

Commands must keep using env.elb. env.push takes WalkerOS.DeepPartialEvent only, so a command string has no meaning there.

Both exits, plus env.command, env.logger and env.sources, are runtime-owned: the collector writes these five capabilities into the env last, so an author value for any of them is ignored, uniformly. env stays the author's bag for platform and vendor dependencies only. To end the pipeline somewhere else, declare terminus on the source registration (a sibling of code); it is explicit and total, and stored flow configs cannot carry it (validation rejects unknown source keys).

// Event → pipeline
await env.push({ name: 'page view', data: { title } });

// Command → raw collector
await env.elb('walker consent', { marketing: true });

Call env.push with the event alone. The collector supplies id, ingest, mapping and preChain itself; passing your own options is for server sources threading a per-request scope via context.withScope.

Two consequences for testing a source:

  • The pipeline's end is captured when the source is constructed. Replacing collector.push after startFlow resolves is invisible to pipeline pushes, so assert on a spy destination instead of a collector.push mock. For a deterministic raw capture at the source→collector boundary, declare terminus on the registration: it receives the event exactly as the source emitted it, and the entire pipeline is skipped (no before/next chains, no cache, no state, no mapping, no minted span id, no ingest, no respond, no status.sources counting, no Observe records).
  • A next/before chain adds await hops between the emit and delivery. For a fire-and-forget emit, settle microtasks before asserting.

Transformer Wiring

Sources can wire to pre-collector transformer chains via the next property:

sources: {
  browser: {
    code: sourceBrowser,
    next: 'validate'  // First transformer to run after this source
  }
}

The transformer chain runs before events reach the collector. See understanding-transformers for chain details.

Per-Scope Context (server sources)

A single source factory instance handles many concurrent invocations: Express processes overlapping requests, Lambda reuses one handler across calls, queue consumers loop over messages. Each logical unit of work is a scope. Server sources MUST wrap each invocation with context.withScope(rawScope, respond, body):

A server source's job is to adapt its platform's request into the shared Source.Scope shape, so everything downstream is source-independent. Build the scope first, then pass it:

const push = async (req, res) => {
  const respond = createRespond((options) => {
    /* wire options into res */
  });

  const scope = buildScope(req); // normalize: method, url, path, query,
  // headers, body, ip, raw

  await context.withScope(scope, respond, async (env) => {
    for (const event of toEventList(scope.body)) {
      await env.push(event);
    }
  });
};

normalizeHeaders, normalizeQuery and normalizeBody from @walkeros/core do the per-field work; toEventList and isBatchBody decide how many events the body carries. Never pass the platform object straight to withScope: the parameter is typed Source.Scope | undefined, and a Headers instance or a WHATWG Request cannot be descended by a mapping path at all.

Inside body, env.push carries that scope's ingest (extracted from the scope via config.ingest mapping) and respond end to end through the pipeline. Concurrent scopes never share ingest or respond.

ingest is request scoped, not event scoped. With N events in one scope, a transformer writing per-event annotations into ingest records only the last.

Browser sources skip withScope. A browser tab is a single logical scope for its lifetime; calling env.push directly is correct.

Response Delegation (env.respond)

When a server source passes a respond to withScope, every transformer and destination in the pipeline can call env.respond?.({ body, status?, headers? }) to customize the HTTP response. First call wins (createRespond is idempotent), so the source's default response is a no-op if a step already responded.

See @walkeros/server-source-express for the reference implementation.

createTrigger Pattern

Every source exports a createTrigger factory from its examples (dev entry) that follows the unified Trigger.CreateFn interface:

type CreateFn<TContent, TResult> = (
  config: Collector.InitConfig,
  options?: unknown,
) => Promise<Trigger.Instance<TContent, TResult>>;

createTrigger simulates real-world invocations from the outside: full blackbox, no source instance access. Each package implements it differently:

SourceContentTrigger typeMechanism
BrowserHTML stringclick, load...DOM injection + events
ExpressHTTP req shapePOST, GETReal fetch() requests

The trigger lazily calls startFlow(config) on first invocation. Tests capture events via spy destinations. See using-step-examples for testing patterns.

Setup (optional)

Sources can implement an optional setup() lifecycle to provision external resources, for example registering a webhook with a third-party provider, creating a Pub/Sub subscription, or pre-allocating queue resources. Setup is never invoked by the runtime, push, init, or deploy. It runs only when an operator explicitly types walkeros setup source.<name>.

Today the command cannot run a source's setup. It reads the package's export as an object with a setup method, while a source exports its init function, so it stops with Package <name> has no default export (or has no export "<name>" when the step sets import) and creates nothing. Provision a source's resources with the platform's own tooling; walkeros-create-source names the packages affected.

The signature is (ctx: LifecycleContext<Config<T>, Env<T>>) => Promise<unknown>, where LifecycleContext carries { id, config, env, logger }. Idempotency is the package's responsibility: the framework adds no opinion. Use resolveSetup(ctx.config.setup, DEFAULTS) from @walkeros/core to normalize the boolean | object shape into a concrete options object.

See walkeros-create-source, walkeros-understanding-destinations, walkeros-understanding-stores, and the walkeros setup CLI documentation for the authoring template and operator workflow.

Related Skills

Source Files:

Package READMEs:

Documentation:

Understanding walkerOS Stores

Overview

Stores provide key-value storage that other components (sources, transformers, destinations) consume via environment injection. They are the 4th component type in Flow.Json alongside sources, transformers, and destinations.

Core principle: Stores are passive infrastructure. They don't process events or participate in chains; they provide state that other components read and write.

Store interface

See packages/core/src/types/store.ts for the canonical interface.

Instance

PropertyTypePurposeRequired
typestringStore type identifierRequired
configStore.ConfigSettings and envRequired
get(key) => StoreValue | undefinedRead a valueRequired
set(key, value, ttl?) => voidWrite a value (see below)Required
delete(key) => voidRemove a valueRequired
destroyDestroyFnCleanup on shutdownOptional

All methods can be sync or async (return Promise).

Value type and file mode

Stores hold one canonical value type: structured data (StoreValue), with binary (Uint8Array) as a first-class leaf. StoreValue is string | number | boolean | null | Uint8Array | StoreValue[] | { [key: string]: StoreValue } (undefined is reserved as the "miss" sentinel and is never a stored value). A shared codec in @walkeros/core/node (serializeStoreValue / deserializeStoreValue, Node only, StoreCodecError on a bad value) round-trips that value to and from each backing.

Store.Config.file?: boolean (default false) picks the mode, decided once at init:

  • Structured (default): values are StoreValue data, serialized by the shared codec.
  • File (file: true): a byte-native backend (fs, S3, GCS) persists raw bytes byte-exact. set() accepts a Uint8Array or string and stores it untouched; get() hands the exact bytes back. Use for serving assets such as walker.js. The Sheets store is structured-only and rejects file: true at init.

TTL is owned by the cache layer, not the store. The store persists values; a cache wrapper manages expiry. The ttl argument of set is only a hint for TTL-native backends: the fs, S3, GCS and Sheets stores ignore it and keep the value. flow_validate warns when a store sets both file: true and cache, and when a @walkeros/server-transformer-file is wired to a byte-native store that does not set file: true.

Init function (context pattern)

Stores use the same context-based init pattern as other components:

import type { Store } from '@walkeros/core';

export const storeMyStore: Store.Init = (context) => {
  const { config, env, logger, id } = context;
  const settings = config.settings || {};

  return {
    type: 'my-store',
    config,
    get(key) {
      /* ... */
    },
    set(key, value, ttl) {
      /* ... */
    },
    delete(key) {
      /* ... */
    },
    destroy() {
      /* cleanup */
    },
  };
};

Context contains:

PropertyTypePurpose
configStore.ConfigSettings from flow config
envStore.EnvEnvironment dependencies
loggerLogger.InstanceScoped logger
idstringStore identifier
collectorCollector.InstanceReference to collector

Lifecycle

Stores have the simplest lifecycle of all component types:

Startup:  Stores → Destinations → Transformers → Sources
Shutdown: Sources → Destinations → Transformers → Stores
  • Init first: Stores init before all other components so they're available when sources/transformers/destinations start
  • Destroy last: Stores destroy after all other components so transformers can flush during their own destroy
  • No lazy init: Unlike destinations, stores don't support require or deferred activation; they are always eager

Wiring stores via $store.

Bundled mode (flow.json)

Use $store.storeId in a component's env to inject a store instance:

{
  "stores": {
    "assets": {
      "package": "@walkeros/server-store-fs",
      "config": { "file": true, "settings": { "basePath": "./public" } }
    }
  },
  "transformers": {
    "file": {
      "package": "@walkeros/server-transformer-file",
      "config": { "settings": { "prefix": "/static" } },
      "env": { "store": "$store.assets" }
    }
  }
}

The bundler resolves $store.assets to a runtime reference. Invalid references are caught at build time. walkeros validate also catches typos at validation time, including unknown store names and the colon-instead-of-dot mistake (e.g. $store:assets is flagged with the suggested form $store.assets).

Integrated mode (TypeScript)

Pass the store definition object itself, no $store. prefix:

import { startFlow } from '@walkeros/collector';
import { storeFsInit } from '@walkeros/server-store-fs';
import { transformerFile } from '@walkeros/server-transformer-file';

const assets = {
  code: storeFsInit,
  config: { file: true, settings: { basePath: './public' } },
};

await startFlow({
  stores: { assets },
  transformers: {
    file: {
      code: transformerFile,
      env: { store: assets }, // same object as stores.assets
    },
  },
});

Note: In integrated mode, put the same store definition object in stores and in env. The collector replaces it with the initialized instance at startup, matched by object identity (a copy does not match), and only for top-level env keys. The $store. string prefix is a bundler feature: in startFlow it stays a plain string.

Available stores

Built-in cache tier (__cache)

The collector ships a built-in in-memory cache with LRU eviction, TTL, and entry/byte caps. Enable it on any store by setting Flow.Store.cache. No separate package import needed:

{
  "stores": {
    "files": {
      "package": "@walkeros/server-store-fs",
      "config": { "settings": { "basePath": "./data" } },
      "cache": { "rules": [{ "ttl": 60 }] }
    }
  }
}

Use this for the common "cache in front of a slow backing store" pattern (API, GCS, Sheets, etc.).

@walkeros/server-store-fs (filesystem)

File-based store for serving static assets. Server-only.

import { storeFsInit } from '@walkeros/server-store-fs';

@walkeros/server-store-s3 (S3-compatible object storage)

S3-compatible store using s3mini (~26 KB minified, zero dependencies). Works with AWS S3, Cloudflare R2, Scaleway, DigitalOcean Spaces, Backblaze B2, MinIO, and any S3-compatible provider. Structured by default (stored as application/json); set file: true to serve raw bytes byte-exact with the real mime. Server-only.

import { storeS3Init } from '@walkeros/server-store-s3';

Settings:

SettingTypeRequiredDefaultPurpose
bucketstringYes-S3 bucket name
endpointstringYes-S3-compatible endpoint URL
accessKeyIdstringYes-S3 access key ID
secretAccessKeystringYes-S3 secret access key
regionstringNo"auto"AWS region (SigV4 signing)
prefixstringNo-Key prefix for scoping

Primary use case: Serving static files in managed deployments where files live in a bucket rather than being baked into a Docker image.

@walkeros/server-store-gcs (Google Cloud Storage)

Zero-dependency GCS store using raw fetch + GCS JSON API. Built-in auth: ADC on Cloud Run / GKE, or explicit service account JWT. Server-only.

import { storeGcsInit } from '@walkeros/server-store-gcs';

Settings:

SettingTypeRequiredDefaultPurpose
bucketstringYes-GCS bucket name
prefixstringNo-Key prefix for scoping

Credentials live at config.credentials (sibling of settings): a service-account JSON (string | object) for non-GCP environments, set as $secret.<NAME> (an unset secret stops the flow at startup); omit it for ADC. A value missing client_email or private_key is ignored without a log line, and the store falls back to ADC, the runtime's default credentials.

Primary use case: Serving static files on GCP infrastructure (Cloud Run, GKE) where ADC handles authentication.

@walkeros/server-store-sheets (Google Sheets)

Zero-dependency Google Sheets store using raw fetch + Sheets v4 REST API. One row per key, one cell per value (JSON-serialized). Built-in auth shared with the GCS store. Server-only. Structured-only: cells hold structured JSON, so it rejects file: true at init and rejects values carrying a binary (Uint8Array) leaf. Use fs, S3, or GCS for byte-exact serving.

import { storeSheetsInit } from '@walkeros/server-store-sheets';

Settings:

SettingTypeRequiredDefaultPurpose
idstringYes-Spreadsheet ID (segment in URL)
sheetstringNo'Sheet1'Sheet (tab) name
keystringNo'A'Column letter for keys
valuestringNo'B'Column letter for JSON values
headerRowsnumberNo1Header rows to skip when reading

Credentials live at config.credentials (sibling of settings): a service-account JSON (string | object) for non-GCP environments, set as $secret.<NAME> (an unset secret stops the flow at startup); omit it for ADC. A value that is not valid JSON, or is missing client_email or private_key, is ignored without a log line, and the store falls back to ADC, the runtime's default credentials.

Primary use case: Demos and small prototypes where the spreadsheet is the operator-facing UI for tweaking lookup data. Quota: 60 read and 60 write requests per minute per user per project. Enable the built-in cache via Flow.Store.cache on the store declaration to absorb the quota, otherwise quota burns in seconds. Not a production CRM substitute. See Website: Sheets Store for the cache-wiring example.

@walkeros/server-store-api (third-party HTTP API)

Read-only store for a third-party API, such as a CRM. settings.url is an absolute URL with one {key} placeholder after the host; each get is one GET with the encoded key. A 404, 204 or JSON null is a miss; any other failure throws a StepError. Pass an optional bearer token as config.credentials ({ "token": "$secret.NAME" }). set, delete and file: true are refused. Misses are not cached by the cache block.

import { storeApiInit } from '@walkeros/server-store-api';

Stores in flow config

Declare any store consumed by one or more components in the stores section of the flow config and wire it via $store.<id> in component env. The built-in cache tier (Flow.Store.cache) covers the "fast in-memory cache in front of a slow backing store" pattern without a separate memory store.

Accessing stores at runtime

After startFlow(), stores are available on the collector instance:

const { collector } = await startFlow({
  stores: {
    files: { code: storeFsInit, config: { settings: { basePath: './data' } } },
  },
});

// Read/write
await collector.stores.files.set('key', 'value'); // fs keeps it, no expiry
const value = await collector.stores.files.get('key');
await collector.stores.files.delete('key');

Hooks

Store operations (get, set, delete) are wrapped with useHooks during initialization, enabling pre/post interception via the collector's hooks system.

Available hook names:

Hook namePre hookPost hook
StoreGetpreStoreGetpostStoreGet
StoreSetpreStoreSetpostStoreSet
StoreDeletepreStoreDeletepostStoreDelete

Hooks fire on every store operation regardless of which component triggered it (cache system, transformer via env, destination via env, direct access on collector.stores).

const { collector } = await startFlow({
  stores: {
    files: { code: storeFsInit, config: { settings: { basePath: './data' } } },
  },
});

// Intercept all store reads
collector.hooks.preStoreGet = ({ fn }, key) => {
  console.log('Reading key:', key);
  return fn(key);
};

Key differences from other components

AspectSources/Transformers/DestinationsStores
Event flowParticipate in push chainNo push, no chain
next/beforeChain connection fieldsNone (passive)
LifecycleInit after storesInit first, destroy last
requireDeferred activation supportedAlways eager
Interfacepush(event, context)get/set/delete(key)

Dev exports (dev.ts)

Every store package must export schemas and examples via dev.ts, matching the convention used by sources and destinations:

Required files

src/
├── schemas/
│   ├── settings.ts   # Zod schema for store settings
│   └── index.ts      # Re-export + zodToSchema() conversion
├── examples/
│   └── index.ts      # Example Store.Config objects
└── dev.ts            # export * as schemas; export * as examples

Schema pattern

// src/schemas/settings.ts
import { z } from '@walkeros/core/dev';

export const SettingsSchema = z.object({
  mySetting: z.string().describe('Setting description for docs and MCP'),
});

export type Settings = z.infer<typeof SettingsSchema>;

// src/schemas/index.ts
import { zodToSchema } from '@walkeros/core/dev';
import { SettingsSchema } from './settings';
export { SettingsSchema, type Settings } from './settings';
export const settings = zodToSchema(SettingsSchema);

tsup + package.json

// tsup.config.ts
import { defineConfig, buildModules, buildDev } from '@walkeros/config/tsup';
export default defineConfig([buildModules(), buildDev()]);

Add ./dev export to package.json:

"./dev": {
  "types": "./dist/dev.d.ts",
  "import": "./dist/dev.mjs",
  "require": "./dist/dev.js"
}

Why this matters

  • buildDev() generates dist/walkerOS.json at build time
  • MCP tools fetch walkerOS.json from CDN for package discovery and schema validation
  • Website docs import that walkerOS.json and render the settings with <Configuration type="store" data={data} /> instead of hardcoded markdown tables
  • Without dev.ts, a store is invisible to MCP and docs tables rot

Hints (Optional)

Stores can optionally export hints: lightweight, actionable context for AI agents beyond what schemas and examples already provide. Create src/hints.ts:

import type { Hint } from '@walkeros/core';

export const hints: Hint.Hints = {
  'persistence-behavior': {
    text: 'Describes persistence guarantees. See settings schema for options.',
  },
};

Export from src/dev.ts alongside schemas and examples:

export * as schemas from './schemas';
export * as examples from './examples';
export { hints } from './hints';

Most stores don't need hints; only add them for non-obvious behaviors, prerequisites, or troubleshooting patterns.

Setup (optional)

Stores can implement an optional setup() lifecycle to provision external resources, for example creating a SQLite table, initializing an S3 bucket, or running a one-off schema migration. Setup is never invoked by the runtime, push, init, or deploy. It runs only when an operator explicitly types walkeros setup store.<name>.

The signature is (ctx: LifecycleContext<Config<T>, Env<T>>) => Promise<unknown>, where LifecycleContext carries { id, config, env, logger }. Idempotency is the package's responsibility: the framework adds no opinion. Use resolveSetup(ctx.config.setup, DEFAULTS) from @walkeros/core to normalize the boolean | object shape into a concrete options object.

See walkeros-create-destination, walkeros-create-source, walkeros-understanding-destinations, walkeros-understanding-sources, and the walkeros setup CLI documentation for the authoring template and operator workflow.

Related Skills

Source files:

Documentation:

Understanding walkerOS Transformers

Overview

Transformers are middleware for validating, enriching, and redacting events in the walkerOS pipeline. They run in chains at configurable points between sources, collector, and destinations.

Core principle: Transformers transform events. They don't capture (sources) or deliver (destinations); they modify events in-flight.

Use Cases

Use CasePurposeExample
ValidateEnsure events match schema contractsJSON Schema validation
EnrichAdd server-side data to eventsUser segments, geo data
RedactRemove sensitive data before destinationsStrip PII, anonymize IPs

Available Packages

PackageEnvPurpose
@walkeros/transformer-validatebothEnforce JSON Schema contracts on events at runtime
@walkeros/transformer-ga4bothDecode GA4 Measurement Protocol hits into events
@walkeros/server-transformer-botserverAnnotate an automation score and client category
@walkeros/server-transformer-fingerprintserverHash request fields into a rotating visitor id
@walkeros/server-transformer-fileserverServe static files from a store

File step answers

@walkeros/server-transformer-file owns the paths inside settings.prefix ("/" owns every routed path; without a prefix it owns none). A hit answers with the file, a miss inside the prefix answers 404, and a failing store read answers 503. A missing store stops the flow at startup. Route only file requests to a step that owns /, as flow-complete does with a .js case: any other path routed to it answers 404 and its event stops there.

Contract validation

@walkeros/transformer-validate is the runtime arm of a contract. Event shapes live in the top-level contract block; the transformer references one via $contract.<name> in its contract setting and validates the canonical event. mode: "strict" drops invalid events (chain-stop); mode: "pass" (default) annotates event.source.valid and continues so a downstream step can route on the verdict. format: true additionally checks the canonical WalkerOS.Event structure. Filtering is the same mechanism: an inline schema that rejects the unwanted events plus mode: "strict"; there is no separate ignore field. See Website: Validate transformer.

Transformer Interface

See packages/core/src/types/transformer.ts for canonical interface.

Init Function (Context Pattern)

Transformers use a context-based initialization pattern:

import type { Transformer } from '@walkeros/core';

export const transformerMyTransformer: Transformer.Init<Types> = (context) => {
  const { config = {}, env, logger, id } = context;
  // Apply defaults inline: flow.json is developer-controlled, so no
  // runtime validation. Shape checks live in ./schemas and are used by
  // `walkeros validate` and dev tooling, never at runtime.
  const userSettings = config.settings || {};
  const settings = {
    ...userSettings,
    // example default: threshold: userSettings.threshold ?? 100,
  };

  return {
    type: 'my-transformer',
    config: { ...config, settings },
    push(event, pushContext) {
      // Process event
      return { event };
    },
  };
};

Init Context contains:

PropertyTypePurpose
configTransformer.ConfigSettings, mapping, next chain
envTypes['env']Environment deps (stores via $store.)
loggerLoggerLogging functions
idstringTransformer identifier
collectorCollector.InstanceReference to collector
ingestIngestEmpty (_meta only); see Push Context

Instance Methods

MemberPurposeRequired
typeTransformer type identifierRequired
configThe transformer's configRequired
pushProcess event, return modified/falseRequired
initOne-time initializationOptional
destroyCleanup resourcesOptional

Return Values

The push function controls event flow:

ReturnBehavior
{ event }Continue chain with modified event
voidContinue chain, event unchanged
falseStop chain, event dropped
{ event, next }Route via next in place of its own next, then the chain continues
{ event, respond }Continue chain with wrapped respond function
Result[]Fork: each result is its own copy with a derived event.id

A push that throws also stops the chain and drops the event; the collector logs it and counts it in status.failed. Catch a library error inside push when the event should continue.

push(event, context) {
  if (!event.data?.id) {
    context.logger.error('Missing required id');
    return false;  // Stop chain
  }

  event.data.enrichedAt = Date.now();
  return { event };  // Continue with modified event
}

Inline Code Transformers

For simple transformations without external packages, use inline code with the $code: string prefix in JSON configs. The $code: prefix tells the CLI bundler to parse the following string as executable JavaScript:

{
  "transformers": {
    "enrich": {
      "code": {
        "push": "$code:(event) => { event.data.enrichedAt = Date.now(); return { event }; }"
      },
      "next": "validate"
    }
  }
}

Inline code structure:

PropertyPurpose
code.initCode run once during initialization
code.pushCode run for each event

Push code has access to:

  • event: the event being processed
  • context: push context with logger, config, etc.

Return values in push code:

  • Return { event } to continue chain with modified event
  • Return undefined to pass event unchanged
  • Return false to drop event from chain

Example: Filtering internal events

{
  "transformers": {
    "filter": {
      "code": {
        "push": "$code:(event) => { if (event.name.startsWith('internal_')) return false; return { event }; }"
      }
    }
  }
}

Mixing inline and package transformers:

{
  "transformers": {
    "addTimestamp": {
      "code": {
        "push": "$code:(event) => { event.data.processedAt = new Date().toISOString(); return { event }; }"
      },
      "next": "validate"
    },
    "validate": {
      "package": "@walkeros/transformer-validate",
      "config": { "settings": { "format": true } }
    }
  }
}

Pipeline Integration

Transformers run at three points around the collector:

Source → [Source chain] → Collector → [Collector chain] → fan-out → [Destination chain] → Destination
          (source.next)                (collector.next)              (destination.before)

Source Chain

Runs after source captures event, before collector enrichment:

sources: {
  browser: {
    code: sourceBrowser,
    next: 'validate'  // First transformer in the source chain
  }
}

Collector Chain

Runs once per event after collector enrichment, before the destination fan-out. Every destination receives its output; a stop here (or a transformer returning false) drops the event for all destinations:

startFlow({
  next: ['bot', 'validate'], // collector.next in flow.json
  // ...
});

Destination Chain

Runs per destination, after the collector chain, before that destination receives the event. Use it for per-destination work and filtering; a stop here skips only this destination:

destinations: {
  gtag: {
    code: destinationGtag,
    before: 'redact'  // First transformer in the destination chain
  }
}

Chain Linking

Transformers link together via next. The chain starts at the step that names its first transformer, here the source:

sources: {
  express: {
    code: sourceExpress,
    next: 'fingerprint'  // Chain head
  }
},
transformers: {
  fingerprint: {
    code: transformerFingerprint,
    config: {
      settings: { salt: process.env.FINGERPRINT_SALT }, // Required
      next: 'enrich'  // Chain to next transformer
    }
  },
  enrich: {
    code: transformerEnrich,
    config: { next: 'redact' }
  },
  redact: {
    code: transformerRedact
    // No next = end of chain
  }
}

Branching and fan-out

Transformers can redirect events to different chains using the branch() factory from @walkeros/core. The route replaces the transformer's own next for this event; after it, the enclosing array continues:

import { branch } from '@walkeros/core';

push(event, context) {
  return branch(event, 'parser');                  // Single target
  return branch(event, ['a', 'b']);                // Sequence: a, then b
  return branch(event, { many: ['a', 'b'] });      // Fork: two copies
  return branch(event, { stop: true });            // End this copy
}

Conditional routing is built into the next/before properties through the one operator, so no router transformer is needed:

"next": {
  "one": [
    { "match": { "key": "ingest.path", "operator": "prefix", "value": "/api" }, "next": "apiHandler" },
    { "next": "default" }
  ]
}

one entries are evaluated in order, first match wins. A RouteConfig is a disjoint union: each config sets at most one of next (gated link), one (first-match dispatch), many (all-match fan-out), or stop (drop), never more than one. Every match reads { ingest, event } and is evaluated when the event reaches it, so it sees what earlier steps wrote. Besides _meta, ingest holds only what the source's config.ingest maps and what earlier steps write into it: without { "map": { "path": "path" } } in that config, ingest.path is empty and the /api entry never matches. An entry with no match always matches (use it as a fallback). If no entry matches, the event passes through unchanged. Use many (allowed in every chain field) when every matching branch should run: each matching entry becomes an independent copy of the event with its own derived event.id, and each copy finishes the rest of the path on its own (never merged):

"next": {
  "many": [
    { "match": { "key": "event.consent.analytics", "operator": "eq", "value": "true" }, "next": "ga4Pipeline" },
    { "next": "auditLog" }
  ]
}

A consent value is a boolean, and eq compares it as text, so a granted consent matches "true".

Paths and pass-through steps

walkerOS uses two vocabulary terms for chain composition:

  • Path: the multi-step chain that an event walks through a flow's transformers section.
  • Pass-through step (short: pass): a single step inside a path that declares no code and no package. The runtime synthesizes the push, so the step contributes structure without shipping executable code.

A pass-through step ships in three variants. Each variant uses a different operative field; combine them on the same step when it helps.

Variant 1: chain-only (before / next)

A named hop that shares a chain across multiple call sites. Use it to avoid duplicating arrays in before / next references:

{
  "transformers": {
    "validateThenEnrich": {
      "before": ["validate", "enrich"]
    }
  },
  "destinations": {
    "gtag": {
      "package": "@walkeros/web-destination-gtag",
      "config": { "settings": { "ga4": { "measurementId": "G-DEMO123456" } } },
      "before": "validateThenEnrich"
    },
    "meta": {
      "package": "@walkeros/web-destination-meta",
      "config": { "settings": { "pixelId": "1234567890" } },
      "before": "validateThenEnrich"
    }
  }
}
Variant 2: cache-only

A step that declares only a cache block. Useful for deduplication or short-circuit halts. cache.stop: true in a source chain or collector.next halts the event for all destinations (not just the local chain); in a destination.before chain it skips the rest of that chain and the destination receives the cached event:

{
  "transformers": {
    "dedup": {
      "cache": {
        "stop": true,
        "rules": [{ "key": ["event.id"], "ttl": 60 }]
      }
    }
  }
}
Variant 3: mapping-only

A step that declares only a mapping: Mapping.Config. The runtime synthesizes a push that calls processEventMapping and mutates the event in-flight:

{
  "transformers": {
    "redactPII": {
      "mapping": {
        "policy": {
          "user.email": { "value": "[redacted]" }
        }
      }
    }
  }
}

See walkeros-understanding-mapping for the mapping primitives (policy, data, mapping[].name, etc.) and the "Mapping at the transformer position" section for the dual semantic.

Mapping at transformer position vs destination position

mapping is the same field shape (Mapping.Config) in both positions, but the semantic is disambiguated by where the step sits:

PositionWhat mapping produces
DestinationA vendor-shaped payload (the destination consumes data)
TransformerA mutated event that continues through the chain

At the transformer position, only event-mutating fields apply: policy, and per rule condition, policy, name and ignore. consent, include, data and per rule consent, include, remove, batch, settings, extend, data, silent do nothing there. A config.mapping wins over the step-level mapping; next to code or package, a mapping never runs. walkeros validate and the bundle reject all of these (MAPPING_NO_OP), and the collector logs one ERROR line for them when the transformer registers. mapping[].ignore: true drops the event from the chain (not "skip this destination", which is the destination-position semantic).

Closed schema (unknown keys are errors)

Transformer step entries follow a closed schema. Unknown keys at the top of a step are validation errors (UNKNOWN_KEY; the message lists the allowed keys, among them package, import, code, config, env, before, next, cache, state and mapping). This catches misrouted keys (e.g. { rules: [], stop: true } placed at the top of a step instead of nested under cache:) at validate time instead of letting them silently pass through at runtime.

An empty {} is a valid no-op step. Declaring both code and package on the same step is rejected with CONFLICT.

Chain resolution safety

Every chain position runs through one runner (runTransformerChain in @walkeros/collector) over the core continuation stack (startChain / advanceChain). Routes resolve per hop via getNextSteps(spec, root), which needs the root { ingest, event } and returns NextSteps: ids (plus a continuation for the rest of a sequence), a stop, or forks. A member next that leads back to a step already on its insertion path is skipped, and each copy is capped at 256 steps. A step listed twice in an array runs twice. An unknown transformer id is logged as an ERROR line and skipped, and the chain continues; walkeros validate reports it as an UNKNOWN_ROUTE_TARGET error. Static tools without an event use getRouteGraph(spec, transformers?), which enumerates every branch of the same compiled route.

Composition principle

A transformer owns its own chain. When a chain references a transformer by name, that transformer's own before chain runs before its push, and its next chain after, both recursively, with cycle detection. This holds inside an explicit array too: the array is the backbone, and a member's own next is inserted right after that member before the array continues (["bot", "validate"] with bot.next = "foo" runs bot, foo, validate). Cache halt signals (cache.stop: true) in a source chain or collector.next propagate to every destination. The grammar's recursive Route shape (string | Route[] | RouteConfig) compiles element-by-element, so sequences can mix transformer IDs and inline one / many / next routes (next: ["dedup", { one: [...] }] is valid). This is the model to default to when adding new chain primitives.

See walkeros-understanding-flow for the full connection rules between sources, transformers, and destinations.

Push Context

The push function receives a context with event metadata:

PropertyPurpose
configTransformer configuration
envEnvironment dependencies
loggerScoped logger for output
idTransformer identifier
collectorAccess to collector instance
ingestRequest metadata from source
push(event, context) {
  const { logger, id, ingest } = context;

  logger.debug('Processing', { transformer: id, event: event.name });

  // Access request metadata if available
  if (ingest?.ip) {
    event.data = { ...event.data, clientIp: ingest.ip };
  }

  return { event };
}

Response Delegation (env.respond)

Transformers can customize HTTP responses by calling context.env.respond?.({ body, status?, headers? }). This is useful for validation transformers that reject events with custom error responses, or transformers that short-circuit the pipeline. First call wins (idempotent). The respond function is optional, only present when the source provides one.

Transformer Paths

PathDescription
packages/transformers/Transformers for both platforms (validate, ga4)
packages/server/transformers/Server transformers (bot, fingerprint, file)

Related Skills

Source Files:

Documentation:

Using the walkerOS CLI

Overview

The walkerOS CLI (walkeros) validates, simulates, tests and bundles event collection flows. It does not run them: a built server flow runs with runneros start from @walkeros/runner (the walkeros/flow image).

Core workflow:

  1. Configure - Write Flow.Json JSON config
  2. Validate - walkeros validate flow.json
  3. Test - Simulate events (mocked) or push (real)
  4. Bundle - walkeros bundle flow.json -o dist/
  5. Run - runneros start dist/flow.mjs, or deploy through the app

Quick Start

# Install (Node.js 22.12 or later)
npm install -g @walkeros/cli

# Bundle a flow into dist/
walkeros bundle flow.json -o dist/

# Test with simulated event
walkeros push flow.json -e '{"name":"page view"}' --simulate destination.demo

# Push real event
walkeros push flow.json -e '{"name":"page view"}'

An event needs name ("entity action"). One written as entity and action only is rejected as invalid: a simulation prints mapping: none (skipped before mapping) and no calls, a real push sends nothing, and both still print success: true.

Commands Overview

CommandPurposeSafe?
bundleGenerate JS bundle from config✅
pushPush an event: real API calls on a server flow, recorded on a web flow; --simulate mocks steps⚠️
setupRun a component's setup() to provision external resources⚠️
validateValidate configs/events✅
cacheManage caching✅
authCheck who the token in WALKEROS_TOKEN belongs to✅
apiCall the walkerOS app API, or list its operations⚠️
diagnosticsShow the CLI version, the app URL, and whether the app answers✅
telemetryShow, enable or disable anonymous usage telemetry✅

For detailed command reference, see commands-reference.md.


Common Workflows

Development Workflow

1. Write flow.json config
2. Bundle: walkeros bundle flow.json
3. Simulate: walkeros push flow.json -e event.json --simulate destination.demo
4. Fix issues, repeat 2-3
5. Push test: walkeros push flow.json -e event.json
6. Deploy: in the walkerOS app, or
   walkeros api POST /api/projects/{projectId}/flows/<flowId>/deploy
   (see the CI recipe in the CLI docs)

Multi-Flow Development

# Bundle specific flow
walkeros bundle flow.json --flow myFlow

# Bundle all flows (--all needs an output directory)
walkeros bundle flow.json --all -o dist/

# Test specific flow
walkeros push flow.json --flow myFlow -e event.json --simulate destination.demo

Local Development Server

# Build, then start the artifact with the runtime (a separate package)
walkeros bundle flow.json -o dist/
npx --package=@walkeros/runner runneros start dist/flow.mjs --port 3000

Server port note: Under runneros the runtime owns the port: it listens on --port (or PORT, default 8080), serves /health and /ready, and hands every other request to the flow's HTTP handler. A source's own port setting is not used there.


Flow.Json Configuration

Minimal Config

{
  "version": 4,
  "flows": {
    "default": {
      "config": {
        "platform": "web",
        "bundle": {
          "packages": {
            "@walkeros/web-destination-gtag": { "version": "^4" }
          }
        }
      },
      "destinations": {
        "gtag": {
          "package": "@walkeros/web-destination-gtag",
          "config": {
            "loadScript": true,
            "settings": { "ga4": { "measurementId": "G-XXXXXX" } }
          }
        }
      }
    }
  }
}

Without "loadScript": true walkerOS does not load gtag.js: unless the page loads it itself, events go nowhere while the push still reports them as done.

Config Structure

{
  "version": 4,
  "flows": {
    "<flowName>": {
      "config": {
        "platform": "web" | "server",  // Platform (required)
        "url": "",                      // Optional: public URL, read by $flow.<name>.url
        "settings": {},                 // Platform-specific settings (optional)
        "bundle": {
          "packages": {},               // NPM packages pacote will install
          "overrides": {},              // Transitive dep version pins (npm-style)
          "traceInclude": []            // Optional: nft escape hatch (paths/globs)
        },
        "observe": {}                   // Optional, web flows only: url + binding, level, sample
      },
      "sources": {},                    // Event sources
      "destinations": {},               // Event destinations (each with its own mapping)
      "transformers": {},               // Transformer chain (optional)
      "stores": {},                     // Key-value stores (optional)
      "collector": {},                  // Collector configuration
      "variables": {}                   // Flow-level $var values (optional)
    }
  }
}

You do NOT need npm install for step packages. flow.json's config.bundle.packages is the single source of truth. Pacote installs them transparently when you run walkeros bundle. Only @walkeros/cli belongs in your project's package.json (as a devDependency).

config.bundle.overrides pins transitive dependency versions. Use it when a vendor SDK's declared peer/dep range conflicts with another required version in the same tree. Example: {"@amplitude/analytics-types": "2.11.1"} forces that exact version everywhere in the install graph. Direct packages specs always win over overrides; overrides only substitute transitive resolution.

Schema version stays at 4. Build-time fields live under flow.<name>.config.bundle.{packages, overrides, traceInclude, env}. A config.bundle.external field fails walkeros validate (Unrecognized key); the bundle only warns and builds, so remove it.

config.observe bakes only public values. For web flows, walkeros bundle emits the url + binding connect pair statically into the bundle; the per-session credential arrives out-of-band at runtime (?elbObserve= URL param), so no secret ever enters the artifact. A partial pair (only one of the two set) warns at build time and wires nothing. A server flow ignores config.observe: the runner reads its level from WALKEROS_OBSERVE_LEVEL instead (see Runtime environment below).

For detailed configuration options, see flow-configuration.md.


Testing with Step Examples

Simulate with --step

Target a specific step and provide input as SourceInput ({ content, trigger? }):

# Simulate a source step with trigger metadata (a click needs its selector)
walkeros push flow.json --simulate source.browser --event '{"content":"<button data-elb=\"cta\" data-elbaction=\"click\">Sign up</button>","trigger":{"type":"click","options":"button"}}'

# Simulate a destination step with an event
walkeros push flow.json --simulate destination.gtag -e '{"name":"order complete","data":{"value":149.97}}'

A click trigger without options finds no element (Trigger: element not found for selector "undefined") and the source emits no events. Example output of the destination command (a destination step prints its matched mapping key and every recorded vendor call, arguments as compact JSON cut at 300 chars):

success: true
  destination.gtag
    mapping: none
    call window.gtag("config","G-XXXXXX",{})
    call window.gtag("event","order_complete",{"send_to":"G-XXXXXX"})
  Duration: 467ms
  • mapping: <key> when a rule matched, mapping: none when the destination pushed without a rule, mapping: none (skipped before mapping) plus no calls when nothing was sent. Then why, when consent is the reason: pending: waits for consent (require) (never started; names the unmet require entries) or skipped: consent (requires marketing; granted functional) (started, event denied by its consent check; after mapping: <key> it is that rule's own consent). Source, transformer and collector steps print their recorded calls, then event <json> or no events; a failed step prints error: <message>. A source step records the walker commands the source issues itself as elb calls in its own call shape, e.g. call elb("walker consent",{"functional":true,"marketing":true}) for a CMP, call elb("user",...) and call elb("session",...) for the session source. Not recorded: the collector's own commands (starting consent, globals, custom, shutdown) and the wiring commands on, hook, destination.
  • --json puts the same data under simulations (one { step, name, events, calls: [{ fn, args, ts }], mappingKey?, skipped?, error? } per simulated step, in order; skipped is { reason: 'pending', require } or { reason: 'consent', required, granted }; a multi-destination simulate stops at the first failure). stdout is pure JSON, logs go to stderr, so | jq works without --silent. Values pass through toPrintable (@walkeros/core/node): Error to { name, message }, Buffer to UTF-8, bigint to string, Map/Set to arrays, cycles to "[Circular]".
  • Text and JSON output (and MCP flow_simulate and flow_push) are scrubbed with scrubSecrets, the same redactor the loggers use: service accounts, PEM keys, Authorization, access_token, credential-named fields and high-entropy runs show as ***. The values of every $secret.NAME the flow references (set in the env, 6+ chars) are masked exactly, raw, JSON-escaped and URL-encoded. The step still receives the real values. With --json, a number whose printed form contains a known value becomes "***", so the JSON still parses. Simulate and a real push of a flow config route the flow's own logs through the masking CLI logger, so flow DEBUG lines need --verbose; the run's own CLI lines mask the known values too. The runner masks the secret values it fetches; values from --env-file or the container env get the pattern rules only.
  • Only the target starts. Destination simulate keeps only the target destination (no source, no other destination initializes); the flow's transformers still start, and a store without a mock env runs for real. Source simulate keeps only the simulated source (no browser page view or CMP decision from another source); captured events stop at the collector. A source package that declares examples.env.simulation (SQS, Pub/Sub pull) runs on its mock client with those calls recorded.
  • No real vendor calls (destinations only). A package destination whose export has no mock env (examples.env.push) is refused before the flow starts (an inline code step has no package and runs as given): No mock env for <package> export <exportName>: simulate would call the real vendor. Add examples.env.push to the package's dev examples. A named export of a package version without exportExamples is refused with ...: this package version predates export-keyed examples; use a version with exportExamples or a local path. A store whose package ships a mock env runs on it; a store without one runs for real. Sheets and GCS still need a real-format service-account key, since they sign their token request before the mocked fetch answers (see the CLI docs, "Secrets in simulate").
  • --ingest <json|file|url> supplies the request context (a JSON object, e.g. {"ip":"203.0.113.7","userAgent":"Mozilla/5.0"}) for transformer.*, collector.* and destination.* simulation. It reaches the destination before chain and context.ingest (or collector.next); the CLI always sets _meta. Rejected for source.* and a real push: --ingest applies to transformer, collector and destination simulation only. A programmatic pushCommand({ ingest }) on a real push or source simulate errors the same way; simulateTransformer, simulateCollector and simulateDestination take ingest directly.
  • --consent <json|file|url> (a JSON object of booleans) is the collector consent any simulated step starts from, sources included (programmatic consent; MCP and collector: state.consent). startFlow applies it before the flow runs, as a consent command only the simulated step hears: a require: ["consent"] destination starts, its consent check sees the granted keys, and a Consent Mode (como) target records its gtag('consent','update',...). A consent-gated source (the session source with settings.consent) starts from it; without it, no events. The starting consent is never recorded as a source call. Rejected for a real push only: --consent sets the collector's starting consent for a simulation; a real push uses the flow's own consent.
  • --command <config|consent|user|run> makes a destination simulation run collector.command(name, event) instead of a push (a step example's command); the event is the command's data and its calls are recorded. Starting consent applies first, then the command. Elsewhere: --command applies to destination simulation only.
  • --page-url <url> sets the page of a simulated web source (absolute http/https only). Precedence: --page-url, then the input trigger's options.url, then http://localhost. Elsewhere: --page-url sets the page of a simulated web source; for request context use --ingest.

Consent mental model (what decides whether a destination receives an event):

SettingEffectIn simulate
require: ["consent"]Destination stays pending until the collector has ANY consent state--consent starts it; else pending: waits for consent
config.consent: { marketing }Each event checked against collector consent plus event.consent; denied = skippedgrant via --consent or the event; else skipped
event.consentPer event, counted on top of collector consentpart of the -e event
  • Trace-mode limit. Simulate injects the mock env before init, so clients built in init (BigQuery writer, Pub/Sub client) are recorded. Runtime trace mode (an Observe session at trace level) wraps the env per push only, so calls through a client built in init are not recorded there.

Same flow via MCP (flow_simulate)

From an AI assistant the equivalent tool is flow_simulate. A few specifics:

  • step is required. Pass the target as "type.name", e.g. "source.browser", "collector.default", "destination.gtag", or "transformer.router". There is no all-steps mode.
  • Four step types: source, collector, transformer, destination. Source-step event shape is { content, trigger? }, where content is the walkerOS event { name, data } and trigger is optional { type?, options? }. There is no env field. Destination and transformer steps take a plain walkerOS event { name, data, consent? }.
  • collector is the enrichment step plus the collector chain. It takes a post-next partial event plus an optional state snapshot { consent?, user?, globals?, timing? }, applies the collector's enrichment, then runs collector.next, and returns every event the destinations would receive (none when a stop drops it, several when many forks it).
  • transformer, collector and destination steps accept an optional ingest (a raw ingest without _meta). Supply it to test a request decoder standalone (a GA4 decoder reading ctx.ingest.url: pass ingest: { url: "..." }), or to see the client IP and user agent a conversion-API destination sends (ingest: { ip, userAgent }). The result is scrubbed of credentials like the CLI output.
  • Sources are simulatable as a step, including the @walkeros/source-demo demo source.
  • configPath is a file path, an http(s) URL or inline JSON.
  • A flow saved in the walkerOS app runs locally through a config link: the hosted walkerOS MCP server's flow_manage (action: "config_url") returns a URL that lasts about ten minutes, and flow_simulate, flow_bundle, flow_push and flow_load take it as their config input.

flow_load loads a flow from a local path, URL or inline JSON. In a returned config, package names, platform, versions and IDs come back literally; every other string value comes back wrapped, for example "title": "<user_data>Home</user_data>". Remove the wrapper from each value before writing the config back, or it becomes part of the value.

From a terminal, walkeros diagnostics (or --json) reports the CLI version, the app URL and where it comes from, and whether the app answers its health route:

cli 4.x.y
app https://app.walkeros.io (default)
health reachable, status ok, app 3f2a91c

Validate flow config

Validate every flow and every step in one run:

walkeros validate flow.json

What validate guarantees:

  • Everything by default: schema, references (resolved per flow with the runtime resolver; an unknown $var, a missing store, an unresolvable $flow or a cycle is an error), routes, the root contract, the bundle preflight, cross-step example compatibility, the examples of @walkeros/transformer-validate steps against their linked contract, and each step's settings against its package schema (an unknown key is an error where the package closes its schema, other findings are warnings; --offline skips the fetch).
  • Narrowing is explicit: --flow <name> narrows per-flow checks (file-level checks still run); --path <section.key> checks one entry in every flow that has it, or only in --flow.
  • Every skip is reported in details.skipped and the text output, with a code. A step that imports a named export (step import, else bundle.packages[pkg].imports[0]) is checked against that export's entry in the package's exportSchemas; a single-export package uses settings. The bundler runs the step's import, else the default export, never an imports entry, so set import on the step to check and run the same export. A multi-export package version without exportSchemas, or one that does not list its exports, is a skip (NO_SETTINGS_SCHEMA).
  • The result states its scope: a Scope: line and details.scope.
  • --strict fails on warnings and skips (exit 2; validate() and flow_validate return valid: false) and makes contract example disagreements errors (exit 1). It never changes what runs.
  • Stable codes: every error and warning has a code; branch on it, never on the message.
  • Not checked, also under --strict: an unknown key at the file root, on a flow or in its config is dropped without a message (a misspelt "destinaitons" passes), and a mapping rule's settings are not checked. A passing validate is a first filter; simulate before you bundle.

For full details on writing and testing with step examples, see using-step-examples.


$code: Prefix (Inline JavaScript)

Embed JavaScript functions in JSON configs:

{
  "fn": "$code:(event) => event.data.price * 100",
  "condition": "$code:(event) => event.data?.value > 100"
}

Important: The CLI bundler converts $code: strings to actual JavaScript functions during build. This is essential for mappings in JSON configs.

For mapping patterns, see understanding-mapping.


Quick Reference

Bundle Command

walkeros bundle <config> [options]

Options:
  -o, --output <path>  Write to a file or directory (a server flow also to a .tar.gz/.tgz archive)
  -f, --flow <name>    Bundle one flow of a multi-flow config
  --all                Bundle all flows (needs -o)
  --stats              Show bundle statistics
  --json               JSON statistics (implies --stats); with -o the JSON replaces the bundle at that path
  --no-cache           Disable the package and build caches
  --release <id>       Stamp a config release id on event.source.release
  --manifest [source]  Build from a manifest (see "Manifest builds")
  -v, --verbose        Verbose output
  -s, --silent         Silent mode

Output:

  • Web: dist/walker.js (single self-contained IIFE). Every browser entry (this bundle, managed wraps, walker.js) calls the Tag Mode loader moin() from @walkeros/web-core first; it stays idle unless the page was opened with ?elbMoin.
  • Server: dist/{flow.mjs, package.json, node_modules/} (always a directory; nft-traced)
  • Server archive: flow.tar.gz / flow.tgz (the server bundle directory packed into a single gzip file)

Use -o ./dist/walker.js for web, -o ./dist/ for a server directory, or -o ./flow.tar.gz for a server archive. Web single-file bundles do not support archive output. Without -o the bundle is written to stdout, which for a server flow is flow.mjs alone, without its node_modules/.

Also: --release <id> stamps a config release id on event.source.release (it beats an authored collector.release; the default is a 12-hex content id of config, resolved packages and CLI version, so the same input gives the same release; env and secret values are not inputs, only their names); --manifest [url|path] builds from a manifest (see "Manifest builds" below).

Push Command

walkeros push <config> [options]

Options:
  -e, --event <json|file|url>   Event to process (required; a mistyped file path is read as an event name)
  -f, --flow <name>              Flow to use
  -p, --platform <web|server>   Platform override
  --simulate <step>              Simulate a step (repeatable for destination.*). Format: source.NAME | destination.NAME | transformer.NAME | collector.NAME
  --mock <step=value>            Mock a step with a specific return value (repeatable); chain members via destination.NAME.before.ID or collector.next.ID
  --ingest <json|file|url>       Request context for transformer/collector/destination simulate
  --consent <json|file|url>      Starting collector consent for a simulation, any step
  --command <name>               Destination simulate runs this command (config|consent|user|run) instead of a push
  --page-url <url>               Page URL of a simulated web source (http/https)
  --snapshot <source>            JS file to eval before execution (sets global state)
  -o, --output <path>            Write the result to a file
  --json                         JSON on stdout (incl. simulations); logs to stderr
  -v, --verbose / -s, --silent   Verbose or silent output

On a real push, a destination that writes to the console still prints there: with @walkeros/destination-demo, push --json puts the demo's lines on stdout before the JSON, so the output does not parse with jq.

Push takes a flow config. A file walkeros bundle wrote fails as push input (see the CLI docs, "Input types").

Validate Command

walkeros validate <input> [options]

Options:
  -t, --type <type>    Validation type (default: flow). Also: event, mapping, contract
  --path <path>        Check one entry in every flow that has it (e.g. destinations.snowplow, stores.cache)
  -f, --flow <name>    Narrow per-flow checks to one flow
  --strict             Fail on warnings and skipped checks
  --offline            Do not fetch package schemas (settings not checked)
  --json               JSON output
  -o, --output <path>  Write the result to a file

Exit codes:
  0 = Valid (skips alone do not change it)
  1 = Errors found (contract violations count as errors under --strict)
  2 = No errors, but warnings or skipped checks found (with --strict only)
  3 = Validation could not run (missing file, invalid JSON, option that does not apply)

Running a built flow (runneros, not the CLI)

The CLI has no run command. The runtime is a separate package, @walkeros/runner, with the binary runneros:

runneros start [artifact] [options]

Options:
  --flow-id <id>        App flow ID (enables heartbeat and secrets)
  --project <id>        Project ID (default WALKEROS_PROJECT_ID; required with --flow-id)
  -p, --port <number>   Port (default: 8080)
  --env-file <path>     Load a dotenv file first (existing env wins; a group- or other-readable file is refused)
  --json                JSON output
  -v, --verbose         Verbose output
  -s, --silent          Silent mode

[artifact] (or BUNDLE, or flow.mjs in the working directory) is a .mjs/.js/.cjs entry, a .tar.gz/.tgz archive holding flow.mjs, or an http(s) URL to either. The runtime does not bundle: it has no bundler installed; a local flow config is refused and one fed by URL or stdin fails at import. Never generate a container or command that hands flow.json to the runtime; bundle first. The full reference is "Deploying a server flow" below.

Setup Command

# Provision external resources for one component (explicit only, never
# triggered by push, real or simulated, a deployment, or the runtime)
walkeros setup <kind>.<name> [-c ./flow.json] [-f <flow>] [--json] [--verbose] [--silent]

The target uses the same <kind>.<name> syntax as walkeros push --simulate (source, destination, or store). Package resolution follows the flow's pins exactly like bundle: the version from config.bundle.packages (or inline @scope/pkg@x.y.z, bundle pin wins) is downloaded through the shared pacote pipeline and cache, then imported from the extracted tree. Setup works via npx without a local install; path: entries in config.bundle.packages support local development with a built package directory. Today it cannot load the AWS (SQS) or GCP (Pub/Sub) source, nor the GCS or Sheets store: it exits 1 and creates nothing (see the CLI docs, "Setup command").


Bundler Gotchas

  • Circular copies: Never include the output directory itself (e.g., include: ["./dist"] when output is dist/). The CLI detects this and errors.
  • include is server-only: root include (default ./shared when that folder exists) is copied next to server bundles only. A web build copies nothing and logs include is ignored for web builds: a browser bundle cannot read local folders. A server build served from the build cache copies include too. A manifest (hosted) build refuses include only on a server flow; on a web flow it is accepted and ignored.
  • Caches and temp dirs: caches live in $TMPDIR/cache/ (os.tmpdir(); on Linux TMPDIR is often unset, read $TMPDIR/... in output as /tmp/...). Package entries are written to a temp sibling and renamed in once complete; a failed write warns Package cache write failed for <name>@<version>: <message>. The build continues without caching it. Entries from an older CLI miss once. walkeros cache clear also removes interrupted writes. Build cache keys cover a local package's (path) built contents and the CLI version, so a rebuild needs no cache clear; cache clear --builds removes both build caches. Each run works in $TMPDIR/walkeros/<kind>/<6 chars> (push, build, bundle, wrap, archive, setup).
  • Runtime paths: runneros sets CWD to the bundle directory. File paths in settings resolve relative to the bundle, not the project root.
  • Component names: Source, transformer, destination, and store names must be valid JavaScript identifiers (camelCase). Validate refuses a hyphenated name such as gtag-wrapper (INVALID_COMPONENT_NAME); use gtagWrapper.
  • Range conflicts: When two transitive consumers declare incompatible ranges for the same dep (e.g., arrify@^3.0.0 vs arrify@^2.0.0), the bundler resolves the chosen range to a concrete version and nests non-satisfying specs under their consumer. If a post-install warning surfaces declared-vs-installed mismatches, pin the dep in config.bundle.overrides. Set BUNDLER_STRICT_RANGES=0 to bypass strict range validation when the npm registry is unreachable.

Server bundles use nft tracing

Server flows are bundled with @vercel/nft. The CLI:

  1. Pacote installs every package declared in flow.<name>.config.bundle.packages into a per-build install root. Users do not run npm install for step packages; only @walkeros/cli lives in their package.json.
  2. esbuild stage 1 externalizes all step packages.
  3. esbuild stage 2 emits a small ESM flow.mjs that imports from those externalized packages.
  4. nft traces flow.mjs, finds every file actually reachable at runtime (including __dirname-loaded .proto files and other assets), and copies only those files into dist/node_modules/.

There is no walkerOS.bundle.external annotation. nft figures it out.

After tracing, every declared bundle.packages entry the bundle actually imports (bare imports of the built bundle, incl. imports) must be in the trace, else the build throws nft-trace: resolved packages missing from trace. A declared package that is neither imported nor traced (e.g. a type-only dependency) only warns: Package <name> is declared in bundle.packages but nothing imports it; it is not in the bundle. Remove it from bundle.packages if unused.

Bundle directory (the server flow's unpacked artifact):

dist/
├── flow.mjs        # ESM entry, expects to be at /app/flow/flow.mjs in prod
├── package.json     # informational sidecar (not used by the runner)
└── node_modules/    # only the files nft traced

The same directory can be packed into a .tar.gz/.tgz archive (see the Bundle Command section), and runneros start accepts either form. Web flows are a single dist/walker.js, served by any static host.

Canonical Dockerfile

ARG WALKEROS_VERSION

FROM node:24-alpine AS builder
ARG WALKEROS_VERSION
WORKDIR /build
RUN npm init -y && npm install --save-dev @walkeros/cli@${WALKEROS_VERSION}
COPY flow.json ./
RUN npx walkeros bundle flow.json -o dist/

FROM walkeros/flow:${WALKEROS_VERSION}
COPY --from=builder /build/dist/ /app/flow/

Build with docker build --build-arg WALKEROS_VERSION=<version> ..

Notes:

  • The build stage only needs @walkeros/cli. flow.json drives every step package install; pacote handles it.
  • COPY --from=builder /build/dist/ /app/flow/ copies the whole directory (flow.mjs + package.json + node_modules/).
  • The image sets BUNDLE=/app/flow/flow.mjs, PORT=8080 and CMD ["runneros", "start"]. No command and no BUNDLE needed.
  • One version pins both: walkeros/flow:<v> contains @walkeros/runner@<v>, matching @walkeros/cli@<v>. Images older than @walkeros/runner have no runneros; check with docker run --rm walkeros/flow:<v> runneros --version. Do not assume latest or a major tag carries it.

Escape hatch: traceInclude

If nft cannot statically reach a runtime asset (rare: require() of a path constructed from a runtime variable), declare it explicitly under flow.<name>.config.bundle.traceInclude. Paths and globs both work; both resolve against the install root, not the project directory:

"flows": {
  "default": {
    "config": {
      "platform": "server",
      "bundle": {
        "packages": { "@walkeros/server-destination-gcp": {} },
        "traceInclude": [
          "node_modules/some-pkg/data/*.json",
          "node_modules/another-pkg/lib/runtime-loaded.js"
        ]
      }
    }
  }
}

Cache (CI)

The bundler caches pacote downloads under process.env.NPM_CACHE_DIR (default $TMPDIR/cache/npm). On CI, persist that path with actions/cache:

- uses: actions/cache@v4
  with:
    path: .walkeros-cache/npm
    key: walkeros-${{ hashFiles('**/flow.json') }}
- run: npx walkeros bundle flow.json -o dist/
  env:
    NPM_CACHE_DIR: ${{ github.workspace }}/.walkeros-cache/npm

CI smoke check: cd dist && node -e "import('./flow.mjs').then(()=>console.log('ok'))" plus du -sh node_modules (a server flow with the BigQuery destination holds about 20 MB in about 1,300 files; use .dockerignore).


Deploying a server flow

Build, run, orchestrate

StageHasMust not have
Build (walkeros bundle)Registry access, esbuild, nft, a Node toolchainProduction secrets or traffic
Run (runneros start)The artifact, its runtime secrets, the portA package manager, registry egress, a bundler
Orchestrate (you, CI, the walkerOS app)The decision of what runs whereThe ability to execute the artifact

The runtime cannot bundle because no bundler is installed in it, not because a flag forbids it. Generate two steps (bundle, then start), never one.

Artifact shape

  • Server: a DIRECTORY, dist/flow.mjs + dist/package.json + dist/node_modules/. Every COPY and -v carries the whole directory (COPY dist/ /app/flow/, -v "$PWD/dist:/app/flow:ro"), never flow.mjs alone. Or one file: walkeros bundle flow.json -o flow.tar.gz.
  • Web: a single dist/walker.js for any static host. The runtime has no static-file mode; to serve files from a server flow use include + @walkeros/server-store-fs + @walkeros/server-transformer-file.

Runtime environment

VariableMeaning
BUNDLEArtifact path or URL (image default /app/flow/flow.mjs)
PORTPort (default 8080)
WALKEROS_FLOW_ID, WALKEROS_PROJECT_IDApp flow and project; with a runner token they enable secrets and the heartbeat
WALKEROS_DEPLOYMENT_IDDeployment ID sent with every heartbeat; the app accepts a heartbeat only with the deployment the runner token belongs to, else every beat logs Heartbeat rejected (404) at WARN
WALKEROS_FLOW_RELEASERelease the loaded flow must carry, or the boot fails
WALKEROS_FLOW_NAMEFlow name it must carry; checked only with WALKEROS_FLOW_RELEASE
WALKEROS_DEPLOY_TOKEN / WALKEROS_TOKENRunner token (wos_run_...), deploy token first; a person's automation token is refused. Env only, never a config file
WALKEROS_APP_URLApp base URL (default https://app.walkeros.io)
WALKEROS_HEARTBEAT_INTERVALSeconds (default 60, minimum 10)
WALKEROS_CACHE_DIRError cache dir (then CACHE_DIR, then XDG ~/.cache/walkeros)
WALKEROS_OBSERVE_LEVELoff, standard or trace

Secrets are fetched ONCE at boot when connected (401/403 is fatal). There is no config polling and no hot-swap: a new flow version is a rebuild and a redeploy. /health is always 200; /ready is 200 once the collector is constructed and, with WALKEROS_FLOW_RELEASE set, the loaded flow matches it; 503 otherwise. A mismatch exits with Artifact mismatch: expected <name>@<release>, loaded <name>@<release>.

Constraints

  • Filesystem: archive, URL and stdin inputs write to /app/flow/, so a read-only root filesystem works only with a local .mjs artifact in place. Point WALKEROS_CACHE_DIR at a writable path (the image's /app/cache).
  • Architecture: the image is linux/amd64 only.
  • Default store: a state step without store uses the in-process __cache store, correct only on ONE instance. Do not scale such a flow out without a shared store.

Full page: website/docs/apps/runtime.mdx.

Build-time values and package specs

  • config.bundle.env declares the values web $env.NAME references resolve to at bundle time (literal strings only; a reference inside a value is refused). A local build layers it over the shell env; a hosted or manifest build uses it alone. $flow siblings resolve against the ENTRY flow's bundle.env. Values end up in a public bundle: never secrets. Server flows ignore it; their $env is read at runtime.
  • Package specs must be registry version, range or tag, for direct pins, overrides values and transitive dependencies. Git, file:, link: and tarball URLs fail with UNSUPPORTED_PACKAGE_SPEC. Use a local path for development.

Manifest builds

walkeros bundle --manifest <url|path> (a bare flag reads BUILD_MANIFEST_URL) builds from a JSON manifest: version: 1, toolchain (exact CLI version, else TOOLCHAIN_MISMATCH), flowConfig, flowName?, buildEnv?, artifacts[] (outputName, putUrl, contentType?, headers?, plus target for a bundle or target: "wrap" + platform + skeleton for a wrap), and resultPutUrl. The CLI PUTs each artifact, then a result { ok, toolchain, artifacts: [{ target, outputName, bytes, sha256 }], error?: { code, message, outputName? } }. Unknown top-level keys are ignored; artifact entries are strict; local filesystem inputs are refused with LOCAL_PATH_NOT_ALLOWED. Only --json, -v, -s combine with it. Schemas: BuildManifestSchema, BuildResultSchema, BUILD_ERROR_CODES from @walkeros/cli.


Troubleshooting

Bundle Fails

  1. Check the config: walkeros validate flow.json (add --flow <name> to narrow to one flow)
  2. Check package names: Ensure packages exist on npm
  3. Clear cache: walkeros cache clear

Events Not Processing

  1. Validate event: walkeros validate event.json --type event
  2. Check mapping: Event must match entity/action in mapping
  3. Use simulate first: walkeros push flow.json -e event.json --simulate destination.demo -v

Destination Not Found in Simulation

Destination "NAME" not found in collector. Available: ... means the flow has no destination with that key; pick one from the list.

Destination Silent (0 Events Received)

Read the line under mapping::

  1. pending: waits for consent (require): the destination has require: ["consent"] and never started. Pass starting consent: --consent '{"functional":true}' (MCP: state.consent). Do not remove require to test.
  2. skipped: consent (requires marketing; granted functional): its consent setting denied the event. Grant the key in --consent or in the event's consent.
  3. Check mapping: The event name must match a mapping rule (entity/action keys). Unmapped events pass through unmodified.
  4. Check policy: Policy shapes the payload, not the rule choice at a destination; verify policy isn't redacting fields the payload needs.

Web Simulation Transport

Web flows run in JSDOM. fetch, navigator.sendBeacon and XMLHttpRequest are polyfilled as tracked no-ops, so the flow's own requests are recorded, not sent; scripts, iframes and stylesheets the page loads are still fetched. Captured network calls are included in PushResult.networkCalls when present.

Local Packages Not Found

Use absolute or relative paths in flow.<name>.config.bundle.packages:

{
  "config": {
    "platform": "web",
    "bundle": {
      "packages": {
        "my-destination": {
          "path": "./local/my-destination"
        }
      }
    }
  }
}

An App Answers 426

The walkerOS app answers 426 Upgrade Required to a CLI below its minimum version. walkeros api prints the answer body, which carries the upgrade message and minVersion, and exits 1. Upgrade the CLI and retry:

npm install -g @walkeros/cli@latest

See Upgrading for the answer body and the upgrade checklist.


Where CLI Lives

LocationPurpose
packages/cli/CLI source code
packages/cli/src/commands/Command implementations
packages/cli/examples/Example flow configs
website/docs/apps/cli.mdxFull CLI documentation

Networking

Outbound requests to the walkerOS app (WALKEROS_APP_URL, default https://app.walkeros.io) carry X-WalkerOS-Client: cli and X-WalkerOS-Client-Version headers so the host can attribute usage and enforce its minimum version. No PII.

Telemetry

The CLI supports anonymous usage telemetry (installation id, command name, outcome, duration, environment) to help improve the tool. Telemetry is off by default and requires explicit opt-in. No persistent identifier is written before consent.

  • Opt in: walkeros telemetry enable.
  • Opt out: walkeros telemetry disable.
  • Current state: walkeros telemetry status.
  • Forced off (override): set DO_NOT_TRACK=1 or WALKEROS_TELEMETRY_DISABLED=1.
  • Debug: set WALKEROS_TELEMETRY_DEBUG=1 to print the payload to stderr (requires opt-in; no network traffic).
  • Docs: see apps/telemetry for the full event list, privacy details, and legal basis.
  • Contract (source of truth): packages/cli/src/telemetry/flow.json.

Related Skills

Detailed References:

Using the walkerOS Logger

Overview

The logger is walkerOS's standard logging system, available in all sources and destinations via env.logger or logger parameter. It provides scoped, level-aware logging that replaces console.log.

Core principle: Don't log what the collector already logs, and never log a failure yourself. A step reports a failure by throwing a StepError; the collector writes the one ERROR line per cause. Log only meaningful operations like external API calls and transformations, at debug.

Logger Access

In Sources

export const sourceFetch: Source.Init<Types> = async (context) => {
  const { env } = context; // env.logger is available here
  // Logger is scoped automatically by collector: [type:sourceId]
  env.logger.info('Server listening on port 3000');
  // ... build and return the source instance
};

In Destinations

export const destinationDataManager: DestinationInterface = {
  async init({ config, env, logger }) {
    // logger parameter is scoped automatically: [datamanager]
    logger.debug('Auth client created');
  },

  async push(event, { config, data, env, logger }) {
    // logger parameter is scoped: [datamanager]
    logger.debug('API response', { status: 200 });
  },
};

Note: You don't need to create or configure the logger; it's provided automatically with proper scoping.

Logger Methods

interface Logger.Instance {
  error(message: string | Error, context?: unknown | Error): void;
  warn(message: string | Error, context?: unknown | Error): void;
  info(message: string | Error, context?: unknown | Error): void;
  debug(message: string | Error, context?: unknown | Error): void;
  throw(message: string | Error, context?: unknown): never;
  json(data: unknown): void;
  scope(name: string): Logger.Instance;
}

Log Levels

  • ERROR (0): the user's intent is not met (an event was not delivered, a setting has no effect)
  • WARN (1): walkerOS acted and the intent is still met
  • INFO (2): lifecycle and self-healing (server startup)
  • DEBUG (3): the per-event trace and deliberate decisions (API calls, transformations, a skipped event)

Default: ERROR only (must configure to see WARN/INFO/DEBUG)

Context Parameter

All methods accept optional structured context:

logger.debug('Sending to API', {
  endpoint: '/events',
  method: 'POST',
  eventCount: 5,
});
// Output: DEBUG [datamanager] Sending to API { endpoint: '/events', method: 'POST', eventCount: 5 }

When to Log (and When NOT to)

❌ DON'T Log These (Collector Handles)

  • Init status: "Initializing...", "Init started", "Init complete"
  • Push status: "Processing event...", "Event received"
  • Generic status: "Settings validated", "Config loaded"
  • Duplicate scoping: Don't add source/dest name (already in scope)

Why: Collector can log these automatically since it calls init/push and has scoped logger.

✅ DO Log These (Meaningful Operations)

  • External API calls: Before/after with request/response details
  • Auth operations: Token refresh, client creation/failures
  • Transformations: Complex mappings or data processing
  • Notes that are not failures: a deliberate decision, an advisory about configuration (warn: walkerOS acted, the intent is still met)
  • Never failures: a missing setting, a failed request or an unsendable event is a thrown StepError, not a log line

Usage Patterns

Pattern 1: Validation Errors (Throw a StepError)

import { StepError } from '@walkeros/core';

// ✅ GOOD - A missing setting is a failure: throw, never return or log
async init({ config }) {
  const { apiKey, projectId } = config.settings || {};

  if (!apiKey)
    throw new StepError('Config settings apiKey missing', { code: 'config' });

  if (!projectId)
    throw new StepError('Config settings projectId missing', {
      code: 'config',
    });
}

Why a thrown StepError:

  • The collector turns the throw into a failed push (or a failed start), counts it, and writes one ERROR line per cause and minute with the code
  • A step that also logged the failure would write a second line for the same cause
  • code and retryable tell a server source whether to ask the producer to resend (config, missing-input and a 4xx are not retryable)
  • logger.throw only throws. It writes no log line, so it is not a way to report a failure

Never return early from init or push instead of throwing: a step that returns normally counts as started, and a push that returns normally counts as delivered. return false from init fails the push with the code not-ready, which does not name the missing setting.

Pattern 2: External API Calls

import { StepError } from '@walkeros/core';

// ✅ GOOD - Trace the call at debug, throw on a failure
async push(event, { config, logger }) {
  const endpoint = 'https://api.vendor.com/events';

  logger.debug('Calling API', { endpoint, method: 'POST', eventId: event.id });

  const response = await fetch(endpoint, {
    method: 'POST',
    body: JSON.stringify(event),
  });

  logger.debug('API response', { status: response.status, ok: response.ok });

  if (!response.ok)
    // The status sets retryable (429 and 5xx retry, other 4xx do not). The
    // response body goes on `body`, never into the message, and never into a
    // log line.
    throw new StepError(`HTTP ${response.status}`, {
      code: 'transport',
      status: response.status,
      body: await response.text(),
    });
}

Prefer sendWeb / sendServer: they already throw a StepError with the status and body.

Pattern 3: Auth Operations

import { StepError } from '@walkeros/core';

// ✅ GOOD - Trace the client creation, throw on failure
async init({ config, logger }) {
  try {
    const authClient = await createAuthClient(config.settings);
    logger.debug('Auth client created');
    return {
      env: { authClient },
    };
  } catch (error) {
    throw new StepError('Authentication failed', {
      code: 'config',
      cause: error,
    });
  }
}

The SDK's own message stays on cause, so a vendor error that carries personal data never reaches a log line.

Pattern 4: Server Startup (Sources)

// ✅ GOOD - Log server listening (high-level info); routeLines lists one line
// per method and path the source serves
if (settings.port !== undefined) {
  server = app.listen(settings.port, () => {
    env.logger.info(
      `Express source listening on port ${settings.port}\n` + routeLines,
    );
  });
}

Anti-Patterns (What NOT to Do)

❌ BAD: Verbose Init Logging

async init({ logger }) {
  logger.debug('Data Manager init started'); // Redundant
  logger.info('Data Manager initializing...'); // Redundant
  logger.debug('Settings validated'); // Redundant

  const authClient = await createAuthClient();
  logger.debug('Auth client created'); // OK

  logger.info('Data Manager ready'); // Redundant
}

Problem: Collector knows when init is called. Only log meaningful operations (auth client creation).

❌ BAD: Redundant Push Logging

async push(event, { logger }) {
  logger.debug('Processing event', {
    // Redundant
    name: event.name,
    id: event.id,
  });

  // Do work...

  logger.info('Event processed'); // Redundant
}

Problem: Collector knows when push is called and can log automatically.

❌ BAD: Using console.log

// ❌ NEVER use console.log in sources/destinations
console.log('Processing event:', event.name);

// ✅ Use logger instead
logger.debug('API call', { endpoint });

Migration Checklist

When updating a source/destination to use the logger:

  • Remove all console.log, console.warn, console.error statements
  • Remove verbose init/push status logging (let collector handle)
  • Throw a StepError with a code for every failure (apiKey missing, failed request, unsendable event); never log a failure and return
  • Add logger.debug before external API calls (with endpoint, method)
  • Add logger.debug after external API calls (with response status)
  • Add logger.debug for auth operations (client creation, token refresh)
  • Use structured context (objects) instead of string concatenation
  • Verify tests still pass with logger mocked

Testing with Logger

Use createMockLogger from @walkeros/core to check debug traces. To prove that a failure is reported, start a real collector (startFlow), mock only the network or the vendor global, and read the push result; a mocked collector cannot see loss:

import { startFlow } from '@walkeros/collector';

test('a missing apiKey fails the push', async () => {
  const { elb } = await startFlow({
    destinations: { vendor: { code: destination, config: { settings: {} } } },
  });

  const result = await elb({ name: 'page view' });

  expect(result.ok).toBe(false);
  expect(result.failed?.vendor.reason?.code).toBe('config');
});

Log Level Configuration

Default log level is ERROR. To see INFO/DEBUG logs:

import { startFlow } from '@walkeros/collector';

const { elb } = await startFlow({
  logger: {
    level: 'DEBUG', // Show all logs
  },
  destinations: {
    /* ... */
  },
});

Levels:

  • 'ERROR': Only errors (default)
  • 'WARN': Errors + warnings
  • 'INFO': Errors + warnings + info
  • 'DEBUG': Everything

The levels follow the contract under Log Levels. A skipped event writes a DEBUG line; a consent wait writes no line, and the push result lists the destination under queued with the reason consent. Neither is a failure.

Format: on a server flow, "collector": { "logger": { "format": "json" } } prints one JSON object per line (severity, message, time, scope, context), which platforms such as Cloud Run read as structured logs. A web flow rejects format. The flow's own handler and the runtime's handler both see every line, so forwarding to a monitoring service needs no parsing.

Related Skills

Key Files:

Choosing a level on public endpoints

A server source endpoint is exposed to the open internet. Level choice decides whether ambient scanner noise turns into pages.

SituationLevelAlso
Unparseable body, oversized payloaddebugcount on collector.status.sources.<id>.rejected
Unknown pathno lineanswered 404
Event the collector rejects as invaliderror with the reason, one line per source and minute, no stackcounted as rejected; answered 400 only when the source waits for delivery
Settled ok: false arriving after the HTTP ackdebugthe collector's failure line reports the cause at error; the ack already went out
Handler fault, rejected push promiseerroranswered 500

Rules that follow from the table:

  • Never error for anything a stranger can trigger, apart from the collector's invalid-event line, which the source does not repeat. On Cloud Run an error-level stack trace on stderr opens an Error Reporting issue, so scanner traffic becomes an alert; the invalid-event line carries no stack.
  • Do not duplicate the collector's error logs. The collector already logs thrown pipeline failures at error. A source that logs them again doubles every real fault.
  • A rejected push and a settled ok: false are different things. If a helper collapses both into one return shape, log the thrown case inside that helper at error, because the call site can no longer tell them apart.

Reference pattern: packages/server/sources/express/src/index.ts, the error-boundary middleware for body-parser failures and the 400/500 split in the synchronous POST branch.

Best Practice Examples:

Using Step Examples

Overview

Step examples are structured { in, out } pairs (with optional mapping for destinations) that define the expected input/output behavior of each step in a flow. They serve as:

  • Test fixtures for automated it.each testing
  • Simulation input for walkeros push --simulate (pass an example's in as --event; the CLI does not compare the result against out)
  • MCP context for AI-assisted development
  • Documentation showing real-world usage

Every source, transformer, and destination can ship step examples alongside its code.

The Three Type Zones

Each step in a flow sits at a boundary between arbitrary external formats and the walkerOS event model:

           Source              Transformer           Destination
  ┌─────────────────┐   ┌─────────────────┐   ┌─────────────────┐
  │  in: arbitrary   │   │  in: walkerOS   │   │  in: walkerOS   │
  │  (HTTP req, DOM) │   │      Event      │   │      Event      │
  │        │         │   │        │         │   │        │         │
  │        ▼         │   │        ▼         │   │        ▼         │
  │  out: walkerOS   │   │  out: walkerOS  │   │  out: arbitrary  │
  │      Event       │   │   Event | false │   │  (gtag, API)     │
  └─────────────────┘   └─────────────────┘   └─────────────────┘

  arbitrary ──────────── walkerOS.Event ──────────── arbitrary
  • Source converts platform-specific input into a walkerOS event
  • Transformer receives and returns walkerOS events (or false to filter)
  • Destination converts a walkerOS event into vendor-specific output

The in/out/trigger Format

Every step example is an object with in (input), out (array of observable effects), and optional trigger (how to invoke):

type StepEffect = readonly [callable: string, ...args: unknown[]];
type StepOut = readonly StepEffect[];

interface StepExample {
  in?: unknown; // Platform-specific input
  out?: StepOut; // Array of effect tuples - see shape rules below
  trigger?: { type?: string; options?: unknown }; // How to invoke
  mapping?: unknown; // Destination mapping rule
  command?: 'config' | 'consent' | 'user' | 'run'; // Route to walker command
  // also: title, description, public (Metadata), outcome, respond (failures)
}

out is always an array of effect tuples, even for a single effect. Each tuple is [callable, ...args]. The first element names the call, the remaining elements are its arguments. runExamples compares the tuples exactly with the calls it records, so the callable is the recorded path.

Callable conventions

Component typeCallableNotes
Destination with a mock env'window.gtag', 'window.fbq', 'window.dataLayer.push', 'window.ttq.track'The package's simulation path without call: ('call:window.gtag' records as 'window.gtag'). Dotted paths render literally.
Destination (HTTP)'fetch', 'sendServer'The request call the destination makes.
Source'elb'The walker public push API.
Transformer'return'Reserved keyword. Renders as return <value> (no parens).

'return' is reserved. Don't use it for anything else.

Empty out: [] means the step produced no observable effect (filtered input, transformer passthrough, validator rejection). Reserved only for cases where the destination/source deliberately emits nothing.

Source Step Example - Server (Express)

in is an HTTP request shape, out is a tuple of the elb() call the source makes:

export const checkoutPost: Flow.StepExample = {
  trigger: { type: 'POST' },
  in: {
    method: 'POST',
    path: '/collect',
    body: { name: 'order complete', data: { id: 'ORD-123', value: 149.97 } },
  },
  out: [
    ['elb', { name: 'order complete', data: { id: 'ORD-123', value: 149.97 } }],
  ],
};

Source Step Example - Commands (CMP, session)

A source that issues walker commands records them in its own call shape: the first argument verbatim, callable 'elb'. Commands come first, in call order, then the events. A CMP source:

out: [['elb', 'walker consent', { functional: true, marketing: true }]],

The session source (env.command('user' | 'session', ...), then its event):

out: [
  ['elb', 'user', { session: 's3ss10n-id', device: 'd3v1c3-id' }],
  ['elb', 'session', { id: 's3ss10n-id', isStart: true /* ... */ }],
  ['elb', { name: 'session start', data: { id: 's3ss10n-id' /* ... */ } }],
],

A source simulation (walkeros push --simulate source.X, MCP flow_simulate) returns exactly these calls: call elb("walker consent",{...}). The collector's own commands (starting consent from --consent, globals, custom, shutdown) and the wiring commands on, hook, destination are never part of a source's out. A consent-gated source needs --consent (MCP state.consent) to emit anything.

Source Step Example - Browser

in is an HTML string, out is a tuple of the elb() call:

export const clickEvent: Flow.StepExample = {
  trigger: { type: 'click', options: 'button' },
  in: '<button data-elb="cta" data-elb-cta="label:Sign Up" data-elbaction="click:click">Sign Up</button>',
  out: [
    [
      'elb',
      {
        name: 'cta click',
        data: { label: 'Sign Up' },
        trigger: 'click',
        entity: 'cta',
        action: 'click',
        // plus context, globals, nested and source: an out lists every field
        // the step emits
      },
    ],
  ],
};

Transformer Step Example

in is a walkerOS event, out is a ['return', value] tuple with what the transformer returns ({ event }, or false to drop). An empty array means the transformer passed the event through unchanged:

export const step = {
  orderPasses: {
    in: { name: 'order complete', data: { id: 'ORD-123' } },
    out: [
      [
        'return',
        { event: { name: 'order complete', data: { id: 'ORD-123' } } },
      ],
    ],
  },
  debugFiltered: {
    in: { name: 'debug test', data: { message: 'noise' } },
    out: [['return', false]], // Transformer rejects
  },
  passthrough: {
    in: { name: 'page view' },
    out: [], // No modification
  },
};

These outs are what the transformer itself returns, the shape a test that calls push directly compares (see Transformer Tests below). runExamples instead records what reaches the next step, the event as the collector completed it. A dropped event is out: [] there, and an event passed through unchanged is ['return', { event }] with every collector field.

Destination Step Example

in is a walkerOS event, mapping is the mapping rule that transforms it, out is an array of call tuples - one per observable effect the destination produces. Multi-call events (e.g., GA4 + Ads + GTM for a single walker event) flatten into a single array in execution order:

import type { Flow } from '@walkeros/core';
import { getEvent } from '@walkeros/core';

// getEvent pins the timestamp (1700000000000) and derives the id from the event
// name; pass `id` when the expected output must name a readable one.
export const purchase: Flow.StepExample = {
  in: getEvent('order complete', { id: '1700000000-gr0up-1' }),
  mapping: {
    name: 'Purchase',
    data: { map: { value: 'data.value', currency: { value: 'EUR' } } },
  },
  out: [
    [
      'window.fbq',
      'track',
      'Purchase',
      { value: 462, currency: 'EUR' },
      { eventID: '1700000000-gr0up-1' },
    ],
  ],
};

Multi-tool example (one walker event produces GA4, Ads, GTM in order):

out: [
  ['window.gtag', 'event', 'purchase', { transaction_id: 'o1', value: 462 }],
  ['window.gtag', 'event', 'conversion', { send_to: 'AW-123', value: 462 }],
  ['window.dataLayer.push', { event: 'purchase', ecommerce: { ... } }],
],

Each export is a self-contained Flow.StepExample - no intermediate variables, no all aggregation. The mapping field ties the mapping rule to the example so tests can register it dynamically: { [event.entity]: { [event.action]: example.mapping } }.

Consumers iterate all examples via Object.entries(examples.step) - export * as step exposes every named export directly.

Command Step Example

Some destinations need to respond to walker commands (walker consent, walker user, walker run), not events. Set the command field to route in through elb('walker <command>', in) instead of pushing it as an event:

export const consentGranted: Flow.StepExample = {
  command: 'consent',
  in: { marketing: true, analytics: true },
  out: [
    [
      'window.gtag',
      'consent',
      'update',
      {
        ad_storage: 'granted',
        ad_user_data: 'granted',
        ad_personalization: 'granted',
        analytics_storage: 'granted',
      },
    ],
  ],
};

Supported commands: config, consent, user, run.

  • When command is absent (default), in is pushed as a regular event via elb(in).

  • When command is set, mapping is not applied - commands don't flow through event mapping.

  • The out holds the calls the destination makes for the command, as tuples like any other out. The gtag example above runs with config.consent set; without it a ['window.gtag', 'consent', 'default', {...}] call comes first, as in the gtag package's consentModeV2 example.

  • Simulate a command example with --command <name> (MCP: command): the destination simulation runs collector.command(name, in) instead of a push and records the calls it makes. Add --consent when the destination needs starting consent; it applies first, then the command:

    walkeros push flow.json -e '{"functional":true,"marketing":true}' \
      --simulate destination.ga4 --command consent
    

Failure Step Example (outcome, respond)

A step's failure modes are examples too. A failure fixture is public: false and declares what the push result reports through outcome; respond states what the vendor's boundary answers when the failure comes from a response:

export const rejectedByVendor: Flow.StepExample = {
  public: false,
  description: 'The vendor answers 200 and reports the event as failed.',
  in: getEvent('page view'),
  respond: { status: 200, body: { events: [{ status: 'failed' }] } },
  outcome: { status: 'failed', code: 'transport', retryable: false },
};

export const missingInput: Flow.StepExample = {
  public: false,
  description:
    'An event with nothing to send fails instead of counting as delivered.',
  in: getEvent('page view'),
  out: [],
  outcome: { status: 'failed', code: 'missing-input' },
};
  • outcome is { status: 'failed' | 'queued', code, retryable? }. Absent means the step must not fail. The codes are the collector's failure codes (config, missing-input, transport, timeout, vendor-not-loaded, ...).
  • Every example with an outcome needs a description: the package's docs page lists it under "When it fails" (title, description, status and code, never its in), public or not, and the docs build fails on one without a description.
  • respond is { status?, body?, error?: 'network' | 'timeout' }.
  • A step with no vendor call for an example has out: []. A silent rule is an example with mapping: { silent: true } and out: [], and a deliberate skip (ignore: true) is out: [] with no outcome.
  • runExamples(kind, step, examples, options?) and conformance(kind, step, options) from @walkeros/collector/dev run the examples on a real collector and compare the push result with outcome. Conformance profiles: destinations http, sdk-env, web-sdk, vendor-global, socket; transformer and source for the other kinds. The browser profiles (web-sdk, vendor-global) need page from @walkeros/web-core/dev in the options, and a web source may pass it; runExamples takes the same option.

Public examples, claims and literal outs

A destination has exactly one public example per standard event it maps: the most complete, fully implemented case. Its in starts from getEvent(name) (page view, product view, product add, order complete). Edge cases, other tools and failures are public: false; they still run as tests.

  • claims. runExamples('destination', step, examples, { claims }) takes the standard events the destination maps. It fails when a claimed event has no public example, when that example does not start from getEvent(name), or when a public event example is not claimed. Init examples (config as in) are not events and are ignored by claims.
  • Mappings run through the real destination config. The runner looks the example up by event name and applies example.mapping ?? config.mapping. A public example whose setup needs other settings runs in its own runExamples call with those settings.
  • Every example runs as published. runExamples and conformance run every function in an example (in in, mapping, trigger.options, respond.body) the way walkerOS.json publishes it: as source text, revived outside the module it was written in. There is no option to turn this off. A condition that calls an imported helper passes as written and fails here (example <name>: published code fails: ... is not defined), which is how it would fail in a user's flow: a published function may use only its parameters and standard globals. The text revived is the one the test run compiled, not the one in the built dist/walkerOS.json, so a change only the package build makes is not covered.
  • An out is a literal. Write it from the vendor's documented request (or a captured one), cite the source in the example, and only then run the code. Never compute an out with production code (a helper from src/, the destination's own encoder): the example would then agree with the code by construction. A mismatch between research and code is a finding; fix the product or the example and say why.
  • No personal data or credentials in an example: placeholders (G-XXXXXXXXXX), example.com addresses and RFC 5737 IPs (192.0.2.1).
import type { StandardEventName } from '@walkeros/core';
import { runExamples } from '@walkeros/collector/dev';

const CLAIMS: StandardEventName[] = [
  'page view',
  'product view',
  'product add',
  'order complete',
];

it('maps the claimed standard events', async () => {
  await runExamples('destination', destinationX, events, {
    settings,
    env,
    claims: CLAIMS,
  });
});

Minimal example (destinations)

A destination package ships examples.minimal, a Flow.MinimalExample in src/examples/minimal.ts (the path is MINIMAL_EXAMPLE_FILE on @walkeros/core/dev), re-exported from src/examples/index.ts. It holds only the settings the settings schema requires, plus any that init refuses to start without. composeQuickstart (on @walkeros/core/dev) turns it into the install line, the integrated code and the flow.json that the package's docs page renders under ## Installation, and the conformance kit starts the destination with exactly those settings and requires one public success example to be delivered.

// src/examples/minimal.ts
import type { Flow } from '@walkeros/core';

export const minimal: Flow.MinimalExample = {
  settings: { accessToken: 's3cr3t', pixelId: '1234567890' },
  secrets: { accessToken: 'META_ACCESS_TOKEN' },
};

// src/examples/index.ts
export * as env from './env';
export * as step from './step';
export { minimal } from './minimal';
  • settings: required keys only, placeholder values. An optional key stays in the examples that need it; the kit rejects a top-level key the schema does not require, unless init refuses to start without it.
  • secrets (server only): a credential setting mapped to its secret name, <VENDOR>_<SETTING> in A-Z, 0-9 and _. Generated code reads process.env.<NAME> (integrated) and $secret.<NAME> (flow) and never shows the placeholder. secrets on a web destination throws.
  • config: only { loadScript: true }, for a web destination that loads its vendor script only when asked to. The kit fails a script-loaded web destination whose minimal example lacks it.
  • Several exports: one minimal example per export, in the export's folder (src/firehose/examples/minimal.ts) and its exportExamples entry; the test passes exportName and exportExamples, the page <Quickstart data={data} export="destinationFirehose" />.

Every destination conformance test passes one of two options:

await conformance('destination', destinationMeta, {
  profile: 'http',
  examples: examples.step,
  schema: schemas.settings,
  settings: examples.minimal.settings,
  minimal: examples.minimal,
  root: path.join(__dirname, '..', '..'),
});

// A package that cannot have one passes the reason instead of `minimal`, and
// its docs page keeps a hand-written setup:
//   withoutMinimal: 'the install line cannot express the driver choice',

A test with neither fails, and so does one that disagrees with the file: withoutMinimal while src/examples/minimal.ts exists, minimal without it, both, or a blank reason. The doc-lint (npm run validate:docs) fails a page whose package has the file but whose ## Installation shows a hand-written npm install, startFlow( or bundled tab, or no <Quickstart>.

Writing Examples

Best Practices

  1. Use camelCase names -- descriptive of the scenario (checkoutPost, debugFiltered, pageViewBasic)
  2. Cover happy path and edge cases -- include at least one positive and one negative case for transformers
  3. Use realistic data -- real-looking event names, IDs, and values
  4. Keep examples minimal -- in in, only include fields relevant to the step's logic; omit empty/default-value fields (context: {}, nested: [], user: {}, etc.). An out is compared exactly, so it lists every field the step emits, empty ones included
  5. Export from examples/step.ts -- follow the existing dev.ts structure
  6. Title and describe public examples -- add title (2-5 words) and description (one short sentence, 10-25 words) for every example that should appear in docs. Mark test-only fixtures with public: false.

File Structure

// src/examples/step.ts - only Flow.StepExample exports, nothing else
import type { Flow } from '@walkeros/core';
import { getEvent } from '@walkeros/core';

export const purchase: Flow.StepExample = {
  in: getEvent('order complete', { id: '1700000000-gr0up-1' }),
  mapping: { name: 'Purchase', data: { map: { value: 'data.value' } } },
  out: [
    [
      'window.fbq',
      'track',
      'Purchase',
      { value: 462 },
      { eventID: '1700000000-gr0up-1' },
    ],
  ],
};

// src/examples/index.ts
export * as env from './env';
export * as step from './step';

The file exports only Flow.StepExample objects. No intermediate variables, no all, no config. Consumers iterate via Object.entries(examples.step).

Rule: examples live only on the ./dev subpath

Examples (and schemas) must be exported only from src/dev.ts (the ./dev package subpath), never from a production entry (src/index.ts / the package main). Production bundles tree-shake on the import path: if examples is reachable from the main entry, the bundler inlines every example (and its getEvent fixtures) into the shipped JS, bloating consumer bundles with test-only data. Importing them from ./dev keeps them available for simulation and testing while leaving production output clean.

A no-restricted-syntax ESLint rule enforces this, so a export ... examples from a production entry fails lint.

Multi-export packages: exportExamples and exportSchemas

A package whose package.json walkerOS.exports lists more than one export adds a map of EVERY export, the default included, to src/dev.ts:

export const exportExamples = {
  destinationBigQuery: bigqueryExamples,
  destinationPubSub: pubsubExamples,
};

The step's export name comes from resolveExportName (exported from @walkeros/cli): the step's import, else bundle.packages[pkg].imports[0], else the default. The bundler runs only the step's import or the default export, so set import on the step for examples and code to agree. When the module has the map, map[exportName] is used and a missing name resolves to nothing (never a fallback to examples); without a map, examples is used. selectDevExamples(devModule, exportName) implements this. The CLI simulate, MCP flow_examples and package_get, core cdn.ts and the tsup walkerOS.json (exportExamples key) all follow it. Existing named exports (pubsubExamples, sqsExamples, ...) stay for back compat.

The same package also exports exportSchemas, every export to that export's schemas in the shape of a single-export schemas (settings required, mapping, setup where they exist). Reuse the very objects behind schemas.settings for the default entry, so a toBe test pins it:

export const exportSchemas = {
  destinationBigQuery: {
    settings: bigquerySchemas.settings,
    mapping: bigquerySchemas.mapping,
  },
  destinationPubSub: {
    settings: pubsubSchemas.settings,
    mapping: pubsubSchemas.mapping,
    setup: pubsubSchemas.setup,
  },
};

walkeros validate and MCP package_get pick the entry for the step's export with the same rule. The tsup build (buildDev) fails when either map does not list exactly the walkerOS.exports names, when an exportSchemas entry has no settings, and when any source, transformer, destination or store package publishes no schemas.settings (a package without settings publishes z.object({})). Prefixed keys in schemas (pubsubSettings, ...) stay.

Mock env and simulation paths

src/examples/env.ts exports push (the mock env simulate injects) and simulation (the call paths to record). Simulate refuses a package destination whose export has no push mock ("No mock env for ..."), so it never calls the real vendor.

  • A simulation path names the call that CARRIES the vendor request (sendServer, fetch, call:PubSub.topic.publishMessage, call:JSONWriter.appendRows, call:AWS.SNSClient.send, call:fs.createWriteStream.write), never only a constructor.

  • init is not the request. Do not record an init call (it carries the API key and sends nothing); a path may pass THROUGH init's result (call:Mixpanel.init.track). Amplitude and Mixpanel record their tracking calls, not init.

  • Grammar (parseCallPath, shared by the recorder): an optional call: prefix, then .-separated segments; an empty segment makes the path unresolvable. The one recorder is observeEnv(env, paths) in @walkeros/core (collector wrapEnv and web-core getEnv delegate to it). It returns a Proxy view; nothing on the env objects or prototypes is mutated.

  • Mid-path resolution: on an object, a segment is looked up as a property. After a function, a segment is the function's own property or a parent class's static; anything else (including call, bind, name, length) navigates the call result (or the constructed instance, for new). Frozen objects and frozen call results are recorded through a view; only a static on a frozen function stays unresolved. A mid-path Promise (async factory) is followed with .then. Only the leaf call is recorded: fn is the path without call:, args the leaf call's arguments. instanceof, statics and #private fields keep working, and a chain returning this records once.

  • Pin every list in the package tests with the async expectSimulationResolves(examples.env) from @walkeros/core/dev: it walks each path through the push mock (constructing, calling, awaiting) and fails on a missing segment or a non-function leaf.

    import { expectSimulationResolves } from '@walkeros/core/dev';
    it('declares simulation paths that resolve', () =>
      expectSimulationResolves(examples.env));
    
  • Not recordable: anonymous values such as SQLite prepared-statement rows.

  • Sources: a source package that declares simulation runs on its mock client in a source simulation, and those calls are recorded. SQS records call:AWS.SQSClient.send (one receive per 100 ms while running); Pub/Sub pull records call:PubSub.subscription (messages arrive as events). Their module-mock tests import moduleMockEnv, the same env without the client.

Metadata: title, description, public

Every Flow.StepExample accepts three optional metadata fields that control how it surfaces in docs and MCP output:

  • title?: string - overrides the default camelCase-to-spaced heading in website docs. Keep it short (2-5 words), human-readable.
  • description?: string - one short sentence (10-25 words) rendered above each example in docs and surfaced in MCP flow_examples output. Explains what the example demonstrates, not how.
  • public?: boolean - defaults to true. When false, the example is excluded from the Examples section of the website docs and from default MCP flow_examples output. It still runs in tests and remains callable via CLI/MCP flow_simulate. An example with outcome is listed under "When it fails" on the package page either way, so it needs a description.
export const purchase: Flow.StepExample = {
  title: 'Purchase',
  description:
    'A completed order mapped to the Meta Pixel Purchase standard event.',
  in: getEvent('order complete', { id: '1700000000-gr0up-1' }),
  mapping: {
    name: 'Purchase',
    data: { map: { value: 'data.value', currency: { value: 'EUR' } } },
  },
  out: [
    [
      'window.fbq',
      'track',
      'Purchase',
      { value: 462, currency: 'EUR' },
      { eventID: '1700000000-gr0up-1' },
    ],
  ],
};

// Internal fixture - runs in tests, hidden from docs and default MCP output.
export const debugFiltered: Flow.StepExample = {
  public: false,
  in: { name: 'debug test', data: { message: 'noise' } },
  out: [['return', false]],
};

When to mark public: false

Mark public: false when:

  • The example is a transformer rejection (out: [['return', false]]) that only proves filtering works.
  • It's an error-path or malformed-input case.
  • It's a redundant variant kept only for regression coverage.
  • It's an internal fixture used by cross-package tests.

Keep public (omit the field) when:

  • It's the happy path.
  • It teaches a distinct mapping or usage pattern.
  • It's the only example for some entity/action the package handles.

Source Trigger Metadata

Source step examples can include a trigger field for simulation:

{
  in: '<button data-elb="cta" data-elb-cta="label:Sign Up" data-elbaction="click">Sign Up</button>',
  trigger: { type: 'click', options: 'button' }, // a click needs its selector
  out: [['elb', { name: 'cta click', data: { label: 'Sign Up' } /* ... */ }]],
}

When simulating via CLI or MCP, the step example maps to SourceInput:

  • in -> content (the actual source input)
  • trigger -> trigger (which mechanism to fire)

Destination and transformer examples don't use trigger.

Simulating with Step Examples

Use --simulate <kind>.<name> to target a specific step, then provide the input with --event. For a source step, that is SourceInput ({ content, trigger? }):

# Simulate a source step with trigger metadata (a click needs its selector)
walkeros push flow.json --simulate source.browser --event '{"content":"<button data-elb=\"cta\" data-elbaction=\"click\">Sign up</button>","trigger":{"type":"click","options":"button"}}'

The CLI does not compare the result against out. For each simulated step it prints the matched mapping key and every recorded call (--json: under simulations), and exits 1 only when the push fails with an error; a mapping that produces the wrong output still exits 0. Assert on out in your own tests (see Testing with Examples below). Add --ingest to give a transformer, collector or destination step a request context, --consent for the collector consent it starts from, --command for a command example, and --page-url for the page of a web source (default: the trigger's options.url, else http://localhost). See using-cli.

The MCP flow_examples tool returns trigger metadata alongside in/out, and mapping for destination examples, giving full visibility into how input events are transformed to vendor-specific output.

Validating Examples

Cross-step example validation is included automatically when validating a flow. It does not run any step:

# Validate flow config including step example compatibility
walkeros validate flow.json

For steps connected through next or a destination's before chain, flow validation checks that:

  • At least one upstream out is structurally compatible with a downstream in: same type; two objects share at least half of the keys of the object with fewer keys; any two arrays are compatible regardless of content. No compatible pair is an error. This is a shape check, not an equality check.
  • A connection is only checked when both steps declare examples; a step without an examples key produces no warning.
  • When both steps declare examples but one side has no usable value, the connection produces a warning. An upstream out counts when it passes on at least one event (next bullet); a downstream in counts unless its example sets command.
  • An out passes on the events of its effects: elb gives args[0] when it is an object; return gives a.event for { event }, a for a bare event, each item for an array (fan-out); false, message.ack, message.nack, response, respond and vendor calls give nothing.
  • Contracts bind only where a @walkeros/transformer-validate step links them, as at runtime. Each validate step's own examples run through that step's resolved settings.contract, mode, format and output, and the verdict must agree with its out (valid: passed on; invalid: ["return", false] in strict, passed on with the isValid flag false in pass). A disagreement is CONTRACT_VIOLATION (warning, error with --strict). No other example is checked against the config-level contract block, and a flow without a validate step gets no contract checks.

Testing with Examples

The primary use of step examples is automated testing with it.each.

Source Tests (createTrigger)

Sources export createTrigger from their examples. It follows the unified Trigger.CreateFn interface: receives Collector.InitConfig, lazily starts the flow, and returns a trigger function that simulates real-world invocations.

Each package implements createTrigger differently:

  • Browser: Injects HTML into DOM, starts flow, dispatches native events
  • Express: Boots real HTTP server, sends fetch() requests

runExamples feeds each example's in through the package's createTrigger and compares the elb calls the source makes with out:

import { runExamples } from '@walkeros/collector/dev';
import { sourceExpress } from '@walkeros/server-source-express';
import { examples } from '@walkeros/server-source-express/dev';

it('runs the step examples', async () => {
  await runExamples('source', sourceExpress, examples.step, {
    createTrigger: examples.createTrigger,
    settings: { port: 0 },
  });
});

The run passes, but today the option is typed for unknown content, so tsc rejects a package's typed createTrigger here (TS2769). The express package's own src/__tests__/stepExamples.test.ts shows a typed loop that narrows in with a type guard instead.

Browser source note: The browser source's elbLayer processes events via a detached promise chain (fire-and-forget). A test that collects events with its own spy destination after an interactive trigger (click, submit) polls until they arrive:

while (!events.length) await Promise.resolve();

CMP Source Tests (Consent Assertion)

CMP sources push walker consent commands, not regular events. Assert on collector consent state instead of using a spy destination:

const instance = await examples.createTrigger({
  consent: {},
  sources: {
    usercentrics: {
      code: sourceUsercentrics,
      config: { settings: {} },
    },
  },
});

await instance.trigger(
  example.trigger?.type,
  example.trigger?.options,
)(example.in);

// Yield for the detached elb('walker consent') chain
await new Promise((resolve) => setTimeout(resolve, 20));

// out is [['elb', 'walker consent', consent]]
expect(instance.flow?.collector.consent).toEqual(example.out?.[0]?.[2] ?? {});

Server Function Handler Tests (collector.sources)

Server function handlers (fetch, AWS Lambda, GCP CloudFunction) don't own servers. Their createTrigger accesses the source instance from collector.sources after startFlow and calls source.push() with platform-native types:

function findSource(collector) {
  for (const source of Object.values(collector.sources || {})) {
    if (source.type === 'fetch') return source;
  }
}

// In trigger:
const source = findSource(flow.collector);
const response = await source.push(request);

Destination Functional Tests

For destinations, runExamples('destination', ...) (shown under Public examples above) is the test: it starts a real collector, applies each example's mapping through the destination config, records the calls named by examples.env.simulation, and compares them with out. Pass the package's env ({ push, simulation }), the settings, and for a web destination page from @walkeros/web-core/dev (run in a jsdom test environment).

Transformer Tests

A test that calls the transformer's push directly wraps its return in the out shape (runExamples('transformer', ...) records what reaches the next step instead, see Transformer Step Example):

import type { WalkerOS } from '@walkeros/core';
import { createMockContext } from '@walkeros/core';
import { transformerMyTransformer } from '../transformer';
import { examples } from '../dev';

describe('transformer', () => {
  it.each(Object.entries(examples.step))('%s', async (name, example) => {
    const instance = await transformerMyTransformer(
      createMockContext({ id: 'myTransformer' }),
    );
    const result = await instance.push(
      example.in as WalkerOS.DeepPartialEvent,
      createMockContext({ id: 'myTransformer' }),
    );
    const actual =
      result === false
        ? [['return', false]]
        : result
          ? [['return', result]]
          : [];
    expect(actual).toEqual(example.out);
  });
});

Checklist

When adding step examples to a package or flow:

  • Create src/examples/step.ts with Flow.StepExample exports
  • Export from src/examples/index.ts
  • Use camelCase, descriptive example names
  • Include at least one happy-path example
  • For transformers, include a false (filtered) case
  • Add it.each test using step examples
  • Exactly one public example per standard event the destination maps, from getEvent(name), with claims in a runExamples call; every out a literal written from the vendor's documentation
  • Failure examples (public: false, outcome, respond, a description) for every event the step cannot send, a silent example for a destination, and a conformance test (conformance from @walkeros/collector/dev)
  • Destination: src/examples/minimal.ts with only the settings the schema requires or init refuses to start without (credentials as secrets on a server destination, config.loadScript for a script-loaded web destination), exported as examples.minimal and passed to conformance as minimal; or withoutMinimal: '<reason>'
  • Verify examples compile: npm run build
  • Run tests: npm run verify:touched -- <pkg> (L1)
  • Add title and description to every public example.
  • Mark internal/test-only examples with public: false.
  • examples/env.ts exports push and a simulation list naming the request call, pinned with expectSimulationResolves(examples.env).
  • Multi-export package: exportExamples and exportSchemas in dev.ts list every export (the build fails otherwise).
  • In a flow file, add examples sparingly: they are mainly internal and for demos (for example a client flow's source with public: false examples showing its default behaviour).

Related Skills

Using store-level cache

Overview

Flow.Store.cache wraps any store with a read-through, write-through cache tier. Reads check the cache first, fall through to the backing on miss, and populate every tier on the unwind. Writes go to the backing first, then to the cache best-effort.

The wrapping is transparent: a transformer wired to $store.crm does not know whether reads hit a memory cache, a GCS tier, or the underlying Sheets API.

Core principle: the cache is advisory. Backing is the source of truth. Failed cache operations degrade performance, never correctness.

TTL is owned by the cache layer, not the store. The store persists structured values; the cache wrapper manages expiry via the per-rule ttl. An expired entry is treated as a miss and re-fetched from the backing. Caching works over any structured backing.

A store's byte-native file mode (config.file: true) is for serving raw assets byte-exact, not for caching. The cache wraps structured stores, so flow_validate warns when a store sets both file: true and cache. Drop one: serve bytes with file: true and no cache, or cache a structured store with no file.

When to use this skill

  • A transformer reads the same key many times across events (sessions, user lookups, product catalog enrichment)
  • A store has rate limits or slow HTTP round-trips (Sheets, custom API, S3 metadata); @walkeros/server-store-api reads a third-party API, and its misses are not cached
  • You need to deduplicate concurrent reads on a cold key (thundering herd)
  • You want to compose multi-tier caches (memory → GCS → Sheets)

Minimal recipe: cache a Sheets store

The talk-demo use case. A sessions lookup runs on every event but most events in a session share the same key.

{
  "stores": {
    "sessions": {
      "package": "@walkeros/server-store-sheets",
      "config": {
        "credentials": "$secret.SHEETS_CREDENTIALS",
        "settings": {
          "id": "1AbC...",
          "sheet": "Sessions"
        }
      },
      "cache": { "rules": [{ "ttl": 300 }] }
    }
  },
  "transformers": {
    "sessionLookup": {
      "code": {
        "type": "session-lookup",
        "push": "$code:async (event, context) => { if (!event.user?.session) return; const session = await context.env.store.get(event.user.session); if (session) return { event: { ...event, data: { ...event.data, session } } }; }"
      },
      "env": { "store": "$store.sessions" }
    }
  }
}

walkerOS ships no generic lookup transformer, so the consumer here is an inline code transformer. Any transformer or destination that reads its env.store works the same way.

The first lookup hits Sheets and populates the built-in in-memory tier with a 300-second TTL. The next 300 seconds of identical reads hit memory and skip the Sheets API.

Without the cache: 60 events in 60 seconds = 60 Sheets reads, each counted against the Sheets API read quota. With the cache: 60 events in 60 seconds = 1 Sheets read.

Recipe: memoize an object storage store

Same shape, longer TTL because every read is a network round-trip to the bucket:

{
  "stores": {
    "users": {
      "package": "@walkeros/server-store-gcs",
      "config": {
        "settings": { "bucket": "my-user-records", "prefix": "users/" }
      },
      "cache": { "rules": [{ "ttl": 3600 }] }
    }
  }
}

One-hour TTL is reasonable when user records change rarely. Use flow_validate to verify the config; the hit rate shows at runtime in the store's counters (see Observability).

Recipe: TTL by key prefix

Different keys can have different TTLs. Rules evaluate top-down, first match wins. The match context is { key, value? }, not event data:

"cache": {
  "rules": [
    { "match": { "key": "key", "operator": "prefix", "value": "session:" }, "ttl": 300 },
    { "match": { "key": "key", "operator": "prefix", "value": "user:" }, "ttl": 3600 },
    { "ttl": 60 }
  ]
}
  • session:* keys cache for 5 minutes
  • user:* keys cache for 1 hour
  • Everything else caches for 1 minute

A rule without match always matches. Place it last as a fallback.

Recipe: multi-tier composition (memory → GCS → Sheets)

When the working set exceeds the memory tier's capacity, or cached values should survive restarts and be shared across instances, add an object storage layer between memory and the cold backing. The consumer still wires to $store.crm; the tiers resolve automatically. Use fs, S3, GCS or the built-in memory tier as a cache tier (a cache.store target). Nothing rejects the Sheets store there, but every cache read and write then costs a Sheets request.

{
  "stores": {
    "files": {
      "package": "@walkeros/server-store-gcs",
      "config": {
        "settings": { "bucket": "my-cache-bucket", "prefix": "cache/" }
      },
      "cache": { "rules": [{ "ttl": 300 }] }
    },
    "crm": {
      "package": "@walkeros/server-store-sheets",
      "config": { "settings": { "id": "1AbC...", "sheet": "Customers" } },
      "cache": {
        "store": "files",
        "rules": [{ "ttl": 86400 }]
      }
    }
  },
  "transformers": {
    "crmLookup": {
      "code": {
        "type": "crm-lookup",
        "push": "$code:async (event, context) => { if (!event.user?.id) return; const crm = await context.env.store.get(event.user.id); if (crm) return { event: { ...event, data: { ...event.data, crm } } }; }"
      },
      "env": { "store": "$store.crm" }
    }
  }
}

Lookup chain on crm.get(K):

  1. crm's tier is the files store. The files wrapper checks its own tier (memory __cache) first, then GCS. HIT, return. A GCS HIT also populates memory.
  2. On all MISS, call the underlying Sheets store. Each traversed tier populates on the unwind: crm writes to files, which writes GCS and then memory.

TTL ordering: shortest at the top (the files memory tier, 300s), longest at the cold end (the crm entries in GCS, 86400s). Each wrapper checks the expiry of its own entries, also when a lower tier serves them, so a value is served stale for at most the longest TTL in the chain (86400s here).

Async-safe by design. Whether your cache store's get is synchronous (the built-in __cache, an in-memory store) or asynchronous (@walkeros/server-store-fs, S3, GCS, the cache wrapper itself), the collector reads through with an await internally. You can mix sync and async stores freely in a multi-tier chain without any extra configuration; the same HIT/MISS semantics apply.

Recipe: prevent thundering herd on a cold key

Single-flight deduplication is on by default. 50 concurrent store.get('session:abc') calls on a cold cache produce one backing call, not 50. All callers receive the same promise.

This is what makes store-level cache useful on a slow backing under high concurrency. No configuration needed; just set cache on the store.

Verify it works with the inflight_dedups counter (see Observability below).

What cache rules cannot do (compared to event cache)

Store rules are a stricter subset:

  • No key field. The cache key comes from the caller (store.get(K)); there is no event path to compose.
  • No update field. update changes the response a source cache sends; a store read has no response to change.
  • No stop field. Stores always fall through on miss; halting the pipeline is an event-cache concept.
  • Empty namespace: "" is rejected by the schema (re-introduces the collision footgun across stores sharing __cache).

The schema rejects key, update and stop on a store cache. Use the event-level Cache when you need them: key and stop on sources, transformers or destinations, update on a source cache only.

Default tier: built-in __cache

Omitting cache.store falls back to the collector's built-in __cache. It is an in-memory LRU map with:

  • maxEntries: 10000 (not configurable)
  • LRU access ordering on reads
  • Batched eviction down to 80% on overflow
  • Active TTL sweep every 60 seconds

Each wrapped store gets an automatic namespace prefix (the store id) so multiple stores sharing __cache do not collide. Override with cache.namespace: "myns" if you want explicit control.

At the INFO level the collector logs one line per wrapped store at startup:

INFO [store-cache:sessions] store "sessions" caches with namespace "sessions:" via __cache

Write-through error policy

wrapped.set(K, V) runs two steps:

  1. Backing first. Await backing.set(K, V). If this throws, the wrapper throws. The cache is not touched.
  2. Cache best-effort. If the backing succeeded, attempt to populate the cache. If this throws, log a warning and return success.

wrapped.delete(K) follows the same shape. A failed cache delete leaves a poisoned entry that serves stale data until TTL. Both warnings are WARN lines, which the default log level (ERROR) does not show.

Backing is the source of truth. Code that wraps set / delete should assume the cache may be lagging.

Coherence model

Read this before relying on the cache for anything correctness-sensitive:

  • Read-your-writes (in-process): yes. After wrapped.set(K, V), a subsequent wrapped.get(K) in the same process returns V.
  • Cross-process consistency: eventual, bounded by the longest TTL in the chain. No invalidation channel.
  • Tier-skipping repopulation: a MISS in tier N that HITs in tier N+1 repopulates tier N. Subsequent reads hit tier N.

Pick TTLs accordingly. Short TTLs (1-60s) for mostly-static lookups behind a fast backing; long TTLs (minutes-hours) for cold, expensive lookups where staleness is tolerable.

Observability

Each wrapped store exposes counters:

CounterUse this to detect
hitsCache is actually working
missesWorking-set size, cold start
populatesNew keys being added to cache
writesSet call volume
deletesDelete call volume
inflight_dedupsConcurrent reads on a cold key

The eviction counters (evictions_entries for the maxEntries cap, evictions_ttl for expired entries) belong to the built-in tier: collector.stores.__cache.counters. Each read through a wrapped store also emits a flow record to the collector's observers (meta.op: "cache", meta.status hit or miss).

For interactive debugging at runtime:

const { collector } = await startFlow({
  /* ... */
});
const sessions = collector.stores.sessions;
if ('counters' in sessions) console.log(sessions.counters);
// { hits: 412, misses: 18, populates: 18, writes: 0, deletes: 0,
//   inflight_dedups: 7 }

Healthy cache: hits / (hits + misses) rises over time. inflight_dedups proves the herd prevention worked.

Known limitations

  • No negative caching. A get(K) that returns undefined from the backing is not populated. Every subsequent call for a missing key re-hits the backing until the value exists. Workaround: write a sentinel value on the first miss and treat it as "not present" in transformer logic.
  • No cross-process invalidation. Writes from one process do not invalidate caches in other processes. TTL is the only mechanism.
  • Cycles are rejected at startup. A.cache.store = B and B.cache.store = A make startFlow reject with Cycle in cache.store chain: a -> b -> a.
  • A cached store is not destroyed at shutdown. The cache wrapper does not forward destroy, so a store declared with cache never has its destroy called.
  • Renaming a store is a breaking change to anything caching through it (cache.store: "X" references break). Migrate explicitly.

Fetch & stash without $code:

When a step only needs to read a value out of a store or write one into it, you do not need to wire $store into the step's env and hand-write a $code: push. The declarative state block on a source, transformer, or destination does both directions through the mapping engine. The example uses the sessions store declared in the first recipe:

"transformers": {
  "stashGclid": {
    "state": { "mode": "set", "store": "sessions", "key": "event.user.session", "value": "event.data.gclid" }
  },
  "restoreGclid": {
    "state": { "mode": "get", "store": "sessions", "key": "event.user.session", "value": "event.data.gclid" }
  }
}

key and value resolve against { event, ingest }, so every path names its side: event.user.session reads the event, ingest.site reads the pipeline context a server source lifted from the request. A path without one of these prefixes never resolves: walkeros validate rejects it and the runtime warns. key names the store slot and mode sets the direction:

  • set resolves value (a path, constant, fn, or map) and writes that payload to the store under key.
  • get reads key from the store and writes the fetched value to the value path: event.x onto the event, ingest.x into the ingest, where later steps and route matchers see it but no destination does. For a get, value must be a string path (or a ValueConfig with key), not a constant or operator.

A store keyed by request context, such as a per-site registry, reads the key from ingest:

"registry": {
  "state": { "mode": "get", "store": "registry", "key": "ingest.site", "value": "ingest.tenant" }
}

A transformer runs its get before its own next route, so the route can match on ingest.tenant. A source runs its state before its next route too, so a source-level get can feed that route.

To fill several fields from one lookup, add mapping. Its paths are relative to the value read, and the result merges into the target:

"loadUser": {
  "state": {
    "mode": "get", "store": "customers", "key": "event.user.id", "value": "event.user",
    "mapping": { "map": { "ltv": "ltv", "segment": { "key": "segment", "value": "unknown" } } }
  }
}

An object result merges into an object target, anything else replaces it; on a set the stored entry is read, merged and written back (not atomic). The mapping runs on a miss too, so only declared fallbacks are written; fn and condition then receive undefined, while validate receives the resolved value (a declared fallback, or {} after a map).

Omit store to use the built-in __cache tier; state keys there are prefixed with state: so they never collide with cache entries. State is fail-open: a store error is logged and the event passes through unchanged. A store not declared in flow.stores, or one with file: true, is a validation error (walkeros validate and deploy preflight). Use state for simple fetch/stash; reach for $code: only when the logic is genuinely non-declarative.

Full reference: Website: State.

Related Skills

Documentation:

Source files:

Using @walkeros/transformer-ga4

Overview

@walkeros/transformer-ga4 decodes Google Analytics 4 Measurement Protocol v2 hits (/g/collect, /mp/collect) into walkerOS events. It sits in a server source's before chain, reads the raw HTTP request via ctx.ingest, and returns one walkerOS event per GA4 event in the hit (one hit can carry many events).

Boundaries: server-side decoding in front of a server source, GA4 v2 only, G- tids only by default. Per-field patching via extend/remove is supported.

When to use this skill

  • You are wiring transformer-ga4 into a server flow for the first time
  • You want to override one of the 33 default GA4 mappings
  • You need to drop a GA4 event (auto-fired noise like user_engagement)
  • You want to add a custom event key for a gtag('event', '<name>', ...) call
  • You need to widen tidPattern to accept Ads (AW-) or DC (DC-) traffic
  • Events are not arriving and you suspect the decoder is silently dropping them

Quickstart wiring

Pick the source by runtime before copying the example below. gtag.js can send several events in one POST body (one per line, as text/plain), so the source must keep a non-JSON body as ingest.body. These server sources do:

  • @walkeros/server-source-express (shown below), the source runneros and the walkeros/flow image serve. A text/plain body that is valid JSON is parsed; any other text/plain body reaches the decoder as the raw string.

  • @walkeros/server-source-fetch: a (Request) => Response handler with no port, for a runtime that calls it (Cloudflare Workers, Deno, Bun, Node.js 18+ with a fetch adapter), e.g. with collector from startFlow:

    import { Source } from '@walkeros/core';
    import type { SourceFetch } from '@walkeros/server-source-fetch';
    
    export default {
      fetch: Source.getSource<SourceFetch.Types>(collector, 'http').push,
    };
    

    runneros and the walkeros/flow image cannot serve it (they only mount a source's Node httpHandler).

  • sourceCloudFunction from @walkeros/server-source-gcp on Google Cloud Functions.

  • sourceLambda from @walkeros/server-source-aws on AWS Lambda. It answers a POST without a body with 400, so body-less single-event hits are lost there.

The other sources take the same ingest and before config as the express example. The fetch source also takes paths; none of them takes port.

{
  "version": 4,
  "flows": {
    "default": {
      "config": { "platform": "server" },
      "sources": {
        "http": {
          "package": "@walkeros/server-source-express",
          "config": {
            "settings": {
              "port": 8080,
              "paths": ["/g/collect"]
            },
            "ingest": {
              "map": {
                "url": { "key": "url" },
                "path": { "key": "path" },
                "method": { "key": "method" },
                "body": { "key": "body" }
              }
            }
          },
          "before": "ga4"
        }
      },
      "transformers": {
        "ga4": { "package": "@walkeros/transformer-ga4" }
      },
      "destinations": {
        "log": { "package": "@walkeros/destination-demo" }
      }
    }
  }
}

Wiring rules:

  • The express source listens on /collect by default and matches paths exactly. gtag.js sends to /g/collect, so set settings.paths to ["/g/collect"] (or the full path, if the collector URL carries a path prefix). Without it every GA4 hit gets a 404.
  • On the website, point gtag at the collector origin: gtag('config', 'G-XXXXXXXXXX', { server_container_url: 'https://collect.example.com' }). gtag.js appends /g/collect. With the walkerOS gtag destination, set settings.ga4.server_container_url (or settings.ga4.transport_url). Hits then go to the collector instead of Google. The /g/collect suffix, the body-less single-event POST, and one-line-per-event batches are gtag.js behaviour, not a walkerOS contract: confirm them in the browser Network tab.
  • before: "ga4" runs every request on the source through the decoder, and a request without GA4 parameters is dropped. If the source also receives walkerOS events (for example on /collect), gate the chain: "before": { "match": { "key": "ingest.path", "operator": "eq", "value": "/g/collect" }, "next": "ga4" }.
  • Batched hits (several events in one POST body, one per line) reach the decoder through server-source-express as the raw text/plain string, alongside GET and body-less POST hits. A body sent as application/json must be valid JSON, or express answers 400 before the decoder runs.
  • transformer-ga4 belongs in source.before, not destination.before. Decoding is a pre-collector concern: GA4 hits are not yet walkerOS events.
  • The source must populate ctx.ingest.url (required, string) and ctx.ingest.body (optional, string). Raw text body only: pre-parsed JSON will not decode. The express, fetch, GCP and AWS server sources keep a non-JSON text/plain body (gtag's batched POST) as the raw string in ingest.body; only a body that parses as JSON arrives parsed.
  • One HTTP request can fan out to N walkerOS events.

Common mistake: config.ingest must be the map operator with direct field paths on the request scope (no req. prefix), e.g. { "map": { "url": { "key": "url" } } }. The bare { "url": "req.url" } form has no operator: walkeros validate and the bundle reject it (MAPPING_VALUE_NO_OPERATOR). If it still runs, the collector logs an ERROR, ctx.ingest stays empty, and the decoder fails every hit with missing-input.

This wiring contract (before-chain placement plus the config.ingest keys the source must populate) is also surfaced as a package hint, so package_get on @walkeros/transformer-ga4 returns it without this skill.

Configuration recipes

Patch one field in a default mapping (extend + remove)

Use extend to add or override individual fields of a shipped default rule without replacing it in full. Use remove to strip fields from the output.

The purchase default ships id, currency, value, tax, shipping, and coupon. To add affiliation from a GA4 event parameter and drop currency:

"transformers": {
  "ga4": {
    "package": "@walkeros/transformer-ga4",
    "config": {
      "settings": {
        "mapping": {
          "purchase": {
            "extend": {
              "data": { "map": { "affiliation": "params.ep.affiliation" } }
            },
            "remove": ["currency"]
          }
        }
      }
    }
  }
}

extend.data.map is deep-merged onto the default data.map, leaving id, value, tax, shipping, and coupon intact. remove: ["currency"] strips that field from the final payload. A null value in extend clears an inherited field, for example "extend": { "data": { "map": { "tax": null } } } drops tax from the default data.map.

Full replace: if you need to rewrite a rule from scratch (no merge), omit extend and remove and supply the complete rule directly. A rule with neither keyword keeps the existing replace behavior.

"settings": {
  "mapping": {
    "purchase": {
      "name": "order complete",
      "data": {
        "map": {
          "id": "params.ep.transaction_id",
          "value": "params.epn.value",
          "currency": "params.ep.currency",
          "coupon": "params.ep.promo_code"
        }
      }
    }
  }
}

Drop an event

Set ignore: true on any key. This is how user_engagement, session_start, and first_visit are silenced by default:

"settings": {
  "mapping": {
    "click": { "ignore": true }
  }
}

Add a custom event mapping

For events fired via gtag('event', '<your_name>', ...):

"settings": {
  "mapping": {
    "newsletter_subscribe": {
      "name": "newsletter signup",
      "data": {
        "map": { "source": "params.ep.source" }
      }
    }
  }
}

Unknown event names fall through to the '*' key, which by default emits ga4 track with data.event_name set to the original en. Override '*' to change the global fallback rule.

Set a custom tidPattern

By default only G- tids are accepted. Widen via a string regex (compiled at init):

"settings": {
  "tidPattern": "^(G|AW|DC)-"
}

Change the batch cap

settings.maxEvents (default 100) caps the events decoded from one request. A POST body with more non-empty lines is dropped whole, before decoding:

"settings": {
  "maxEvents": 250
}

Custom destination after decode

The transformer returns walkerOS events with entity action names. Anything downstream of the collector treats them as native events and needs no GA4 awareness:

"destinations": {
  "bq": {
    "package": "@walkeros/server-destination-gcp",
    "import": "destinationBigQuery",
    "config": {
      "settings": {
        "projectId": "my-eu-project",
        "datasetId": "events",
        "location": "EU"
      }
    }
  }
}

Mapping path syntax

Mappings reference fields on the decoded GA4Hit shape via dotted paths:

PathMeaning
params.ep.XEvent parameter (string), e.g. params.ep.transaction_id
params.epn.XEvent parameter (numeric), e.g. params.epn.value
params.up.XUser property (string)
params.upn.XUser property (numeric)
items[0].XFirst item in the items array
itemsFull items array (mapped to nested via item rules)

Use params.ep.X not ep.X. The decoder materializes the prefixed keys under the params namespace.

Hit-level fields on the decoded event

Set from the hit, each only when present, value as sent.

Hit paramEvent field
dlsource.url (page URL)
drsource.referrer
uid, cid, siduser.id, user.device, user.session
uluser.language (browser language)
sruser.screenSize (screen resolution)
p, _p, _ssource.platform, source.pageLoadId, source.hitSequence
  • globals is left empty: globals.language is the site's language, not the visitor's. Mappings or destinations that read globals.language or globals.screen from decoded events must switch to user.language and user.screenSize.
  • timestamp is the receive time (Date.now()), not the session start (sid). Event ids do not depend on it.
  • source.url feeds server destinations that send a page URL (Meta, Snapchat, Pinterest, Bing, TikTok, Criteo, Piwik PRO; Piwik PRO fails events without it with missing-input) and the fingerprint transformer's default site input. Fingerprint hashes of GA4-decoded events therefore change once on upgrade.

Troubleshooting

No events arriving

  1. A 404 on /g/collect means the source does not listen there. Set settings.paths to include /g/collect (the express and fetch sources default to /collect).
  2. Check that source.config.ingest is the map operator and populates url (required, string) and body (optional, string), e.g. { "map": { "url": { "key": "url" }, "body": { "key": "body" } } }. A bare { "url": "req.url" } object fails walkeros validate (MAPPING_VALUE_NO_OPERATOR); at runtime ctx.ingest stays empty and every hit fails with missing-input, one ERROR line per cause and minute.
  3. Confirm before: "ga4" is set on the source, not on a destination.
  4. Check the tidPattern: by default ^G- blocks AW- / DC- traffic silently.
  5. Confirm the request really is Measurement Protocol v2. v1 hits will not decode.
  6. A POST body with more non-empty lines than settings.maxEvents (default 100) is dropped whole. The client still gets a success response; the only trace is one ERROR line with the line count and the cap. Raise the cap if your own server-to-server batches are larger.

Wrong field in the output event

  1. Check the path syntax: params.ep.X, not ep.X.
  2. Verify the GA4 hit actually carries the parameter. Use a debug destination to log the decoded params object.
  3. If you used a full-replace rule (no extend), your data.map is the entire rule. Use extend to inherit the defaults and add only the fields you need.

Batched POST hits return 400

@walkeros/server-source-express only rejects a POST body that is sent as application/json and does not parse. gtag.js sends text/plain, which express keeps as the raw string. If a proxy in front of the collector rewrites the content type to application/json, restore text/plain.

gcs consent not applied

v1 decodes basic gcs (G1XX → marketing / analytics booleans) only. Functional/preferences flags and the newer gcd parameter are not decoded. Override the consent path manually if you need richer mapping.

Limitations

  • GA4 v2 only. v1 Measurement Protocol is out of scope.
  • G- tids only by default. Override tidPattern for Ads/DC.
  • Basic gcs only. No gcd, no functional/preferences flags.
  • Raw text body required. Pre-parsed JSON bodies will not decode. Batched gtag POSTs (text/plain, one URL-encoded hit per line) reach ingest.body raw on the express source.
  • At most maxEvents events per request. Default 100; a larger batch is dropped whole before decoding, because each event becomes its own push and a source's batch limit does not cover a raw text body.
  • Server-side only. Web ingest via interception sources is not supported.

Related Skills

See the website docs for the authoritative reference and the README for the in-repo summary.

Writing Documentation

When to Use This Skill

  • Creating a new package README
  • Writing website documentation (MDX)
  • Creating or updating skills
  • Reviewing documentation for quality
  • Documenting Phase 7 of create-destination or create-source

Prerequisites


Documentation Types

Where Content Belongs

TypePurposeAudience
Package READMEInstallation, basic usage, API referencePackage users
Website docsGuides, integration examples, detailed configIntegrators
SkillsProcess knowledge, workflowsAI assistants, contributors

Divio Documentation Types

Keep these separate - don't mix tutorials with reference:

TypePurposeUser State
TutorialLearningStudying, beginner
How-To GuideProblem-solvingWorking, knows what they need
ReferenceInformation lookupWorking, needs facts
ExplanationUnderstandingStudying, needs context

Example Validation (CRITICAL)

The Problem

AI-generated examples can be:

  • Syntactically correct but use non-existent APIs
  • Plausible-looking but don't match actual exports
  • Outdated, referencing deprecated patterns

Source of Truth Hierarchy

TIER 1: apps/quickstart/
  ✓ Tested  ✓ Compiled  ✓ CI-validated
  → USE FOR: All code examples

TIER 2: packages/core/src/eventGenerator.ts
  ✓ Canonical events  ✓ Real data structures
  → USE FOR: Event examples

TIER 3: packages/*/src/index.ts exports
  ✓ Actual public API
  → USE FOR: Verifying API names exist

TIER 4: Package READMEs & Website docs
  ⚠ May contain errors
  → VERIFY against Tier 1-3 before trusting

Pinned onboarding examples

Getting-started and onboarding code examples are sourced from Tier-1 apps/quickstart/src/*.ts (CI-validated) first, then the docs source them. The current canonical examples:

  • first-event.ts, the zero-to-first-event console example used by the quickstart and integrated-mode pages and the README.
  • ga4-ecommerce.ts, the product add to GA4 add_to_cart mapping used by the GA4 reference page and the GA4 ecommerce tutorial.

These are pinned against drift by the doc-lint at apps/scripts/validate-docs.ts (run npx tsx apps/scripts/validate-docs.ts). It extracts canonical tokens from the Tier-1 source and asserts the docs snippets contain them, tolerating intentional doc simplifications (dropping the seen capture, arrow push instead of a method). first-event.ts is pinned on the quickstart, integrated-mode, and README pages, plus the GA4 ecommerce page; react.mdx is pinned shape-only (it pushes no page view).

The doc-lint also runs these structural and coherence checks:

  • elb event form (docs, skills, READMEs): every event is pushed as a partial event, elb({ name: 'entity action', data }). A quoted "entity action" string as the first elb argument is an error; walker commands keep their string name (elb('walker consent', ...)).

  • Bundled-snippet platform guard: every JSON flow with bundle carries config.platform (getting-started, README, flow reference).

  • Docker env guard: quickstart Docker snippets use BUNDLE, not FLOW.

  • Forbidden CLI verbs: no walkeros serve or standalone walkeros simulate in getting-started, guides, or the gtag page.

  • Same-page simulate-key resolution (getting-started): every push --simulate <type>.<key> must resolve to a step key declared in a flow.json on the SAME page; a simulate command on a page with no flow.json fails. This is what catches an integrated page showing a bundled push loop.

  • Flow.json schema validation (all docs): every flow snippet, fragments included, is checked with core's validators (validateFlowConfig, then validateFlowStructure, from @walkeros/core/dev). A fragment that starts at a key gets braces; a flow part (sources, destinations, ...) is checked inside a v4 flow, with the platform from the fragment or the page path; a root part (contract, variables) is checked as a v4 file; on a page whose frontmatter package: is a step package, a step or step config part is checked as that step. Package settings are not checked against the package's schema, and nothing is fetched. version: 3 migration "before" snippets and JSON that is no flow part are skipped and counted in the summary line.

  • Boundary register (getting-started + MCP page): no prices, tier names, or upgrade language. No :::cloud (all docs): there is no walkerOS Cloud, and the callout type no longer exists.

The rule: when you edit a pinned example, edit the Tier-1 .ts source and re-run the doc-lint. If the docs snippet must change too, update both so the lint passes.

Examples must run

A package page shows only setup and examples that a test ran, rendered from the package's walkerOS.json; never a hand-written copy of them:

  • Install and setup are generated (destination pages). ## Installation renders <Quickstart data={data} /> (with export="<name>" for a package with several exports): the install line, the integrated code and the flow.json, composed from the package's examples.minimal, the settings its conformance test delivers with. For a package with src/examples/minimal.ts, the doc-lint fails an npm install, startFlow( or bundled tab under ## Installation, and a section that does not render <Quickstart>. A destination whose conformance test passes withoutMinimal (gtag, sqlite) keeps a hand-written setup, as do source, transformer and store pages.
  • Frontmatter names the data. The page's package: frontmatter and the walkerOS.json it imports name the same package; the doc-lint fails a mismatch.
  • Failures come from examples. <Configuration> lists every step example with an outcome under ## When it fails (title, description, status and code; never its in). Add a failure mode as a failure example with a description in the package instead of a hand-written list on the page; the docs build fails on a failure example without a description.

Hand-written examples (source, transformer, store and guide pages, READMEs) follow the same bar:

  • A flow runs. Every complete JSON flow example ("version": 4) passes walkeros validate --strict and runs with walkeros push --simulate; validate alone does not prove a config does what the page says.
  • Values satisfy the schema. Placeholders match the setting's pattern (a numeric pixel id where the schema asks for digits), and timestamps are milliseconds wherever the code reads milliseconds.
  • Output carries no real version. Example output uses a stand-in such as release: { web: '3' }; a real version number goes stale with the next release.
  • Renamed headings keep their anchor. A heading's slug is a public URL. When its text changes, pin the old slug (## New title {#old-slug}) and update inbound links.

The package side (minimal, withoutMinimal, failure examples) is in using-step-examples.

Getting-started editorial rules

Three rules govern the getting-started surface (decided 2026-07-06, encoded in the doc-lint):

See your event, per mode. Integrated-mode pages (quickstart, react, nextjs, modes/integrated, ga4-ecommerce) verify via their own console output; they must never show walkeros push --simulate. Only bundled-mode pages may show it, and only targeting a <type>.<key> that a flow.json on the SAME page defines.

Open/paid boundary (Self-Host Sufficiency Rule). Every getting-started task must reach a working result with MIT packages, the CLI, and reader-controlled infrastructure only. There is no walkerOS Cloud: never write "Cloud" as a walkerOS product, and never add a "What Cloud adds" pointer. Where docs describe commands or tools that talk to the hosted app (auth, projects, flows, deploy), call it "the walkerOS app". Never prices, tier names, or upgrade language anywhere in open docs. Self-host is always listed first.

Examples recipes. Pages under getting-started/examples/ are lean proof recipes with a fixed shape: exactly one flow.json, one run loop, one expected-output block, plus a short intro and next steps. The run loop uses walkeros push … --simulate <type>.<key> where <key> is a step key declared in the page's own flow.json; recipes are bundled flows, so the per-mode rule applies and the same-page simulate gate enforces resolution. The expected-output block shows what actually arrives (the captured destination call or the logged event), derived from the package's step examples or the real CLI output; never invent config, mappings, or output. Every snippet is derived from validated material (package examples, tested sources, existing validated docs snippets). The shape is enforced by checkRecipeProof in apps/scripts/validate-docs.ts.

Validation Checklist

Before publishing ANY code example:

  • API exists? Check packages/*/src/index.ts exports
  • Pattern validated? Compare against apps/quickstart/
  • Events canonical? Use patterns from eventGenerator.ts
  • Example compiles? TypeScript check passes
  • Imports correct? Package names match actual packages
  • Env vars, CLI flags, container modes exist? Each one is grepped in source before it is documented (e.g. grep -rn "process.env.PORT" packages/runner/src, grep -n -- "--port" packages/runner/src/bin.ts). A name that only appears in other docs is not evidence

Red Flags

Red FlagWhat It Indicates
API name not in package exportsHallucinated or outdated API
Import path doesn't match package.jsonWrong package reference
Event name with underscoreWrong format (should be space)
No imports shownContext missing, harder to validate
Env var or flag with no source grepLikely fiction copied from old docs

DRY Patterns

Configuration snippet (package pages)

When to use: Every package page for destinations, sources, transformers, and stores. The <Configuration> snippet owns the ## Configuration, ## Settings, ## Mapping, ## Examples and ## When it fails H2 sections. One callsite per page, after ## Installation.

import data from '@walkeros/web-destination-clarity/walkerOS.json';
import Configuration from '@site/src/components/snippets/_configuration.mdx';
import Quickstart from '@site/src/components/snippets/_quickstart.mdx';

## Installation

<Quickstart data={data} />

<Configuration type="destination" data={data} />

A page without a minimal example (sources, transformers, stores, gtag, sqlite) shows a <CodeSnippet> install line under ## Installation instead of <Quickstart>.

type must be one of destination, source, transformer, store. data is the full walkerOS.json import. The snippet:

  • emits ## Configuration with a short pointer to the shared group-level configuration reference,
  • emits ## Settings using <PropertyTable schema={data.schemas.settings} /> when the schema is present, otherwise a "no package-specific settings" note,
  • emits ## Mapping using <PropertyTable schema={data.schemas.mapping} /> when present, otherwise a pointer to the standard rule fields,
  • emits ## Examples only if data.examples.step holds a public example (public not false) and renders each public one through <StepExample>,
  • emits ## When it fails when a step example has an outcome, one line per failure example from its description.

Never hand-roll ## Configuration, ## Settings, ## Mapping, ## Examples or ## When it fails on a package page. Place package-specific content (vendor quirks, advanced usage, platform-specific setup) BELOW the <Configuration> block with its own sentence-case H2 heading. Fold ## Features bullet lists into the intro paragraph as prose: no ## Features heading on package pages.

Shared configuration reference (group index pages)

On group index pages (docs/destinations/index.mdx, sources/index.mdx, transformers/index.mdx, stores/index.mdx), add a ## Configuration section that documents the wrapper fields around settings: the fields shared across all packages of that type (consent, mapping, env, id, …).

Drive the table from the core Zod schemas, filtering dev-only fields at the representation layer. configJsonSchema holds the config under definitions (its root has no properties), so unwrap it before omitting fields:

import { schemas } from '@walkeros/core/dev';
import { omitSchemaProperties } from '@site/src/utils/schema';

export const destinationConfig = (({ definitions }) =>
  omitSchemaProperties({ ...definitions.DestinationConfig, definitions }, [
    'settings',
    'init',
    'mock',
  ]))(schemas.DestinationSchemas.configJsonSchema);

## Configuration

<PropertyTable schema={destinationConfig} />

Available schemas:

  • schemas.DestinationSchemas.configJsonSchema
  • schemas.SourceSchemas.configJsonSchema
  • schemas.TransformerSchemas.configJsonSchema
  • schemas.StoreSchemas.configJsonSchema

Use omitSchemaProperties to hide settings (documented per-package) and dev-only fields such as init and mock from the user-facing table (the definition names are DestinationConfig, SourceConfig, TransformerConfig and StoreConfig).

Code display: CodeView vs CodeBox

@walkeros/explorer exports two components for code display:

  • <CodeBox> renders Monaco, the full editor. Use when the user can edit the code (playgrounds, interactive demos). Ships the IDE.
  • <CodeView> renders Shiki, read-only. Use for static code in docs, marketing pages, and anywhere editing is not a requirement. Same <Box> frame as <CodeBox> (traffic lights, header, copy button, tabs) so visuals match. Zero Monaco runtime cost.

Tokens match because Shiki uses the same TextMate grammars as VS Code / Monaco, with the matching dark-plus / light-plus themes.

<CodeView> and <CodeSnippet> render their code server-side, so it appears in the static HTML and the markdown/llms.txt export. Code is shown exactly as authored (no automatic reformatting). npm run validate:snippets (part of validate:sources, run in CI) fails on a docs <CodeSnippet> that Prettier would format differently, and npm run validate:snippets -- --write fixes it in place. Excerpts Prettier cannot parse (settings: {...}, { ... } placeholders, a lone opening tag) are only checked for indentation: indent them by hand, two spaces per level. JSX belongs in language="tsx", not typescript.

Rule: default to <CodeView>. Only reach for <CodeBox> when the code must be editable.

PropertyTable component

<PropertyTable schema={...} /> from @walkeros/explorer is the primitive. It renders a pure table, with no heading and no captions. Never wrap it in a custom heading on package pages; use the <Configuration> snippet instead.

When NOT to use:

  • Pages without package configuration
  • Reference tables (Logger API, CLI commands)
  • Conceptual explanations

Schema Exports (dev.ts)

Every destination/source should export schemas:

// src/dev.ts
export * as schemas from './schemas';
export * as examples from './examples';

Don't Duplicate

  • Link to source files instead of copying type definitions
  • Reference apps/quickstart/ examples instead of writing from scratch
  • Use PropertyTable instead of hardcoded markdown tables

Writing Hints (src/hints.ts)

Hints are the "experienced colleague" layer in walkerOS.json: they tell AI agents when, why, and what to watch out for beyond what schemas and examples convey. Surfaced via MCP package_get. Not human-facing docs.

Audience: AI agents configuring packages on behalf of users.

Core Principle: Expand Awareness, Don't Narrow It

Hints should open up the space of what's possible, not prescribe a single path. An LLM reading hints should think "I have more options than I realized", not "I must follow these steps exactly."

Writing Rules

RuleDoDon't
Describe capabilities"Supports SA key, ADC, and custom client""Use a SA key file when outside GCP"
Reference schemas/examples"See settings.projectId in the schema"Repeat what the schema description says
Explain why behind defaults"Defaults to EU location; override via location""Set location to US"
Flag non-obvious interactions"When snakeCase: true, all data keys transform before send"Describe obvious behavior
Symptoms → causes"Empty table? Check projectId and dataset existence"Step-by-step fix instructions

Key Naming

  • kebab-case, group related hints with prefixes: auth-*, storage-*, query-*, troubleshoot-*
  • Keep keys descriptive enough to scan: auth-methods not a1

When to Add Hints

Most packages don't need hints: schemas and examples cover the common case. Add hints when:

  • Multiple auth or config strategies exist and it's non-obvious when to use which
  • Non-obvious default behaviors need explaining
  • Features interact in ways the schema can't express
  • Prerequisites outside walkerOS are required
  • Common troubleshooting patterns exist

Accuracy Check

Before publishing hints, verify each claim is factually correct. Don't describe features that aren't implemented. Don't assume behavior; confirm it.

Quality Check

  • Each hint expands awareness (describes capabilities, not prescriptions)
  • Code snippets reference schema fields, not duplicate them
  • No hint restates what a schema .describe() already says
  • Troubleshooting hints use symptom → cause format
  • Hints are atomic: one concept per hint
  • Every claim is factually correct against current implementation

Export Pattern

// src/hints.ts
import type { Hint } from '@walkeros/core';

export const hints: Hint.Hints = {
  'auth-methods': {
    text: 'Supports three auth methods: ...',
    code: [{ lang: 'json', code: '{ "settings": { ... } }' }],
  },
};
// src/dev.ts
export * as schemas from './schemas';
export * as examples from './examples';
export { hints } from './hints';

Note: hints is a direct export (not * as), because it's already a Record<string, Hint>.


Quality Checklist

Structure

  • Follows appropriate Divio type (Tutorial/How-To/Reference/Explanation)
  • Code example within first 100 words
  • First example under 20 lines
  • Uses <details> for advanced content
  • Has "Next Steps" or "Related" section

Content

  • All event names use "entity action" format with space
  • Events are pushed as partial events: elb({ name, data })
  • Flow config shown as primary usage pattern
  • Examples are complete and copy-pasteable
  • Includes imports in code examples

AI Readability

  • Clear semantic headers (H2, H3, H4 hierarchy - no skipped levels)
  • Tables for structured data
  • Links to source of truth TypeScript files
  • Static fallback content alongside dynamic components

Consistency

  • Uses standard table formats
  • Follows package/skill/website templates
  • Terminology matches: walkerOS, collector, destination, source
  • Headings use sentence case (e.g., "Next steps" not "Next Steps")
  • walkerOS.json convention followed (walkerOS field in package.json, buildDev in tsup)

Templates

Package README Template (thin pointer)

READMEs are thin pointers to the website docs, which are the single source of truth. A README carries only: logo, title, one-line description, install, ONE current version: 4 Quick Start, and a prominent link to the package's docs page. No ## Features, no ## Configuration Reference, no inline API reference, no ## Examples block. All reference content lives on the website so it can never drift in two places.

The docs link MUST be an absolute https://www.walkeros.io/docs/... URL, never a relative path: packages publish to npm individually and relative links break on npmjs.com. These absolute docs links are validated by npm run validate:links (it maps the URL to the website/docs/ source file and asserts it exists).

<p align="left">
  <a href="https://www.walkeros.io">
    <img alt="walkerOS" title="walkerOS" src="https://www.walkeros.io/img/walkerOS_logo.svg" width="256px"/>
  </a>
</p>

# @walkeros/[package-name]

[one-sentence description, adapted from the docs page intro]

[Documentation](https://www.walkeros.io/docs/[path]) &bull;
[NPM Package](https://www.npmjs.com/package/@walkeros/[package-name]) &bull;
[Source Code](https://github.com/elbwalker/walkerOS/tree/main/packages/[path])

## Installation

```bash
npm install @walkeros/[package-name]
```

## Quick start

```json
{
  "version": 4,
  "flows": {
    "default": {
      "config": { "platform": "[web|server]" },
      "[sources|destinations]": {
        "[name]": {
          "package": "@walkeros/[package-name]",
          "config": { "settings": { "[requiredSetting]": "[value]" } }
        }
      }
    }
  }
}
```

## Documentation

Full configuration, mapping, and examples live in the docs:
**https://www.walkeros.io/docs/[path]**

## Contribute

Feel free to contribute by submitting an
[issue](https://github.com/elbwalker/walkerOS/issues), starting a
[discussion](https://github.com/elbwalker/walkerOS/discussions), or getting in
[contact](https://calendly.com/elb-alexander/30min).

## License

MIT

Verify the Quick Start against apps/quickstart/ and the package's real export name before publishing, and run it through walkeros validate --strict. It shows the settings the package requires (a destination's examples.minimal): "config": {} for a package with required settings fails --strict. Platform-agnostic packages (core, collector) show their most representative current usage instead of a config.platform flow.

walkerOS.json

The walkerOS field in package.json describes the package's role, and the build (buildDev) writes it with the schemas, examples and hints into dist/walkerOS.json:

{
  "walkerOS": {
    "type": "destination",
    "platform": ["web"],
    "docs": "https://www.walkeros.io/docs/destinations/web/[name]"
  }
}

type is the step kind, platform a platform or a list of them, docs the docs page URL; a package with several exports adds exports, each export name mapped to a short description. The README does not repeat it.

Website Doc Template (MDX)

Website docs fold features into the intro paragraph (no ## Features heading). Thin-pointer READMEs also have no ## Features heading; key capabilities go into the one-line description. This template is website-only.

---
title: [Title]
description: [SEO description]
sidebar_position: [N]
package: '@walkeros/[package]'
---

import data from '@walkeros/[package]/walkerOS.json';
import Configuration from '@site/src/components/snippets/_configuration.mdx';
import Quickstart from '@site/src/components/snippets/_quickstart.mdx';

# [Title]

<PackageLink package="@walkeros/[package]" />

[1-sentence description that folds in key features as prose; no separate `##
Features` heading on website docs.]

## Installation

<Quickstart data={data} />

{/* Without a minimal example: <CodeSnippet code={`npm install @walkeros/[package]`} language="bash" /> */}

<Configuration type="destination" data={data} />

## Next steps

- [Related guide 1](/docs/...)

Priority Matrix

Issue Classification

PriorityCriteriaAction
P0 CriticalIncorrect examples, wrong APIs, security issuesFix immediately
P1 HighMissing PropertyTable, outdated domains, missing sectionsFix soon
P2 MediumInconsistent terminology, skipped headingsPlan to fix
P3 LowStyle issues, minor wordingBacklog

Non-Negotiables

Punctuation

Never use em dashes (U+2014) or en dashes (U+2013). Use a comma, period, or rephrase the sentence instead: write "free and without sampling caps" rather than joining "free" and "without sampling caps" with an em dash. Check a file with grep -nP "[\x{2013}\x{2014}]" <file>.

Event Naming

Event naming ("entity action", space not underscore): see walkeros-understanding-events.

Package References

CORRECT: `@walkeros/collector` (with backticks)
WRONG:   @walkeros/collector (no backticks)

Domain References

CORRECT: `www.walkeros.io` or relative paths
WRONG:   the company's old domain (the doc-lint warns on any mention of it)

Process

For New Package Documentation

  1. Verify examples exist in apps/quickstart/ or create them first
  2. Write README using template above
  3. Write website doc using MDX template
  4. Run quality checklist
  5. Verify all code examples against Tier 1-3 sources

For Documentation Updates

  1. Identify issue priority using matrix above
  2. Check current state against source of truth
  3. Make minimal changes - don't over-engineer
  4. Verify examples still compile
  5. Run quality checklist

Schema naming (.meta({id, title}))

Every exported schema in packages/core/src/schemas/*.ts (and in the *Schema exports of destination / source / transformer / store packages) must carry .meta({ id, title, description }) so the generated JSON Schema links back to the canonical TypeScript name. Without meta, the website's PropertyTable falls back to __schema0, object, or any.

Convention:

  • id: PascalCase, namespace-prefixed so it is globally unique across every schema in packages/core/src/schemas. Examples: WalkerOSConsent, DestinationConfig, CollectorPushContext, LoggerConfig.
  • title: dotted form matching the VS Code TS hover: WalkerOS.Consent, Destination.Config, Collector.PushContext, Logger.Config.
  • description: one short sentence. May repeat an existing .describe() where both are present.

Reference implementation (copy this shape for new schemas):

export const ConsentSchema = z.record(z.string(), z.boolean()).meta({
  id: 'WalkerOSConsent',
  title: 'WalkerOS.Consent',
  description: 'Consent state mapping (group name to granted state).',
});

Coverage is enforced by packages/core/src/schemas/__tests__/meta-coverage.test.ts. When adding a new schema, run that test: it fails with the exact schema name if meta is missing.

Destination / source / transformer / store packages that ship their own SettingsSchema should also call .meta({ id, title }) on it, as the CMP sources do (id: 'CookiebotSettings').


Related Skills

Sponsors

Details

Marketplace
elbwalker/walkerOS
Category
Community