
walkeros
Skills and MCP tools for walkerOS - build sources, destinations, transformers, and flows.
- GitHub stars
- 351
Install
/plugin install walkeros@elbwalkerRun it inside Claude Code after adding the marketplace with /plugin marketplace add elbwalker/walkerOS.
Contents
Discovered from sourceSkills 22
Create a CMP Source
A CMP source is a specialized walkerOS source that listens to a consent
management platform's events and translates consent states into
elb('walker consent', state) calls.
Every CMP source follows the same skeleton with only 5-6 decision points that vary per CMP. This skill turns "build a new CMP source" into a structured fill-in-the-blanks workflow.
Prerequisites
Before starting, read these skills:
- understanding-flow - How sources fit in architecture
- understanding-sources - Source interface
- create-source - General source creation workflow
- testing-strategy - How to test
- writing-documentation - Documentation standards (for Phase 8)
Supporting files
This skill includes reference files you can copy:
- examples/ - Generic CMP consent examples to adapt for your
CMP
- inputs.ts - Consent input scenarios (full, partial, minimal, revocation)
- outputs.ts - Expected walkerOS consent states after mapping
- env.ts - Mock environment factories (createMockElbFn, createMockWindow, mockEnv)
- templates/cmp/ - Complete CMP source implementation
template
- index.ts - Source implementation with detection paths + handleConsent
- types.ts - Type definitions (Settings, CMP API, Types bundle)
- test-utils.ts - Test utilities (MockWindow, createMockElb, createCmpSource)
- index.test.ts - Test suite skeleton (8 describe blocks)
Note: These are generic templates for the skill. The actual CMP package's
examples go in packages/web/sources/cmps/[name]/src/examples/.
Canonical template
CookieFirst at packages/web/sources/cmps/cookiefirst/ is the canonical
template. Copy its structure for every new CMP source.
Note: The CookieFirst package.json predates this skill and lacks the
walkerOS-source keyword. New CMP sources MUST include it (the CookiePro and
Cookiebot packages do).
Process overview
1. Research -> Fill in the CMP research template
2. Examples -> Create input/output examples (full, partial, minimal, revocation, edge cases)
3. Mapping -> Define category map with sensible defaults
4. Scaffold -> Copy from CookieFirst template
5. Convention -> walkerOS.json, buildDev
6. Test -> Write tests FIRST (TDD): 25-32 tests across 8-9 describe blocks
7. Implement -> Wire up detection paths + handleConsent + destroy
8. Document -> README + docs page, registered with the other CMP sources
Phase 1: Research the CMP
Goal: Fill in every field of this template before writing any code.
CMP research template
Fill in ALL of these fields for the target CMP:
| Field | Description | Example (CookieFirst) |
|---|---|---|
| CMP name | Official product name | CookieFirst |
| Global window object | window.X shape and key properties | window.CookieFirst.consent (boolean map) |
| SDK events | Event names, CustomEvent detail structure | cf_init (Event), cf_consent (CustomEvent with consent detail) |
| Callbacks/hooks | Function wrapping patterns | None |
| Category naming | Human-readable vs opaque IDs | Human-readable (necessary, functional, performance, advertising) |
| Consent access pattern | Is consent read from a property (window.CMP.consent) or an API method (CMP.getConsent())? | Property: window.CookieFirst.consent |
| Explicit consent detection | API or mechanism to check if user actively chose | consent === null means no explicit choice |
| "Already loaded" detection | How to detect CMP loaded before source | window.CookieFirst.consent exists and is non-null |
| Official docs URL | Link to CMP's developer/API documentation | CookieFirst Public API docs |
| npm packages / TS types | Available type packages | None (define own types) |
| Timing constraints | Does the source need to load before/after the CMP? Any require config needed? | No require: consent sources should init immediately |
| Event registration mechanism | How does the CMP register event listeners? addEventListener, callback assignment, SDK method? | addEventListener (standard DOM events) |
| Cleanup/unsubscribe mechanism | How to remove listeners on destroy? removeEventListener, nullify callback, SDK unsubscribe method? | removeEventListener (standard DOM cleanup) |
| SDK readiness pattern | How does the CMP signal its SDK is ready? DOM event, callback array, global flag, Promise? | cf_init DOM event |
Detection paths
Every CMP source needs up to 3 detection paths:
| Path | Purpose | Questions to answer |
|---|---|---|
| Already loaded | CMP loaded before source | Is there a global object to check? What state does it expose? Is consent read from a property or an API method? |
| Init listener | CMP loads after source | What event/callback fires on SDK init? Is it a DOM event, callback assignment, or SDK readiness array? |
| Change listener | User updates consent | What event fires on consent change? Is it the same as init? |
Event registration patterns
Not all CMPs use addEventListener. Fill in "Event registration mechanism" in
the research template to determine which pattern applies:
| Pattern | CMPs | Registration | Cleanup |
|---|---|---|---|
| DOM events | CookieFirst, CookiePro, Cookiebot | addEventListener(name, handler) | removeEventListener(name, handler) |
| Callback assignment | CookiePro (OptanonWrapper) | window.OptanonWrapper = handler | window.OptanonWrapper = original |
| SDK readiness array | Didomi (didomiOnReady) | window.didomiOnReady.push(handler) | No unsubscribe (fires once) |
| SDK method | Didomi (on) | CMP.on('consent.changed', handler) | Vendor-specific (check docs) |
This affects the source skeleton (Phase 7), MockWindow shape (Phase 6), and destroy implementation.
Decision matrix
Fill in this matrix for your CMP. Reference implementations for comparison:
| Decision | CookieFirst | Usercentrics | CookiePro | Cookiebot |
|---|---|---|---|---|
| Already loaded? | window.CookieFirst.consent | Getter read at init (V3 __ucCmp.getConsentDetails(), V2 UC_UI.getServicesBaseInfo()) | window.OneTrust + window.OptanonActiveGroups | window.Cookiebot with hasResponse |
| Init listener | cf_init event | UC_UI_INITIALIZED event, then the getter | OptanonWrapper callback (self-unwrap) | None separate: the change events re-fire on every load with a stored choice |
| Change listener | cf_consent event | UC_UI_CMP_EVENT decisions (ACCEPT_ALL, DENY_ALL, SAVE), then the getter | OneTrustGroupsUpdated event | CookiebotOnAccept / CookiebotOnDecline events |
| Consent shape | Boolean map { category: bool } | Category states (V3 categories, V2 services grouped by categorySlug) | Comma-separated IDs ,C0001,C0003, | Boolean map plus method and stamp |
| Category naming | Human-readable | Admin-configured | Opaque IDs (C0001-C0005) | Fixed (necessary, preferences, statistics, marketing) |
| Explicit check | consent === null | Official consent metadata (V3 consent.type; V2 an EXPLICIT entry in service consent history) | IsAlertBoxClosed() | consent.method === 'explicit' |
| Default categoryMap | Populated (human names to walkerOS) | Empty (pass-through) | Populated (opaque IDs need mapping) | Populated, necessary left unmapped |
| Number of change events | 1 (cf_consent) | 1 (UC_UI_CMP_EVENT, which also fires for UI events that are no decision) | 2 (OptanonWrapper + OneTrustGroupsUpdated) | 2 (accept, decline), plus CookieConsentOn* aliases that are not used |
| Consent layers | Single (categories only) | Single (categories; services are read only to group them) | Single (categories only) | Single (categories only) |
| Consent access | Property (window.CookieFirst.consent) | API method (the getter; the event carries no consent) | Property (window.OptanonActiveGroups) | Property (window.Cookiebot.consent) |
| Event registration | addEventListener | addEventListener | Callback assignment + addEventListener | addEventListener |
Gate: Research complete
- All fields in the research template filled
- Detection paths identified (which of the 3 apply)
- Decision matrix row completed for the new CMP
- Official docs URL captured
Phase 2: Create examples
Goal: Define realistic consent data BEFORE writing implementation.
Required example inputs
Create at minimum these scenarios in src/examples/inputs.ts. See
inputs.ts for the generic template.
| Example | Purpose | Description |
|---|---|---|
fullConsent | All categories accepted | User clicked "Accept All" |
partialConsent | Some categories accepted | User customized consent |
minimalConsent | Only essential/necessary | User clicked "Deny All" or similar |
implicitConsent | Page-load defaults (if applicable) | CMP loaded but user hasn't chosen. Note: behavior varies by CMP: some grant nothing by default, others grant functional/necessary. Research the specific CMP's default consent state. |
noConsent | No consent data available | CMP hasn't loaded yet |
revocationInput | Consent withdrawal | User goes from full to partial |
Add CMP-specific edge cases:
- Case sensitivity: Uppercase/lowercase variants of consent fields
- Service-level consent: If CMP supports individual service booleans
- Custom categories: Non-standard category IDs
- Empty/malformed data: Edge cases for the consent shape
Required example outputs
Create expected walkerOS consent states in src/examples/outputs.ts. See
outputs.ts for the generic template.
Key rule: denied groups MUST have explicit false, not be omitted (see
mandatory check #1).
Example env
Create mock factories in src/examples/env.ts. See env.ts
for the generic template.
Step examples
src/examples/step.ts turns the inputs into Flow.StepExample entries: the CMP
input as in, a trigger that says how to deliver it, and the consent command
the source issues as out ([['elb', 'walker consent', state]]).
src/examples/trigger.ts exports the createTrigger that feeds them (see
create-source, Phase 3). The docs page,
walkeros push --simulate source.<id> and the step example tests read them;
packages/web/sources/cmps/usercentrics/src/examples/step.ts builds its
examples from its inputs.ts and outputs.ts.
Export via dev.ts
// src/dev.ts
export * as examples from './examples';
export * as schemas from './schemas';
schemas carries the settings schema (categoryMap, explicitOnly, ...):
buildDev() fails the build without it.
Gate: Examples valid
- All example files compile
- Can trace: each input -> expected output for each example
- Edge cases included (case sensitivity, empty data, revocation)
Phase 3: Define category mapping
Goal: Decide default categoryMap and document mapping rationale.
When to use a populated default map
Use a populated default when the CMP uses opaque or non-standard category names that are meaningless without mapping:
// CookiePro: opaque IDs require mapping
export const DEFAULT_CATEGORY_MAP: Record<string, string> = {
C0001: 'necessary', // Strictly Necessary, grants no consent group (check 11)
C0002: 'analytics', // Performance
C0003: 'functional', // Functional
C0004: 'marketing', // Targeting
C0005: 'social', // Social Media
};
When to use an empty default map
Use an empty default when the CMP uses human-readable, admin-configured category names that can pass through as-is:
// Usercentrics: admin-configured names pass through
const settings: Settings = {
categoryMap: config?.settings?.categoryMap ?? {},
};
Merging behavior
Custom entries merge with (and override) defaults:
const mergedCategoryMap = {
...DEFAULT_CATEGORY_MAP,
...(config?.settings?.categoryMap ?? {}),
};
Any deny denies for many-to-one mappings
When multiple CMP categories map to the same walkerOS group, the group is true
only if every one of them is true: any deny denies. All CMP sources follow
this rule. For a presence list (CookiePro), an inactive ID is a denial. Keep the
default map one category per group, so the rule only matters for a user's own
map.
// Any deny denies: once false, stays false
state[mapped] = state[mapped] === undefined ? value : state[mapped] && value;
Dual-layer consent (categories + services/vendors)
Some CMPs expose consent at multiple layers (e.g., Didomi: purposes + vendors). When the decision matrix shows "Consent layers: Dual," decide how to handle:
Option A: Map primary layer only (recommended for most cases). Use the
category/purpose layer and ignore the service/vendor layer. This matches
walkerOS's category-level WalkerOS.Consent model directly.
Option B: Expose both layers via settings. Add a setting like
consentLayer: 'categories' | 'services' and map whichever the user chooses.
Use this when the CMP's service-level consent is meaningfully different from its
category-level consent.
Document the chosen approach in the README under "How it works."
Gate: Mapping defined
- Default categoryMap chosen (populated or empty, with rationale)
- Any-deny-denies rule for many-to-one mappings documented
- Merge behavior with user overrides documented
- Dual-layer strategy decided (if applicable per decision matrix)
Phase 4: Scaffold
Goal: Create package structure mirroring CookieFirst.
Directory structure
packages/web/sources/cmps/[name]/
├── package.json
├── tsconfig.json
├── tsup.config.ts
├── jest.config.mjs
├── README.md
└── src/
├── index.ts # Main source export
├── dev.ts # Dev exports (examples, schemas)
├── schemas/
│ └── index.ts # Settings schema
├── types/
│ └── index.ts # Types, Settings, CMP API interface, declare global
├── examples/
│ ├── index.ts # Re-exports
│ ├── inputs.ts # CMP consent input examples
│ ├── outputs.ts # Expected walkerOS consent outputs
│ ├── step.ts # Step examples built from inputs and outputs
│ ├── trigger.ts # createTrigger
│ └── env.ts # Mock factories
└── __tests__/
├── index.test.ts # Full test suite
└── test-utils.ts # createMockElb, createMockWindow, createSource
package.json template
[version] is the version every @walkeros/* package currently has (see
packages/core/package.json); they are released together.
{
"name": "@walkeros/web-source-cmp-[name]",
"description": "[CMP Name] consent management source for walkerOS",
"version": "[version]",
"license": "MIT",
"walkerOS": {
"type": "source",
"platform": "web",
"docs": "https://www.walkeros.io/docs/sources/web/cmps/[name]"
},
"main": "./dist/index.js",
"module": "./dist/index.mjs",
"types": "./dist/index.d.ts",
"exports": {
".": {
"types": "./dist/index.d.ts",
"import": "./dist/index.mjs",
"require": "./dist/index.js"
},
"./examples": {
"types": "./dist/examples/index.d.ts",
"import": "./dist/examples/index.mjs",
"require": "./dist/examples/index.js"
},
"./dev": {
"types": "./dist/dev.d.ts",
"import": "./dist/dev.mjs",
"require": "./dist/dev.js"
},
"./walkerOS.json": "./dist/walkerOS.json"
},
"files": ["dist/**", "CHANGELOG.md"],
"scripts": {
"build": "tsup --silent",
"clean": "rm -rf .turbo && rm -rf dist",
"dev": "jest --watchAll --colors",
"lint": "eslint \"**/*.ts*\"",
"test": "jest",
"typecheck": "tsc --noEmit",
"update": "npx npm-check-updates -u && npm update"
},
"dependencies": {
"@walkeros/core": "[version]",
"@walkeros/collector": "[version]"
},
"repository": {
"url": "git+https://github.com/elbwalker/walkerOS.git",
"directory": "packages/web/sources/cmps/[name]"
},
"author": "elbwalker <hello@elbwalker.com>",
"keywords": [
"walker",
"walkerOS",
"walkerOS-source",
"source",
"web",
"[name]",
"consent",
"cmp"
]
}
Config files
Copy these exactly from CookieFirst, updating only the globalName:
tsconfig.json- extends@walkeros/config/tsconfig/web.jsontsup.config.ts- usesbuildModules,buildExamples,buildBrowser,buildES5andbuildDevjest.config.mjs- extends@walkeros/config/jest/web.config
Phase 5: walkerOS.json convention
Add walkerOS field to package.json
{ "walkerOS": { "type": "source", "platform": "web" } }
Use buildDev() in tsup.config.ts
Use the standard buildDev() helper from @walkeros/config/tsup (consistent
with create-source and create-destination skills):
import { buildDev } from '@walkeros/config/tsup';
// In defineConfig array:
buildDev(),
The CookieFirst tsup.config.ts already has it.
Hints (Optional)
If your CMP source has non-obvious behaviors or troubleshooting patterns, add
hints. See walkeros-writing-documentation skill for full guidelines and
walkeros-create-source skill for the hints pattern.
Gate: Convention met
-
walkerOSfield in package.json with type and platform - Dev build configured in tsup.config.ts
- Keywords include
walkerOSandwalkerOS-source
Phase 6: Test (TDD: write tests BEFORE implementation)
Goal: Write the full test suite first. Watch it fail. Then implement.
Test structure (8-9 describe blocks)
| Describe block | Tests | What it covers |
|---|---|---|
initialization | 4-6 | No errors, correct type, default settings, custom settings, listener registration |
explicit consent filtering | 3-4 | Explicit events processed, implicit ignored/processed based on setting, case-insensitive type |
non-consent event filtering | 2 | Non-consent events ignored, events without detail ignored |
category mapping | 5-8 | Full/partial/minimal consent, custom mapping, unmapped categories, any deny denies |
[CMP-specific parsing] | 2-4 | CMP-specific consent format parsing (service-level, string parsing, etc.) |
event handling | 3 | Consent changes, multiple changes, revocation |
consent revocation | 2 | Full->partial, full->minimal (explicit false values verified) |
cleanup | 2 | Destroy removes listeners, restores wrapped functions |
no window environment | 1 | Handles missing window gracefully |
Test utilities pattern (MockWindow)
See test-utils.ts for the complete template
including MockWindow interface, createMockElb, createMockWindow, and
createCmpSource factories.
For CMPs that use callback assignment or SDK methods instead of
addEventListener (e.g., Didomi's onReady, CookiePro's OptanonWrapper),
adapt MockWindow to expose those callbacks as testable properties.
Test template
Use the test template: index.test.ts
Gate: Tests fail for the right reason
- Tests fail with "Cannot find module '../index'" (module doesn't exist yet)
- Tests use example outputs for assertions (not hardcoded values)
- Consent revocation test included (full grant -> revoke -> verify explicit
false)
Phase 7: Implement
Goal: Wire up detection paths, handleConsent, and destroy. Make tests pass.
v4 source identity
CMP sources register their type literal via SourceMap in src/types.ts.
CMPs are web by default - platform is optional in the augmentation:
import type { Source } from '@walkeros/core';
declare module '@walkeros/core' {
interface SourceMap {
// Replace with the CMP's package-level identifier.
cookiefirst: {
type: 'cookiefirst';
platform?: 'web'; // optional: CMP runs on web by default
};
}
}
The source returns { type: 'cookiefirst', config, push, init, destroy } from
Source.Init; the type literal must match the SourceMap key.
CMP sources do NOT set source.url or source.referrer. A CMP only knows
about consent state, not the page that produced it; the browser, dataLayer and
session sources set the page context of the events they push. Don't reach for
window.location in CMP source code, leave those fields untouched.
Settings interface pattern
See types.ts for the complete type definitions
including Settings, InitSettings, Types bundle, CMP API interfaces, and
declare global window augmentation.
The core settings of a CMP source:
categoryMap?: Record<string, string>: maps CMP categories to walkerOS consent groups (every CMP source)explicitOnly?: boolean: for a CMP that reports implicit or default consent before the visitor decides (Usercentrics, CookiePro, Cookiebot). Defaulttrue: the source reports only explicit choices, so a visitor the CMP never asks sends no consent;falsealso reports implicit or default consent. CookieFirst reports consent only after a choice (consentisnullbefore), so its source has no such setting.globalName?: string: only where the CMP lets a site rename its global; Cookiebot hard-codeswindow.Cookiebot, so its source has no such setting
Source skeleton
Every CMP source follows this skeleton with 5-6 decision points. See index.ts for the complete template.
Key implementation steps:
- Resolve window (
env.windowfallback toglobalThis) - Merge settings with defaults
- Track listener references for cleanup
handleConsentfunction (explicitOnly, categoryMap with any deny denies,elb('walker consent', state))- Detection path: Already loaded
- Detection path: Init listener
- Detection path: Change listener
The factory MUST be side-effect-free; do the detection paths in init().
Steps 5-7 attach listeners and perform the "already loaded" static consent read,
which emits elb('walker consent', state). Put them inside an init() method
on the returned instance, NOT in the factory body. The collector runs the
factory in Pass 1 (before all sources are merged) and init() in Pass 2.
Emitting consent from the factory races source merge order and can leave a later
require:["consent"] source parked. Return
{ type, config, push, init, destroy }; init runs the static read + attaches
listeners, destroy removes them. (The collector also matches require against
current recorded state, so a consent read from init() still activates
dependent steps regardless of order, but a side-effect-free factory is the
contract.)
Note on init listeners: Some init listeners (like CookieFirst's cf_init)
read consent from the global window object, not from event.detail. Others
(like cf_consent) receive consent via event.detail. Check which pattern your
CMP uses for each detection path.
Gate: All tests pass
-
npm run verify:touched -- web-source-cmp-<name>passes (L1: typecheck + lint + test; the argument is the package name after@walkeros/) -
npm run buildpasses
Phase 8: Document
README structure
Keep it short and point to the docs page, like the existing CMP sources (sentence-case headings, imports in code examples):
<logo>
# @walkeros/web-source-cmp-[name]
[One line: what the source integrates and how it maps categories.]
Documentation | NPM Package | Source Code
## Installation
## Quick start (startFlow with imports)
## Documentation (link to the docs page)
## Contribute
## License
Create the docs page
Create the docs page at website/docs/sources/web/cmps/[cmp-name]/index.mdx,
following the CookiePro page
(website/docs/sources/web/cmps/cookiepro/index.mdx): installation with
imports, the <Configuration> block fed by the package's walkerOS.json, how
it works, the default category mapping, a custom mapping example and timing
considerations. Then register the page:
- Add it to the CMPs category in
website/sidebars.ts - Add a row to the table in
website/docs/sources/web/cmps/index.mdx, and name the source where that page andwebsite/docs/guides/consent/index.mdxcompare the CMP sources - Add the package to
website/package.jsonso the page can import itswalkerOS.json
Mandatory checks
These are non-negotiable patterns every CMP source MUST follow. Violating any of these is a privacy compliance issue or a correctness bug.
1. Ensure every consent update includes explicit false for denied groups
The collector uses merge semantics (assign()), so omitting a key means "no
change," NOT "denied." Every consent state passed to
elb('walker consent', state) must include explicit false for denied groups,
not just true for granted ones.
Boolean-map CMPs (CookieFirst, Usercentrics group-level): The CMP's consent
object already contains explicit false values (e.g.,
{ marketing: false, functional: true }). These flow through naturally via
iteration, no extra code needed.
Presence-based CMPs (CookiePro): Only active groups are listed (e.g.,
",C0001,C0003,"). Absence means denied. You MUST give EVERY mapped group an
explicit value, true only when every ID mapped to it is active (any deny
denies):
// Presence-based CMPs: walk the map, not the active IDs
const active = new Set(activeIds.map((id) => id.toLowerCase()));
Object.entries(normalizedMap).forEach(([id, group]) => {
const granted = active.has(id);
state[group] = state[group] === undefined ? granted : state[group] && granted;
});
2. Case-insensitive comparison for all CMP string fields
Use .toLowerCase(). CMP docs are inconsistent on casing.
// Explicit type check
if (settings.explicitOnly && detail.type?.toLowerCase() !== 'explicit') return;
// Category ID lookup
const mapped = normalizedMap[id.toLowerCase()];
3. Apply categoryMap consistently in ALL code paths
If there are multiple parsing branches (group-level vs service-level, or already-loaded vs event-listener), mapping MUST work identically in each.
4. Prevent dual-firing on consent events
Many CMPs fire multiple signals for the same consent action. If you don't guard
against this, handleConsent fires twice per user action. Three known patterns:
The collector's exactly-once delivery does not solve this for you: it guarantees each subscriber sees a given state version once, but two upstream CMP signals (a callback plus a DOM event) are two distinct consent commands, so both are delivered legitimately. Coalescing duplicate upstream signals is the source's job, as below.
Pattern A: Callback + event (CookiePro) CMP fires both a callback
(OptanonWrapper) and a DOM event (OneTrustGroupsUpdated) on the same action.
Use the callback for init only and self-unwrap after first call:
actualWindow.OptanonWrapper = () => {
if (originalWrapper) originalWrapper();
handleConsent();
actualWindow.OptanonWrapper = originalWrapper; // Self-unwrap
};
Pattern B: Change events that re-fire on load (Cookiebot) CMP fires one of
several events per decision (CookiebotOnAccept, CookiebotOnDecline), and
fires it again on every page load with a stored choice. Register the SAME
handler for all of them, and listen to one name set only: Cookiebot also
dispatches each event under a CookieConsentOn* alias. Because the static read
at init and the load-time event report the same choice, drop a state identical
to the last one the source sent.
Pattern C: One event bus for every UI event (Usercentrics) Usercentrics
fires UC_UI_CMP_EVENT for decisions and for UI events such as a view change.
Handle only the decision types (ACCEPT_ALL, DENY_ALL, SAVE) from the event
detail, and read the consent from the getter (the event carries none). A view
toggle then publishes nothing.
Research step: During Phase 1, fill in "Number of change events" in the decision matrix. If >1, determine which pattern applies and plan accordingly.
5. Document timing/race conditions in the README
What if the CMP loads before the source? What about explicitOnly: false? Each
CMP has different timing behavior: document it explicitly under "Timing
considerations."
For "previous-choice" detection (was this an active user decision or a
first-visit default?), read the CMP's official consent metadata, the consent
type or decision history (e.g. Usercentrics V3 consent.type, V2 the service
consent history), not a per-pageload event type field. A per-pageload event
type conflates first-visit defaults with a returning visitor's restored
choice, so a returning visitor's prior consent would be dropped under
explicitOnly.
6. Test consent revocation end-to-end
Full grant -> revoke -> verify denied (explicit false values). This is the
most common source of bugs.
test('handles consent withdrawal (full to partial)', async () => {
const mockWindow = createMockWindow(inputs.fullConsent);
await createCmpSource(mockWindow, mockElb);
// Initial: full consent, read at init
expect(consentCalls[0].consent).toEqual(outputs.fullConsentMapped);
// User revokes marketing and analytics
mockWindow.__dispatchEvent('cmp_consent', inputs.revocationInput);
expect(consentCalls[1].consent.marketing).toBe(false); // explicit false
expect(consentCalls[1].consent.analytics).toBe(false); // explicit false
});
This is the revocation test of the template
(index.test.ts); cmp_consent stands for your
CMP's change event.
7. Use MockWindow interface in tests
Properly typed, not as unknown as casts scattered through tests. Define one
MockWindow interface in test-utils.ts that extends the source's narrowed
CmpWindow (the surface the source actually touches), not the global Window.
Because Env.window is narrowed to CmpWindow (as the template and the
Cookiebot source do), the mock literal satisfies it directly, with no
as unknown as Window cast.
8. Store category keys in user-expected format
Normalize during init for case-insensitive lookup, but store the original keys in the config so users see what they configured.
// Store original casing in config
const mergedCategoryMap = { ...DEFAULT_CATEGORY_MAP, ...userMap };
// Build normalized lookup for internal use
const normalizedMap: Record<string, string> = {};
Object.entries(mergedCategoryMap).forEach(([key, value]) => {
normalizedMap[key.toLowerCase()] = value;
});
9. walkerOS.json convention
Add "walkerOS": { "type": "source", "platform": "web" } to package.json.
10. README requirements
Keep the README short, as the existing CMP sources do: logo, one-line description, Documentation/NPM/Source Code links, installation, a quick start with imports, a link to the docs page, contribute and license sections, sentence case headings. Configuration, mapping, how it works and timing considerations belong on the docs page (see "Create the docs page").
11. Strictly necessary cookies grant no consent group
Strictly necessary cookies need no consent, and the CMP grants them with every
choice. Never map such a category (necessary, C0001, an essential category)
to a consent group like functional in the default categoryMap: as the only
category of that group, it would grant the group with every choice, and every
destination that requires the group would act on consent the visitor never gave.
Report it under its own name (necessary) or leave it out, and cover it with a
test: a choice that grants only strictly necessary cookies leaves every group
except necessary false.
Validation checklist
Beyond
understanding-development
requirements (build, test, lint, no any):
- All 11 mandatory checks pass
- Research template fully filled
- Decision matrix row complete
- Examples include revocation and edge cases
- Tests: 25-32 tests across 8-9 describe blocks
-
MockWindowinterface in test-utils (not scattered casts) - Category mapping: any deny denies for many-to-one
- Consent state always includes explicit
falsefor denied groups -
destroy()cleans up ALL listeners and restores wrapped functions - Detection never fails silently: a setting the source needs that is missing
throws a
StepError(code: 'config') frominit, andinitnever returns early or returnsfalse; a CMP that is simply absent is not an error (consent stays unknown) - README follows required structure
- Docs page created and registered (Phase 8)
Known limitations
The skill's source skeleton and code templates are based on DOM-event CMPs (CookieFirst, Usercentrics, CookiePro). CMPs that deviate significantly from this pattern will require adaptation:
| Limitation | Affected CMPs | Workaround |
|---|---|---|
Source skeleton assumes addEventListener | Didomi (SDK array/method) | Use the "Event registration patterns" table in Phase 1 to identify the correct pattern, then adapt the skeleton's listener setup and destroy() accordingly. |
| Consent read via property, not API method | Didomi (getCurrentUserStatus()) | If consent is accessed via an API method call rather than a window property, wrap the call in handleConsent and adjust the "already loaded" detection path. |
destroy() may not be possible | CMPs with SDK readiness arrays (fire-once, no unsubscribe) | Document in README that the init callback cannot be removed. Only the change listener needs cleanup. |
| No vendor-level consent model | Didomi (purposes + vendors as separate consent layers) | Use the "Dual-layer consent" guidance in Phase 3. walkerOS Consent is category-level; vendor-level consent requires flattening or a consentLayer setting. |
These are research-phase decisions; the skill's phases, mandatory checks, and validation checklist still apply. The research template and decision matrix capture these variations so they are identified early.
Reference files
| What | Where |
|---|---|
| Skill examples | examples/ |
| Skill templates | templates/cmp/ |
| Canonical template | packages/web/sources/cmps/cookiefirst/ |
| Source types | packages/core/src/types/source.ts |
| Consent guide | website/docs/guides/consent/ |
| Usercentrics source | packages/web/sources/cmps/usercentrics/ |
| CookiePro source | packages/web/sources/cmps/cookiepro/ |
Related skills
- walkeros-create-source - General source creation workflow
- walkeros-understanding-sources - Source interface and push pattern
- walkeros-testing-strategy - Testing with env mocking
- walkeros-writing-documentation - Documentation standards
Create a New Destination
Prerequisites
Before starting, read these skills:
- understanding-flow - How destinations fit in architecture
- understanding-destinations - Destination interface
- understanding-transformers - Transformer chaining to destinations
- understanding-mapping - Event transformation
- testing-strategy - How to test with env pattern (Phase 3 + Phase 8)
- using-step-examples -
Authoritative pattern for
Flow.StepExamplestructure and Three Type Zones - writing-documentation - Documentation standards (for Phase 9)
Choose Your Template
| Complexity | Template | When to Use |
|---|---|---|
| Simple | plausible/ | Single SDK call, minimal config |
| Complex | gtag/ | Multiple services, sub-destinations |
| Server | gcp/ | Server-side, batching, SDK init |
Process Overview
1. Research → Deeply understand vendor SDK, API, and event taxonomy
2. Classify → Determine vendor taxonomy type and integration approach
3. Examples → Define in/out pairs FIRST (start with the end result)
4. Mapping → Define walkerOS → vendor transformation
5. Scaffold → Copy template and configure
6. Convention → Add walkerOS.json metadata and buildDev
7. Implement → Build using examples as test fixtures
8. Test → Verify against example variations
9. Document → Write README
Phase 1: Research
Goal: Deeply understand the vendor SDK before writing any code. Research quality determines implementation quality.
1.1 Find and Install Official SDK
Always prefer the vendor's official SDK package over raw HTTP API calls. The SDK handles transport, batching, retries, and plugin ecosystems; don't reinvent these.
- Install the SDK:
npm install @vendor/sdkand read the actual source - Read TypeScript types: Import types from the SDK package directly. Never duplicate type definitions. This ensures IntelliSense completeness and consistency with SDK updates.
- Understand the full API surface: List every public method, not just
track(). Identity methods, property operations, group management, specialized event types (revenue, etc.)
# Search npm for official packages
npm search [vendor-name]
npm search @[vendor]
# Install and inspect actual types
npm install @vendor/analytics-browser
ls node_modules/@vendor/analytics-browser/lib/esm/
1.2 Understand SDK Architecture
- Init options: What does
init()accept? What can be configured? - Call ordering: Can you call methods before
init()? Is there internal queuing? What are the race condition implications? - Plugin system: Does the SDK support plugins? How are they added?
- Identity management: How does the SDK handle user/device/session IDs? Does it manage cookies/storage? Can this be disabled (since walkerOS manages identity)?
- Consent: Does the SDK have a consent mode or just binary opt-out?
1.3 Identify All Event Methods
Go beyond just track(). Most SDKs have specialized methods:
| Method Category | Examples | walkerOS Handling |
|---|---|---|
| Event tracking | track(), logEvent() | Default push() |
| User properties | identify(), setUserProperties() | mapping.settings |
| Revenue/purchase | revenue(), purchase() | mapping.settings |
| Groups/accounts | setGroup(), groupIdentify() | mapping.settings |
| Identity setters | setUserId(), setDeviceId() | settings.identify |
| Opt-out | setOptOut(), consent() | on('consent') |
| Cleanup | flush(), reset() | destroy() |
1.4 Check Existing Patterns
Review similar destinations in the codebase:
ls packages/web/destinations/
Gate: Research Complete
- Official SDK installed and types inspected
- All public methods listed with their purpose
- Init options understood (what to disable, what to pass through)
- Call ordering / race conditions understood
- Identity, consent, and plugin patterns documented
Checkpoint: Research Review (Optional)
If working with human oversight, pause here to confirm:
- SDK integration approach correct?
- Event mapping makes sense for the use case?
- Any vendor quirks or limitations to handle?
Phase 2: Classify Vendor Taxonomy
Goal: Understand what the vendor expects, which determines destination complexity.
2.1 Taxonomy Types
| Type | Description | Mapping Needed | Example Vendors |
|---|---|---|---|
| Free-form | Any event name accepted, no prescribed schema | Minimal: names pass through | Most modern analytics |
| Rigid | Prescribed event names unlock specific reports | Essential: must map to exact names | Some legacy analytics |
| Schema-based | Self-describing events with formal schemas | Structural: must build schema objects | Data warehouse tools |
2.2 Determine Integration Approach
| Approach | When to use | Pattern |
|---|---|---|
| Vendor SDK as host | SDK has plugins, batching, retries | Load SDK, disable what walkerOS replaces, use as transport |
| Script + command queue | SDK loaded via script tag | Load script, use global function queue |
| HTTP API | No SDK available, or server-side | Direct HTTP calls via sendWeb() or fetch |
Prefer the vendor SDK: it handles transport, retries, and plugin orchestration. HTTP API is a fallback when no SDK exists.
When using the vendor SDK:
- Disable features walkerOS replaces (autocapture, identity storage) by default, but allow users to re-enable via settings
- Pass through all SDK init options for full configurability
- Import types from the SDK package directly
Gate: Classification Complete
- Taxonomy type identified (free-form / rigid / schema-based)
- Integration approach chosen (SDK / script / HTTP)
- Know which SDK features walkerOS replaces vs. uses
Phase 3: Create Examples (BEFORE Implementation)
Mandatory. Examples are the test fixtures for Phase 8. Define expected in
/ out / mapping triples FIRST: start with the end result in mind. Without
examples, you cannot test. Even for free-form vendors where no mapping is
strictly "required," step examples serve as the single source of truth for
tests, simulations, and documentation.
Authoritative pattern: See using-step-examples for the Three Type Zones and lifecycle. This skill reuses that contract; do not diverge.
3.1 Scaffold Directory Structure
mkdir -p packages/web/destinations/[name]/src/examples
mkdir -p packages/web/destinations/[name]/src/{schemas,types}
3.2 Required Files (4 files total)
The reference web destinations (meta, snowplow, plausible, piwikpro, api) use
exactly four files in src/examples/. Match this structure: no events.ts,
outputs.ts, or standalone mapping.ts.
| File | Purpose |
|---|---|
examples/env.ts | Mock environment for testing (no real network calls) |
examples/step.ts | Flow.StepExample entries with in / out / mapping? triples |
examples/minimal.ts | Flow.MinimalExample: only the settings it needs to start |
examples/index.ts | Barrel exports: env, step and minimal |
The step.ts file embeds the input event, the mapping config, and the expected
vendor output together in one Flow.StepExample.
The skill's examples/ folder holds these four files for the
template destination (templates/simple/): copy them to
src/examples/.
3.3 Typing Rules (strict)
No any. Every example value must be explicitly typed.
- Inputs use
WalkerOS.Event(viagetEvent()from@walkeros/core); never hand-roll event literals. - Outputs use vendor SDK types imported from the official package
whenever the vendor publishes them. Do not invent local output types for
payloads the vendor already types (e.g. use Meta Pixel's
fbqargument types, not a localFbqCallinterface). - Step entries are typed
Flow.StepExamplefrom@walkeros/core. - Mock env is typed against the destination's local
Envtype from../types. Noas any, no untyped{}. - Vendor SDK types come from the SDK you installed in Phase 1: reuse them rather than duplicating shapes.
3.4 Code Template: examples/step.ts
The examples match the template destination
(index.ts), which calls a vendor global
window.vendorSdk('track', name, params):
import type { Flow } from '@walkeros/core';
import { getEvent } from '@walkeros/core';
// One step example per supported feature / setting.
// `in` is a WalkerOS.Event (getEvent returns the same event on every run).
// `out` lists the vendor calls the destination makes, as [callable, ...args]
// tuples written from the vendor's documented calls.
// `mapping` is the mapping rule under test (optional: omit it for the default
// push).
// Set `title` + `description` for public examples; mark test-only fixtures
// with `public: false`. See the walkeros-using-step-examples skill.
export const init: Flow.StepExample = {
title: 'Initialization',
description:
'The destination loads the vendor script and initializes the SDK with the API key.',
in: { loadScript: true, settings: { apiKey: 'YOUR_API_KEY' } },
out: [['window.vendorSdk', 'init', 'YOUR_API_KEY']],
};
export const purchase: Flow.StepExample = {
title: 'Purchase',
description:
'A completed order is sent as a purchase with its id, value and currency.',
in: getEvent('order complete'),
mapping: {
name: 'purchase',
data: {
map: {
transaction_id: 'data.id',
value: 'data.value',
currency: { key: 'data.currency', value: 'EUR' },
},
},
},
out: [
[
'window.vendorSdk',
'track',
'purchase',
{ transaction_id: '0rd3r1d', value: 462, currency: 'EUR' },
],
],
};
export const pageView: Flow.StepExample = {
title: 'Page view',
description: 'A page view without a rule is sent under its walkerOS name.',
in: getEvent('page view'),
out: [['window.vendorSdk', 'track', 'page view', {}]],
};
A consent example sets command: 'consent', which routes in through
elb('walker consent', in) instead of an event push (see consentRevoke
below).
Init step example
A destination whose init makes vendor calls (script tags, SDK initializers,
queue setup) can ship them as an examples.step.init entry, as plausible, meta
and snowplow do.
inmirrors the realDestination.Configshape users copy-paste, typically{ loadScript: true, settings: { /* vendor-specific */ } }. Whatever a user would configure in their flow goes here verbatim.outis the ordered list of vendor calls theinit()lifecycle produces. Each effect tuple follows the standard[callable, ...args]shape.
runExamples keeps init calls out of every event example's out and takes
events only, so the init example runs in a test of its own: start a real
collector, call destination.init(...) with an env that records the paths
examples.env.simulation names, and compare the calls with init.out. The
template index.test.ts does this, as
packages/web/destinations/meta/src/__tests__/conformance.test.ts does.
Multi-tool packages (like gtag, which drives GA4, Google Ads, and GTM)
ship per-tool init examples (examples.step.ga4Init, adsInit, gtmInit)
instead of a single init, and their tests run each with its sub-tool's
settings.
Failure, silent and consent examples
Besides the happy path, step.ts carries public: false examples that the
conformance kit and runExamples (Phase 8) run on a real collector. The first
and the last fit the template destination; the two in between show failures
other destinations report:
// A silent rule sends nothing.
export const silentRule: Flow.StepExample = {
public: false,
description: 'A rule with silent: true sends no event.',
in: getEvent('page view'),
mapping: { silent: true },
out: [],
};
// A destination that needs a user id reports an event without one.
export const missingInput: Flow.StepExample = {
public: false,
description: 'Without a user id the push fails with missing-input.',
in: getEvent('page view', { user: { device: 'c00k13' } }),
mapping: { settings: { identify: { map: { id: 'user.id' } } } },
out: [],
outcome: { status: 'failed', code: 'missing-input', retryable: false },
};
// An HTTP destination whose vendor answers 200 with an error in the body.
export const rejectedByVendor: Flow.StepExample = {
public: false,
description: 'The vendor answers 200 and reports the event as failed.',
in: getEvent('page view'),
respond: { status: 200, body: { events: [{ status: 'failed' }] } },
outcome: { status: 'failed', code: 'transport', retryable: false },
};
// A consent revocation reaches the vendor.
export const consentRevoke: Flow.StepExample = {
public: false,
description:
'For a destination with config.consent { marketing: true }, revoking marketing tells the vendor to revoke.',
command: 'consent',
in: { marketing: false },
out: [['window.vendorSdk', 'consent', 'revoke']],
};
outcome is { status: 'failed' | 'queued', code, retryable? }; respond is
what the vendor's boundary answers (status, body, or
error: 'network' | 'timeout'). An example that expects no vendor call has
out: []. A "one of these settings" rule (ga4, ads or gtm) is stated as
anyOf in the settings schema, not only in init.
For destinations, the Three Type Zones collapse to:
in= walkerOS event (WalkerOS.Event)out= vendor output (typed against vendor SDK)mapping= rule under test (optional)
3.5 examples/index.ts (barrel)
export * as env from './env';
export * as step from './step';
export { minimal } from './minimal';
3.6 examples/env.ts
Mock the vendor SDK surface and any DOM touchpoints. Never reach real network,
real cookies, or real globals. Type the exports against your local Env, and
name the calls a simulation records in simulation:
import { mockScriptDocument } from '@walkeros/web-core/dev';
import type { Env } from '../types';
const noop = () => {};
/** Before init: no SDK global yet, `init` installs the stub. */
export const init: Env = {
window: {},
document: mockScriptDocument(),
};
/** After init: the SDK global the push calls. */
export const push: Env = {
window: { vendorSdk: Object.assign(noop, { q: [] as IArguments[] }) },
document: mockScriptDocument(),
};
/** The calls a simulation records. */
export const simulation = ['call:window.vendorSdk'];
3.7 examples/minimal.ts
The minimal example is the setup a reader copies: the docs page renders its
install line, integrated code and flow.json under ## Installation from it
(<Quickstart data={data} />), and the conformance test runs exactly those
settings and requires one public example delivered.
import type { Flow } from '@walkeros/core';
/** The settings the vendor needs to deliver: the API key, with the vendor script loaded by the destination. */
export const minimal: Flow.MinimalExample = {
settings: { apiKey: 'YOUR_API_KEY' },
config: { loadScript: true },
};
settings: only the keys the settings schema requires, plus any thatinitrefuses to start without under the example'sconfig(Plausible'sdomainwithloadScript: true), with placeholder values. Other keys stay in the step examples that need them; the kit fails a minimal example that sets one.config: only{ loadScript: true }, for a web destination that loads its vendor script only when asked to; the kit fails a script-loaded destination without it.secrets(server destinations): each credential setting mapped to its secret name,<VENDOR>_<SETTING>(for example{ accessToken: 'META_ACCESS_TOKEN' }). The generated code readsprocess.env.META_ACCESS_TOKENand$secret.META_ACCESS_TOKEN; a web destination holds no secret.- A package with several exports keeps one per export, in the export's folder
(
src/firehose/examples/minimal.ts), listed inexportExamples.
A package that cannot express its setup in one minimal example (one package
serving several docs pages, an install line with a driver choice) passes
withoutMinimal: '<reason>' to conformance instead, and its page keeps a
hand-written setup.
3.8 Test Fixture Contract (hard rule)
The examples authored here are the Phase 8 test fixtures. No parallel fixtures allowed.
- Every event example runs through
runExamplesfrom@walkeros/collector/devwithclaims(the standard events the destination maps): it runs each one on a real collector aswalkerOS.jsonpublishes it and compares itsoutand itsoutcome. Do not hand-roll that loop. - Tests must NOT contain hardcoded payloads, vendor configs, or expected outputs.
- If a test needs a value that is not in
examples.step, add it tostep.tsfirst, then consume it from the test. Never inline test data. - The only per-test setup allowed is the settings a group of examples needs (e.g. enabling the right sub-tool).
See packages/web/destinations/meta/src/__tests__/conformance.test.ts for a
canonical reference.
3.9 Export via dev.ts
export * as schemas from './schemas';
export * as examples from './examples';
Phase 3 Acceptance Checklist
-
src/examples/env.ts: mock env, no real network, typed against localEnv -
src/examples/step.ts: oneFlow.StepExampleper supported feature / setting, typedin/out/mapping? -
src/examples/minimal.ts:Flow.MinimalExamplewith only the settings the schema requires orinitrefuses to start without (secretsfor server credentials,config.loadScriptfor a script-loaded web destination) -
src/examples/index.ts: barrel exportsenv,stepandminimal - No standalone
events.ts,outputs.ts, ormapping.tsfiles - All vendor SDK types imported from the official package, no
any, no reinvented local output types -
src/__tests__/conformance.test.tsruns the event examples throughrunExampleswithclaims, and a test runs the init example (if any) - Tests contain zero hardcoded payloads / vendor configs / expected outputs:
everything flows from
examples.step -
npm run buildpasses: examples compile against published types - Each example traces:
in→ applymapping→ matchesout
Phase 4: Define Mapping
Goal: Document transformation from walkerOS events to vendor format.
Mapping rules live inside each Flow.StepExample entry in step.ts, no
separate mapping.ts file. Each step example embeds the exact mapping rule
under test alongside its in event and expected out output.
Verify Mapping Logic
For each entry in step.ts, trace:
Input: examples.step.purchase.in (WalkerOS.Event)
↓ Apply examples.step.purchase.mapping
↓ name transforms, data.map applied
Output: Should match examples.step.purchase.out
Gate: Mapping Verified
- Step examples cover: page view + at least one conversion event
- One step example per supported setting / sub-tool
- Each
mappingtraces correctly fromintoout
Phase 5: Scaffold
Template destination: packages/web/destinations/plausible/
cp -r packages/web/destinations/plausible packages/web/destinations/[name]
cd packages/web/destinations/[name]
# Update package.json: name, description, repository.directory
Directory structure:
packages/web/destinations/[name]/
├── src/
│ ├── index.ts # Main destination (init + push)
│ ├── index.test.ts # The init example and other boundary tests
│ ├── __tests__/ # conformance.test.ts: conformance + runExamples
│ ├── dev.ts # Exports schemas and examples
│ ├── examples/ # env.ts, step.ts, minimal.ts, index.ts
│ ├── schemas/
│ └── types/
├── package.json
├── tsconfig.json
├── tsup.config.ts
├── jest.config.mjs
└── README.md
Transformer Chain Integration
Destinations can wire to transformer chains via before in the init config:
destinations: {
myDestination: {
code: destinationMyDestination,
config: { settings: { /* ... */ } },
before: 'redact' // Events go through redactor before this destination
}
}
Phase 6: walkerOS.json Convention
Every walkerOS package ships a walkerOS.json file for CDN-based schema
discovery.
Add walkerOS field to package.json
{
"walkerOS": { "type": "destination", "platform": "web" },
"keywords": ["walkerOS", "walkerOS-destination", ...]
}
Use buildDev() in tsup.config.ts
Keep buildDev() in the defineConfig array (the template package's
tsup.config.ts has it):
import { buildDev } from '@walkeros/config/tsup';
// In defineConfig array:
buildDev(),
It builds src/dev.ts and writes dist/walkerOS.json from the schemas,
examples and hints it exports.
Hints (Optional)
If your destination has capabilities, behaviors, or troubleshooting patterns not
obvious from schemas alone, add hints. See walkeros-writing-documentation
skill for full guidelines.
Create src/hints.ts:
import type { Hint } from '@walkeros/core';
export const hints: Hint.Hints = {
'auth-methods': {
text: 'Supports X, Y, and Z auth methods. See settings schema for all options.',
code: [{ lang: 'json', code: '{ "settings": { ... } }' }],
},
};
Export from src/dev.ts:
export * as schemas from './schemas';
export * as examples from './examples';
export { hints } from './hints';
Guidelines:
- Expand awareness: describe capabilities ("supports X, Y, Z"), don't prescribe one path
- Reference schemas and examples, don't duplicate them
- Verify every claim against actual implementation before publishing
- Key naming: kebab-case, group with prefixes (auth-*, storage-*, troubleshoot-*)
- Most destinations don't need hints; schemas and examples cover the common case
Gate: Convention Met
-
walkerOSfield in package.json with type and platform -
buildDev()in tsup.config.ts - Build generates
dist/walkerOS.json - Keywords include
walkerOSandwalkerOS-destination
Phase 7: Implement
Now write code to produce the outputs defined in Phase 3.
Template Files
Use these templates as your starting point:
| File | Purpose | Template |
|---|---|---|
types/index.ts | Type definitions | types.ts |
index.ts | Main destination | index.ts |
index.test.ts | Init example test | index.test.ts |
__tests__/conformance.test.ts | Failure contract, all examples | conformance.test.ts |
Key Patterns
-
Init receives context: Destructure
config,env,logger,idfrom context -
Push receives context: Includes
data(the mapped data),rule(the matching mapping rule),ingest -
Use
getEnv<Env>(env): Never accesswindow/documentdirectly, and never cast them. Declare your SDK global (declare global { interface Window { vendorSdk?: ... } }) and narrowEnv['window']to the SDK shape intypes.ts. The genericgetEnv<Env>(env)then returns your narrowed env intersected with the real DOM globals, sowindow.vendorSdkanddocument.createElementare fully typed with zero casts. Noas any,as unknown,as Window,as Document,@ts-ignore, or@ts-expect-erroris allowed in production or test code, ever. If something will not type check, fix the types, do not cast. -
Return config from init: Allows updating config during initialization
-
Optional
destroymethod: Implement if the destination holds resources (DB connections, SDK clients, timers) that need cleanup on shutdown. Callflush()or equivalent on the vendor SDK. -
Mapping-based settings (recommended): Use standard walkerOS mapping values (
map,loop,key,value,condition) inmapping.settings.*for vendor-specific operations. Resolve viagetMappingValue(event, value, { collector })inpush(), with thecollectorfrom the push context, and interpret the resolved object's keys as SDK method instructions. This keeps config agnostic and reuses the mapping engine. -
Consent two-layer:
config.consentgates walkerOS event delivery.on('consent')controls vendor SDK internals (opt-out, pause capture, etc.). Both needed for complete consent compliance.on()may be async: the collector awaits it, bounded byconfig.timeout, and delivers the current state before the destination's first push; events pushed meanwhile answerqueuedwith the reasonheld. A state handler that rejects or does not settle in that time writes an ERROR line (consent not applied) and holds the destination's events for one moreconfig.timeout. The first push after that window retries a delivery whose handler rejected, and once a delivery succeeds the held events are delivered. Until one has succeeded, the held events and every later one fail withblocked(the push answersok: false, the events go to the dead-letter queue, an ERRORPush failedline is written) and are not sent when the destination recovers; later pushes retry a rejected delivery once perconfig.timeout. A handler that timed out gets no new delivery until it settles, so one that never settles keeps the destination blocked. Awalker rundiscards events that are still held, with no counter and no log line. Lifecycle handlers such ason('run')are awaited too, bounded by the same timeout, and never hold events.elbcommands andstartFlowresolve only after the destinations'on()handlers for that command have settled or timed out. See consent for the full sequence.- For step-example tests, use
command: 'consent'onFlow.StepExampleto invoke theon('consent')handler. Do not push consent data as an event.
- For step-example tests, use
-
A failure is a throw (the failure contract). A
pushorinitthat returns normally counts as delivered, so:- Throw a
StepErrorfrom@walkeros/corewith acode(configfor a missing or invalid setting,missing-inputfor an event the destination cannot send,transportfor a failed request; addstatusfor an HTTP status,retryableto override the default). Neverreturnearly, neverreturn falsefrominit, never log the failure yourself: the collector counts the push as failed, keeps a DLQ entry and writes one ERROR line per cause (repeats within a minute are counted, not logged again). - Await every call the vendor makes. A call that is not awaited fails after
the push was already counted as delivered. The lint rules
walkeros/await-sendandwalkeros/no-silent-returnflag both. - A response body goes on the error's
body, never into its message; messages and log lines never carry event data. - A value-level problem (one mapped field cannot be built) drops that field,
sends the event and reports once with
reportCause. Only a wholly unsendable event throwsmissing-input.
- Throw a
-
Scripts through
loadVendor(@walkeros/web-core). It injects the script once per page into the documentenvprovides (the page's own by default) and reportsvendor-not-loaded,vendor-not-readyandvendor-timeout. For tests and examples,mockScriptDocument()from@walkeros/web-core/devprovides a document. For a vendor that replays its own stub, callcheck()inpush; for any other,await ready(). Pass attributes the tag needs (data-domain) throughoptions.attributes. -
Honour
rule.silent. A rule withsilent: truesends no event of its own and runs only the side effects it maps (identify, group). Do notreturnearly for it (walkeros/no-silent-returnis an error): writepush = async (event, context) => { if (!context.rule?.silent) await send(event, context); }and run the side effects around it. A silent rule whose side-effect mapping resolves to nothing sends nothing, counts as delivered and writes one DEBUG line (<Vendor> skipped <call>: <mappingKey> resolved to nothing, the event name, never the value). Every destination ships asilentstep example (see Phase 3). -
Consent reaches the vendor.
config.consentdecides: inon('consent'), tell the vendor about a revocation and a grant when it has a switch (getGrantedConsent(config.consent, collector.consent)), and throw aconsent-not-appliedStepError when the vendor does not take it. -
Batching. A server SDK that batches implements
pushBatchand ships a defaultconfig.batchin the destination's own config (the server segment destination:{ size: 15, wait: 10_000 }); each entry has its own outcome, andconfig.batch: { size: 1 }is the per-event setting. See batch scheduling forsize,waitandage.
Reserved fields
config.setup is reserved for the setup lifecycle (see
Adding setup (optional)). Do not use the key setup
for unrelated package metadata or hint keys. The framework wires this field to
SetupFn via resolveSetup, and walkeros setup destination.<name> reads it.
Repurposing the name will collide with that wiring.
Gate: Implementation Compiles
-
npm run buildpasses -
npm run verify:touched -- <platform>-destination-<name>passes (L1: typecheck + lint + test; the argument is the package name after@walkeros/)
Adding setup (optional)
A destination package can implement an optional setup() function to provision
external resources idempotently: BigQuery datasets and tables, Pub/Sub topics,
SQLite tables, warehouse schemas, S3 buckets, webhook registrations on
downstream platforms. Setup runs only when an operator explicitly types
walkeros setup destination.<name>. The runtime never auto-invokes it from
init(), push(), or destroy().
The framework provides the slot, the CLI command, and a resolveSetup helper.
The package owns: what setup means, idempotency, error handling, return value.
For background on how setup fits the destination lifecycle, see understanding-destinations.
Types
// types/index.ts
import type {
Credential,
Destination as CoreDestination,
ServiceAccount,
} from '@walkeros/core';
export interface Settings {
/* runtime push settings */
}
export interface InitSettings {
/* one-time init settings */
}
export interface Mapping {
/* per-event mapping config */
}
export interface Env {
/* injected platform deps (SDK clients, etc.) */
}
// The package's own setup options interface.
// Becomes the U slot of Types; surfaces as `config.setup: boolean | Setup` for users.
export interface Setup {
// package-specific provisioning options
// e.g. for BigQuery: location, partitioning, clustering, schema
location?: string;
partitioning?: { field: string; type: 'DAY' | 'HOUR' };
}
// Service-account / key-pair credentials. Becomes the trailing C slot of Types;
// surfaces as the optional top-level `config.credentials` (sibling of `settings`),
// set with a `$secret.` reference and validated per package.
export type Credentials = Credential<ServiceAccount>;
export type Types = CoreDestination.Types<
Settings,
Mapping,
Env,
InitSettings,
Setup,
Credentials
>;
Where credentials belong. Put service-account / key-pair credentials (the
kind paired with $secret.NAME to inject a secret) on the strictly-typed
top-level config.credentials, not inside settings. Read them from
config.credentials, and fall back to Application Default Credentials when they
are omitted. The raw settings.<sdk> passthrough (e.g. settings.bigquery)
stays as the lower-level escape hatch for SDK-specific auth options.
Implementation
// setup.ts
import type { Destination as CoreDestination, SetupFn } from '@walkeros/core';
import { resolveSetup } from '@walkeros/core';
import type { Setup, Types } from './types';
const DEFAULT_SETUP: Setup = {
location: 'EU',
};
export const setup: SetupFn<
CoreDestination.Config<Types>,
CoreDestination.Env<Types>
> = async ({ config, env, logger }) => {
const options = resolveSetup(config.setup, DEFAULT_SETUP);
if (!options) return; // config.setup is false or unset
// Package-specific provisioning, idempotent.
// Returning a structured object (e.g. { datasetCreated: true })
// makes that data available to operators via `walkeros setup ... | jq`.
};
Wire it in your default export:
// index.ts
import type { Destination as CoreDestination } from '@walkeros/core';
import { setup } from './setup';
import type { Types } from './types';
export const destinationMine: CoreDestination.Instance<Types> = {
type: 'my-destination',
config: {},
async push(event, context) {
// Send the event (Phase 7)
},
setup,
};
export default destinationMine;
When to implement
Implement setup() when your destination needs first-time provisioning of
external resources before events can flow: warehouse tables, Pub/Sub topics, S3
buckets, schema bindings, IAM roles, webhook registrations on downstream
platforms. Skip it when your destination only consumes credentials or HTTP
endpoints the user already provisioned.
Contract
- Triggered only by
walkeros setup <kind>.<name>. Never by runtime push, init, or destroy. - Idempotency is your responsibility. Re-running setup against a fully
provisioned environment must be a safe no-op. Use try-create-catch-409 on REST
APIs,
IF NOT EXISTSon SQL, native idempotent operations where available. The framework does not retry, track state, or detect drift. - Return structured data from
setup()when useful for operator scripting. The CLI emits non-undefined return values as JSON to stdout. - For packages where
setup: true(boolean form) is meaningless because mandatory fields have no safe defaults (e.g., KafkanumPartitions, GitHub webhookwebhookUrl), reject the boolean form with a clear runtime error listing required fields:
if (config.setup === true) {
throw new Error(
'kafka destination setup requires explicit options: ' +
'{ topic, numPartitions, replicationFactor }. There is no safe default.',
);
}
Phase 8: Test Against Examples
Tests verify implementation against the examples from Phase 3. If examples are incomplete, tests will be incomplete.
See testing-strategy for the shared env / dev-examples conventions this phase depends on.
Verify implementation produces expected outputs.
Test Templates
Use the test templates:
conformance.test.ts (the conformance
kit and every event example) and
index.test.ts (the init example). Reference
canonical implementation:
packages/web/destinations/meta/src/__tests__/conformance.test.ts.
Conformance test
Every destination has a conformance test that runs it on a real collector with
conformance('destination', destination, options) from
@walkeros/collector/dev (template:
conformance.test.ts). It checks the
failure contract: a failing send is reported with a cause, a failure never leaks
event data into a log line, a silent example sends nothing, a consent
revocation reaches the vendor, init throws on a missing setting, and every
event the destination cannot send has a declared outcome. It fails with every
broken item at once.
conformance does not compare a failure example's outcome with what the push
reported. The same test file therefore runs the event examples through
runExamples with claims, which compares each example's out and outcome:
a failure example with a wrong outcome fails the test and is not published under
"When it fails".
It also runs the minimal example: pass minimal: examples.minimal (the template
does), and the kit starts the destination with the settings its docs quickstart
shows and requires one public example delivered. A package without
src/examples/minimal.ts passes withoutMinimal: '<reason>' instead. A test
with neither, with both, or with an option that disagrees with the file fails
and says which option to pass, so a copied template without minimal fails.
The profile option says what the destination talks to: http (calls fetch),
sdk-env (a server SDK injected through env), web-sdk (a web SDK started in
init: sdk-env without the init check), vendor-global (a vendor script that
defines a global), socket (a long-lived connection). The browser profiles
(web-sdk, vendor-global) also take page from @walkeros/web-core/dev
(import it after the @walkeros/collector/dev import, pass it right after
profile); without it the test throws
profile <name> runs on a page: pass page from @walkeros/web-core/dev. Server
destinations pass no page. Never prove delivery or loss with a mocked
collector: startFlow and mock only the network or the vendor global.
Key Test Patterns
runExamplesruns every event example on a real collector, withclaimslisting the standard events the destination maps. Do not write per-feature tests with hand-rolled payloads.- The init example runs on its own: its
inis a config, not an event. - Zero hardcoded payloads: every input, vendor config, and expected output
comes from
examples.steporexamples.env. If you need something new, add it to examples first. - Clone env per test:
clone(examples.env.push)so vendor mocks don't leak across tests.
Gate: Tests Pass
-
npm run verify:touched -- <platform>-destination-<name>passes (L1) - The event examples run through
runExampleswithclaims - Tests contain no hardcoded payloads, vendor configs, or expected outputs
- Every assertion reads from
examples.step[...].out - The conformance test passes with the right profile and
minimal: examples.minimal(orwithoutMinimal: '<reason>') -
examples.stephas asilentexample, a failure example (outcome, with adescription) for each event the destination cannot send, and a consent revocation example when the vendor has a consent switch
Phase 9: Document
Follow the writing-documentation skill for:
- README structure and templates
- Example validation against
apps/quickstart/ - Quality checklist before publishing
Key requirements for destination documentation:
- Event mapping table (walkerOS → vendor format)
-
## Installationrenders<Quickstart data={data} />: the install line, integrated code andflow.jsongenerated fromexamples.minimal. No hand-written install or setup snippet there; the doc-lint fails one -
<Configuration type="destination" data={data} />renders settings, mapping, examples and "When it fails" (from the failure examples'description) fromwalkerOS.json
Validation Checklist
Beyond
understanding-development
requirements (build, test, lint, no any):
- Uses
getEnv<Env>(env)pattern (never directwindow/documentaccess, neveras Window/as Document/as unknowncasts in src or tests) -
dev.tsexportsschemas(withsettings;z.object({})when the step has none) andexamples - Multi-export package (two or more
walkerOS.exports):exportExamplesandexportSchemasindev.tslist exactly those exports, the default included, and everyexportSchemasentry hassettings; the build fails otherwise (see using-step-examples) - No silent path: a failure is a thrown StepError, every vendor call is
awaited,
initnever returnsfalse, scripts load throughloadVendor,rule.silentis honoured - Examples match type signatures
- Tests use examples for assertions (not hardcoded values)
-
walkerOS.jsongenerated at build time -
walkerOSfield in package.json - Works with no
env: one test runs the published default export from a pure JSON config (no injectedenv, no mocked SDK) against a local fake endpoint on127.0.0.1, with the real vendor SDK and dummy credentials, and asserts what the fake received. Mocked-env tests cannot catch a missingenv?.X ?? Xfallback, which turns a bundled flow into a silent no-op
Reference Files
| What | Where |
|---|---|
| Simple template | packages/web/destinations/plausible/ |
| Complex example | packages/web/destinations/gtag/ |
| Types | packages/web/core/src/types/destination.ts |
Related Skills
- understanding-destinations - Destination interface and env pattern
- using-step-examples -
Authoritative
Flow.StepExamplepattern and Three Type Zones - testing-strategy - Testing with env mocking and dev-examples-as-fixtures conventions
- writing-documentation - Documentation standards
Create a New Source
Prerequisites
Before starting, read these skills:
- understanding-flow - How sources fit in architecture
- understanding-sources - Source interface
- understanding-transformers - Transformer chaining from sources
- understanding-events - Event structure sources emit
- understanding-mapping - Transform raw input to events
- testing-strategy - How to test with env pattern and dev examples
- using-step-examples -
Authoritative
Flow.StepExamplepattern,createTrigger, Three Type Zones - using-logger - For an HTTP source, see
"Choosing a level on public endpoints": input the source rejects itself is a
debug line plus a
rejectedcounter and a 4xx; an event the collector refuses as invalid gets the collector's one ERROR line per source and minute, which the source does not repeat - writing-documentation - Documentation standards (for Phase 9)
Source Types
A source's identity is split into two fields:
| Field | Meaning | Examples |
|---|---|---|
source.type | The kind of source (its role / mechanism) | browser, dataLayer, cookiefirst, fetch |
source.platform | The runtime that hosts the source (web, server, app, ...) | web, server |
| Platform | Input | Example types |
|---|---|---|
web | DOM events, dataLayer | browser, dataLayer |
server | HTTP requests, webhooks | express, fetch, lambda, cloudfunction |
Augmenting SourceMap
Every source registers its type literal and any source-specific source.*
fields by augmenting SourceMap from @walkeros/core. This is how the
collector and downstream consumers know about the new source kind without
loosening the union to string.
Add this to the source's src/types.ts (or src/types/index.ts):
import type { Source, Elb } from '@walkeros/core';
declare module '@walkeros/core' {
interface SourceMap {
// Replace `mySource` with the source's package-level identifier.
mySource: {
type: 'mySource'; // matches the literal you return from Source.Init
platform?: 'web'; // 'web' | 'server' | 'app' | ...
// Add any extra fields the source surfaces in `event.source.*` here.
// e.g. `version?: string;` is already on the base Source - only add
// truly source-specific keys.
};
}
}
Reference implementations: packages/web/sources/browser/src/types/index.ts,
packages/web/sources/demo/src/types.ts. Conflicting declarations cause compile
errors on purpose, this surfaces naming collisions early.
Source Categories
| Category | Purpose | Examples | Key Concern |
|---|---|---|---|
| Transformation | Convert external format → walkerOS events | dataLayer, fetch | Mapping accuracy |
| Transport | Receive events from specific platform | gcp, aws, express | Platform integration |
Choose Your Template
| Complexity | Template | When to Use |
|---|---|---|
| Simple transformation | fetch/ | Generic HTTP handler, data conversion |
| Platform transport | gcp/, aws/ | Cloud platform integration |
| Browser interception | dataLayer/ | DOM events, array interception |
Process Overview
1. Research → Deeply understand external system, SDK, and data format
2. Classify → Determine source type and integration approach
3. Examples → Define in/out pairs FIRST (start with the end result)
4. Mapping → Define input → walkerOS event transformation
5. Scaffold → Copy template and configure
6. Convention → Add walkerOS.json metadata and buildDev
7. Implement → Build using examples as test fixtures
8. Test → Verify against example variations
9. Document → Write README
Phase 1: Research
Goal: Deeply understand the external system before writing any code. Research quality determines implementation quality.
1.1 Find and Install Official SDK
Always prefer the vendor's official SDK package over raw HTTP API calls. The SDK handles transport, data formatting, and platform specifics - don't reinvent these.
- Install the SDK -
npm install @vendor/sdkand read the actual source - Read TypeScript types - Import types from the SDK package directly. Never duplicate type definitions. This ensures IntelliSense completeness and consistency with SDK updates.
- Understand the full API surface - List every public method and type export. What data structures does the platform provide? What request/event formats exist?
# Search npm for official packages
npm search [vendor-name]
npm search @[vendor]
# Install and inspect actual types
npm install @vendor/sdk
ls node_modules/@vendor/sdk/lib/esm/
1.2 Understand SDK Architecture
- Init options - What does the SDK expose? How is the platform connection established?
- Call ordering - When does data arrive? Is it pushed (webhooks, callbacks) or pulled (polling, intercepting)? What are the timing implications?
- Data format - What does the raw event/request look like? Headers, body structure, query params, authentication tokens?
- Identity signals - Does the external system provide user IDs, session IDs, device IDs? How are they delivered (headers, cookies, body fields)?
- Consent - Does the platform have consent signals? How are they communicated?
1.3 Identify All Data Entry Points
Go beyond just the primary event payload. Most external systems provide multiple data channels:
| Data Channel | Examples | walkerOS Handling |
|---|---|---|
| Event payload | Request body, DOM event data | Default push() |
| Headers/metadata | Auth tokens, content-type, origin | context or user |
| Query params | UTM parameters, tracking IDs | data or context |
| Platform context | Cloud function metadata, Lambda ctx | source or custom |
| Identity | User ID, session ID, device ID | user |
| Consent signals | Opt-in/out flags, consent string | consent |
1.4 Check Existing Patterns
Review similar sources in the codebase:
# List existing sources
ls packages/web/sources/
ls packages/server/sources/
# Reference implementations
# - dataLayer: DOM-based, array interception
# - express: HTTP middleware
# - fetch: Generic HTTP handler (simplest server pattern)
# - gcp: Cloud Functions specific
Gate: Research Complete
- Official SDK installed and types inspected (or HTTP API documented if no SDK exists)
- All data entry points listed with their format
- Init options and call ordering understood
- Identity and consent signal delivery documented
- Raw event/request structure mapped to walkerOS event fields
Checkpoint: Research Review (Optional)
If working with human oversight, pause here to confirm:
- SDK integration approach correct?
- Data capture strategy makes sense for the use case?
- Any platform quirks or limitations to handle?
Phase 2: Classify Source Type
Goal: Understand what the source captures and how it delivers data, which determines implementation complexity.
2.1 Source Categories
| Category | Description | Mapping Needed | Example Sources |
|---|---|---|---|
| Transformation | Converts external event format to walkerOS | Essential - must map fields | dataLayer, fetch |
| Transport | Receives events from a specific platform | Structural - platform unwrap | gcp, aws, express |
| Interception | Intercepts existing data flows | Varies - depends on data format | dataLayer, CMP sources |
2.2 Determine Integration Approach
| Approach | When to use | Pattern |
|---|---|---|
| Platform SDK as host | SDK provides typed request/response | Use SDK types, wrap handler in walkerOS source |
| DOM interception | Capture browser-side events | Listen to DOM events, intercept arrays/globals |
| HTTP handler | Generic webhook/API receiver | Parse request, extract events, forward to collector |
| Callback/event listener | Platform provides event emitter | Register listener, transform events, forward to collector |
Prefer the vendor SDK - it provides typed request/response objects and handles platform specifics. Raw HTTP parsing is a fallback when no SDK exists.
When using the vendor SDK:
- Import types from the SDK package directly
- Use SDK request/response types for handler signatures
- Let the SDK handle platform-specific parsing (body parsing, header extraction)
For an HTTP handler, forward the full DeepPartialEvent the body carries,
renaming only the wire-level event-name key. Never project the body through a
field allowlist: that strips source provenance (release, trace) and other
legitimate fields. express and fetch are the reference pass-through
implementations.
Gate: Classification Complete
- Source category identified (transformation / transport / interception)
- Integration approach chosen (SDK / DOM / HTTP / callback)
- Know what the source captures and how it delivers data
Phase 3: Create Input Examples (BEFORE Implementation)
Mandatory. Examples are the test fixtures for Phase 8. Define expected
trigger / in / out triples FIRST - start with the end result in mind.
Without examples, you cannot test. Even for simple sources, step examples are
the single source of truth for tests, simulations, and documentation.
Authoritative pattern: See using-step-examples for the Three Type Zones,
createTriggercontract, and CI integration. This skill reuses that contract - do not diverge.
3.1 Scaffold Directory Structure
mkdir -p packages/<platform>/sources/[name]/src/examples
mkdir -p packages/<platform>/sources/[name]/src/{schemas,types}
3.2 Required Files (3-4 files)
Use this layout in src/examples/. Every example a test reads lives in
step.ts; do not add inputs.ts, outputs.ts, requests.ts or a standalone
mapping.ts.
| File | Required? | Purpose |
|---|---|---|
examples/step.ts | yes | Flow.StepExample entries with trigger / in / out triples |
examples/trigger.ts | yes | createTrigger implementation following Trigger.CreateFn |
examples/index.ts | yes | Barrel exports: env (if present), step, createTrigger |
examples/env.ts | if needed | Mock env for platform deps (browser window/document, express, etc.) |
env.ts is included whenever the source touches platform globals or injected
deps (browser, dataLayer, demo, the Lambda and Cloud Function sources). Sources
whose tests drive the collector entirely through trigger.ts (e.g.
web/sources/session, server/sources/express) may omit it. When in doubt,
include it.
Some existing packages also keep fixture files their own tests import (fetch:
inputs.ts, requests.ts; dataLayer: events.ts, mapping.ts; the CMP
sources: inputs.ts, outputs.ts). A new source does not add them: anything a
test needs goes into step.ts.
3.3 Three Type Zones for Sources
Sources are the inverse of destinations in the Three Type Zones model:
| Zone | Source semantics |
|---|---|
trigger | How to simulate the invocation (HTTP method, DOM event type, cloud event) |
in | External trigger content - HTTP request, DOM HTML, SDK payload (NOT a walkerOS event) |
out | What the source hands the collector: ['elb', event] tuples (Flow.StepOut) |
Where a destination does WalkerOS.Event → vendor output, a source does
external content → ['elb', event]. Read
using-step-examples before authoring
entries.
3.4 Typing Rules (strict)
No any. Every example value must be explicitly typed.
triggeruses the local source trigger type or a platform-native type (e.g.'load' | 'click'for DOM, HTTP method strings for server sources).inuses the vendor / platform SDK types imported from the official package whenever available (ExpressRequest, FetchRequest, API GatewayAPIGatewayProxyEvent, LambdaContext, GCPCloudEvent, etc.). Do not invent local request types when the platform publishes them.outlists['elb', event]tuples; each event is aWalkerOS.DeepPartialEvent(what the source pushes, before the collector completes it).- Step entries are typed
Flow.StepExamplefrom@walkeros/core. - Mock env is typed against the source's local
Envtype from../types. createTriggeris typed asTrigger.CreateFn<unknown, Result>and narrows its content to the source's ownContentwith a type guard: a simulation passes any JSON, and the conformance kit'screateTriggeroption is aTrigger.CreateFn, which aCreateFn<Content, Result>does not satisfy.
3.5 Code Template - examples/step.ts
import type { Flow } from '@walkeros/core';
// One step example per captured trigger / input shape.
// `trigger` tells createTrigger how to simulate the invocation.
// `in` is the platform-specific content (HTTP request, DOM HTML, SDK payload),
// typed against the platform SDK's published types where available.
// `out` lists what the source hands the collector, as ['elb', event] tuples.
// Set `title` + `description` for public examples; mark test-only fixtures
// with `public: false`. See
// [walkeros-using-step-examples](../walkeros-using-step-examples/SKILL.md).
// Browser example (the browser source's own): a page load.
export const pageView: Flow.StepExample = {
title: 'Page view',
description:
'A page load trigger captures the current URL, title, and referrer as a walker page view event.',
trigger: {
type: 'load',
options: {
url: 'https://example.com/docs',
title: 'Documentation',
referrer: 'https://example.com/',
},
},
in: '', // no external content: DOM-driven trigger
out: [
[
'elb',
{
name: 'page view',
data: {
domain: 'example.com',
title: 'Documentation',
referrer: 'https://example.com/',
id: '/docs',
},
context: {},
globals: {},
trigger: 'load',
source: {
type: 'browser',
platform: 'web',
url: 'https://example.com/docs',
referrer: 'https://example.com/',
},
},
],
],
};
// Server example (the express source's own): an HTTP POST with a walker event.
export const postEvent: Flow.StepExample = {
title: 'POST event',
description:
'An Express POST to /collect with a JSON body becomes a single walker elb event.',
trigger: { type: 'POST' },
in: {
method: 'POST',
path: '/collect',
body: {
name: 'page view',
data: { title: 'Home', url: 'https://example.com/' },
},
},
out: [
[
'elb',
{
name: 'page view',
data: { title: 'Home', url: 'https://example.com/' },
},
],
],
};
3.6 examples/index.ts (barrel)
export * as env from './env'; // omit if the source has no env.ts
export * as step from './step';
export { createTrigger } from './trigger';
3.7 examples/trigger.ts - createTrigger
Every source exports a createTrigger following the unified Trigger.CreateFn
interface. It simulates real-world invocations from the outside: a real request,
DOM event or SDK call, as a user's platform makes it.
import type { Trigger } from '@walkeros/core';
import { isObject } from '@walkeros/core';
import { startFlow } from '@walkeros/collector';
// The source's own shapes (here an HTTP request and its answer)
export interface Content {
method: string;
url: string;
}
export interface Result {
status: number;
}
const isContent = (value: unknown): value is Content =>
isObject(value) &&
typeof value.method === 'string' &&
typeof value.url === 'string';
export const createTrigger: Trigger.CreateFn<unknown, Result> = async (
config,
) => {
let flow: Trigger.FlowHandle | undefined;
const trigger: Trigger.Fn<unknown, Result> =
(type?: string) => async (content) => {
if (!isContent(content)) throw new Error('content is not a request');
if (!flow) {
const result = await startFlow(config);
flow = { collector: result.collector, elb: result.elb };
}
// Package-specific: make a real HTTP request, inject DOM, dispatch an
// SDK call, and return the Result type this source declares.
return /* ... */;
};
return {
get flow() {
return flow;
},
trigger,
};
};
The server template's examples/trigger.ts is a complete one. Reference implementations:
- Browser:
packages/web/sources/browser/src/examples/trigger.ts- DOM injection + native event dispatch - Session:
packages/web/sources/session/src/examples/trigger.ts- no env.ts, trigger drives collector directly - Express:
packages/server/sources/express/src/examples/trigger.ts- real HTTPfetch()to running server - CMP (Usercentrics):
packages/web/sources/cmps/usercentrics/src/examples/trigger.ts- dispatches CMP events, asserts on collector consent state - Fetch (function handler):
packages/server/sources/fetch/src/examples/trigger.ts- accesses source instance viacollector.sources, callssource.push()with platform-nativeRequest - AWS Lambda:
packages/server/sources/aws/src/lambda/examples/trigger.ts- constructs API Gateway event + Lambda context - GCP CloudFunction:
packages/server/sources/gcp/src/cloudfunction/examples/trigger.ts- synthesizes mock req/res (matching GCP Functions Framework)
3.8 Test Fixture Contract (hard rule)
The examples authored here are the Phase 8 test fixtures. No parallel fixtures allowed.
src/__tests__/stepExamples.test.tsMUST iterate examples viait.each(Object.entries(examples.step)).- Tests must NOT contain hardcoded trigger payloads, HTTP requests, DOM HTML, or expected events.
- If a test needs a value that is not in
examples.step, add it tostep.tsfirst, then consume it from the test. - Tests invoke
examples.createTrigger(config)and dispatch each example'strigger.type+incontent, asserting the collector receivesout.
See the canonical source tests
packages/web/sources/browser/src/__tests__/stepExamples.test.ts and
packages/server/sources/express/src/__tests__/stepExamples.test.ts.
3.9 Export via dev.ts
export * as schemas from './schemas';
export * as examples from './examples';
Phase 3 Acceptance Checklist
-
src/examples/step.ts- oneFlow.StepExampleper captured trigger / input shape, typedtrigger/in/out -
src/examples/trigger.ts- exportscreateTriggertyped asTrigger.CreateFn<unknown, Result>, narrowing its content with a guard -
src/examples/index.ts- barrel exportsstep,createTrigger, andenv(when present) -
src/examples/env.ts- included whenever the source touches platform globals or injected deps; typed against localEnv; no real network - No standalone
inputs.ts,outputs.ts,requests.ts, ormapping.tsfiles - All platform / SDK types imported from the official package - no
any, no reinvented request / response shapes -
src/__tests__/stepExamples.test.tsiteratesexamples.stepviait.each(Object.entries(...)) - Tests contain zero hardcoded payloads, requests, or expected events -
everything flows from
examples.step - Edge cases included (minimal input, invalid input)
-
npm run buildpasses - examples compile against published types - Each example traces:
trigger+in→ source push → matchesout
Phase 4: Define Mapping
Goal: Document transformation from input format to walkerOS events.
Mapping lives inside each Flow.StepExample entry in step.ts - no
separate mapping.ts file. Sources typically carry the mapping either in the
source's own settings (see dataLayer for an example) or inline via the
trigger → in → out relationship: the in content is the raw platform
payload; the out lists the events the source pushes after its transformation,
as ['elb', event] tuples.
Verify Mapping Logic
For each entry in step.ts, trace:
Input: examples.step.pageView.trigger + examples.step.pageView.in
↓ createTrigger dispatches the trigger
↓ Source receives platform content, runs its transformation
↓ Source calls env.push / collector.push
Output: Should match examples.step.pageView.out (['elb', event] tuples)
Gate: Mapping Verified
- Step examples cover the main input event types
- Each example name transforms to correct walkerOS event name
- Each example traces correctly from
(trigger, in)toout
Phase 5: Scaffold
Template sources:
- Web:
packages/web/sources/dataLayer/ - Server:
packages/server/sources/fetch/(simplest pattern)
cp -r packages/server/sources/fetch packages/server/sources/[name]
cd packages/server/sources/[name]
# Update package.json: name, description, repository.directory
Directory structure:
packages/server/sources/[name]/
├── src/
│ ├── index.ts # Main export
│ ├── __tests__/ # stepExamples.test.ts and the other tests
│ ├── dev.ts # Exports schemas and examples
│ ├── examples/
│ ├── schemas/
│ └── types/
├── package.json
├── tsconfig.json
├── tsup.config.ts
├── jest.config.mjs
└── README.md
Transformer Chain Integration
Sources can wire to transformer chains via next in the init config:
sources: {
mySource: {
code: sourceMySource,
config: { settings: { /* ... */ } },
next: 'validate' // Events go through validator before collector
}
}
Phase 6: walkerOS.json Convention
Every walkerOS package ships a walkerOS.json file for CDN-based schema
discovery.
Add walkerOS field to package.json
{
"walkerOS": { "type": "source", "platform": "web" },
"keywords": ["walkerOS", "walkerOS-source", ...]
}
Use buildDev() in tsup.config.ts
Keep buildDev() in the defineConfig array (the template package's
tsup.config.ts has it):
import { buildDev } from '@walkeros/config/tsup';
// In defineConfig array:
buildDev(),
It builds src/dev.ts and writes dist/walkerOS.json from the schemas,
examples and hints it exports.
Hints (Optional)
If your source has capabilities, behaviors, or troubleshooting patterns not
obvious from schemas alone, add hints. See walkeros-writing-documentation
skill for full guidelines.
Create src/hints.ts:
import type { Hint } from '@walkeros/core';
export const hints: Hint.Hints = {
'capture-timing': {
text: 'Describes when events are captured. See settings schema for options.',
code: [{ lang: 'json', code: '{ "settings": { ... } }' }],
},
};
Export from src/dev.ts:
export * as schemas from './schemas';
export * as examples from './examples';
export { hints } from './hints';
Guidelines:
- Expand awareness - describe capabilities ("supports X, Y, Z"), don't prescribe one path
- Reference schemas and examples, don't duplicate them
- Verify every claim against actual implementation before publishing
- Key naming: kebab-case, group with prefixes (auth-*, capture-*, troubleshoot-*)
- Most sources don't need hints - schemas and examples cover the common case
Gate: Convention Met
-
walkerOSfield in package.json with type and platform -
buildDev()in tsup.config.ts - Build generates
dist/walkerOS.json - Keywords include
walkerOSandwalkerOS-source
Runtime-only npm dependencies
A third-party npm dependency that cannot be ESM-bundled (uses __dirname,
ships a .node binary, etc.) needs no declaration: server bundles are traced
with nft, which copies every file the bundle reaches at runtime. Do not add a
walkerOS.bundle field to package.json: the CLI ignores it and warns
still declares walkerOS.bundle.<keys>; this is ignored. See
walkeros-using-cli: server bundles use nft tracing.
Phase 7: Implement
Now write code to produce the outputs defined in Phase 3.
Template Files
| File | Purpose | Template |
|---|---|---|
types/index.ts | Type definitions | types.ts |
schemas/index.ts | Zod settings schema | schemas.ts |
index.ts | Main source | index.ts |
dev.ts | Dev entry (schemas, examples) | dev.ts |
examples/ | step.ts, trigger.ts, index.ts | examples/ (step.ts, trigger.ts, index.ts) |
__tests__/stepExamples.test.ts | Step example tests | index.test.ts |
__tests__/conformance.test.ts | Conformance kit (profile source) | conformance.test.ts |
The server template is an HTTP handler that turns a platform's webhook body into walkerOS events and answers the producer from the push result.
Key Patterns
- Context destructuring: Extract
config,env,logger,idfrom context - Settings schema, defaults inline: the Zod schema in
src/schemasdescribes the settings forwalkeros validate,walkerOS.jsonand the docs; the source applies its defaults inline and does not parse settings at runtime (zod stays out of the runtime bundle) - Forward to collector: Call
env.push()to send events to the collector - Platform deps via env: All platform dependencies (window, document,
express, cors) must go through
envwith fallback to globals/imports:env.express ?? express. This enables testing without mocking globals. - Failures are reported, not logged by the source: a source that cannot
start throws a
StepError(code: 'config') from its factory orinit; it never returns early and never returnsfalse. Input the collector rejects comes back asinvalidin the push result and is counted asrejectedby the collector. A failure on the source's own side that the push result does not carry (for example anelbLayerentry that fails) goes throughreportCausefrom@walkeros/core(stepsource.<id>, named by its event, never its payload). Useloggerfordebugtraces of routine operations. - Tell the producer the truth: an HTTP source builds its answer from the
push result with
pushResultToOutcome(oracknowledge(result)):ackanswers 200 with the source's success body,retry(a retryable failure) answers 500, or 207 for the failed events of a batch,reject(invalid input) answers 400. A failure a resend cannot fix is acknowledged, because redelivery would loop. A queue consumer acks onackandrejectand nacks only onretry. The body never contains a destination id or a step's error text. - Return Source.Instance: Return
{ type, config, push }object - Optional
destroymethod: Implement if the source holds resources (HTTP servers, timers, connections) that need cleanup on shutdown
Gate: Implementation Compiles
-
npm run buildpasses -
npm run verify:touched -- <platform>-source-<name>passes (L1: typecheck + lint + test; the argument is the package name after@walkeros/)
Adding setup (optional)
A source package can implement an optional setup() function to provision
external resources idempotently: Pub/Sub subscriptions, webhook registrations on
upstream platforms, queue declarations, SQS queues, polling cursors, inbound API
keys. Setup runs only when an operator explicitly types
walkeros setup source.<name>. The runtime never auto-invokes it from init(),
push, or destroy().
The framework provides the slot, the CLI command, and a resolveSetup helper.
The package owns: what setup means, idempotency, error handling, return value.
Today the command cannot run a source's setup. walkeros setup reads the
package's default export as an object with a setup method, while a source's
default export is its init function, so walkeros setup source.<name> stops
with Package <name> has no default export. The SQS source exports its setup
from its module (packages/server/sources/aws/src/sqs/index.ts), and the
Pub/Sub pull source returns setup on the instance its init returns; the
command reaches neither, so provision their resources with the platform's own
tooling (gcloud, the AWS CLI, your infrastructure code).
For background on how setup fits the source lifecycle, see understanding-sources.
Types
// types/index.ts
import type { Elb, Source as CoreSource } from '@walkeros/core';
export interface Settings {
/* runtime push settings */
}
export interface InitSettings {
/* one-time init settings */
}
export interface Mapping {
/* event extraction mapping */
}
export type Push = Elb.Fn; // the source's push signature (an HTTP handler, ...)
export interface Env extends CoreSource.BaseEnv {
/* injected platform deps (SDK clients, request handlers, etc.) */
}
// The package's own setup options interface.
// Becomes the U slot of Types; surfaces as `config.setup: boolean | Setup` for users.
export interface Setup {
// package-specific provisioning options
// e.g. for Pub/Sub source: subscriptionName, ackDeadlineSeconds, filter
subscriptionName?: string;
ackDeadlineSeconds?: number;
}
export type Types = CoreSource.Types<
Settings,
Mapping,
Push,
Env,
InitSettings,
Setup
>;
Implementation
// setup.ts
import type { Source as CoreSource, SetupFn } from '@walkeros/core';
import { resolveSetup } from '@walkeros/core';
import type { Setup, Types } from './types';
const DEFAULT_SETUP: Setup = {
ackDeadlineSeconds: 60,
};
export const setup: SetupFn<
CoreSource.Config<Types>,
CoreSource.Env<Types>
> = async ({ config, env, logger }) => {
const options = resolveSetup(config.setup, DEFAULT_SETUP);
if (!options) return; // config.setup is false or unset
// Package-specific provisioning, idempotent.
// Returning a structured object (e.g. { subscriptionCreated: true })
// makes that data available to operators via `walkeros setup ... | jq`.
};
Export it next to the source's init function, as the SQS source's module does:
// index.ts
import type { Source as CoreSource } from '@walkeros/core';
import type { Types } from './types';
export { setup } from './setup';
export const sourceMine: CoreSource.Init<Types> = async (context) => ({
type: 'my-source',
config: context.config,
push: context.env.elb,
});
export default sourceMine;
When to implement
Implement setup() when your source needs first-time provisioning of upstream
resources before events can be received: Pub/Sub subscriptions bound to a topic,
webhook registrations on upstream SaaS platforms (Stripe, GitHub, Shopify), SQS
queue declarations, message broker bindings, polling cursors. Skip it when your
source only listens on an HTTP port the runtime already owns or intercepts data
already present (DOM events, dataLayer pushes).
Contract
- Triggered only by
walkeros setup <kind>.<name>. Never by runtime push, init, or destroy. - Idempotency is your responsibility. Re-running setup against a fully
provisioned environment must be a safe no-op. Use try-create-catch-409 on REST
APIs,
IF NOT EXISTSon SQL, native idempotent operations where available. The framework does not retry, track state, or detect drift. - Return structured data from
setup()when useful for operator scripting. The CLI emits non-undefined return values as JSON to stdout. - For packages where
setup: true(boolean form) is meaningless because mandatory fields have no safe defaults (e.g., GitHub webhookwebhookUrl, Pub/Sub sourcetopicName), reject the boolean form with a clear runtime error listing required fields:
if (config.setup === true) {
throw new Error(
'github-webhook source setup requires explicit options: ' +
'{ webhookUrl, repo, events }. There is no safe default.',
);
}
Phase 8: Test Against Examples
Tests verify implementation against the examples from Phase 3. If examples are incomplete, tests will be incomplete.
See testing-strategy for the shared env / dev-examples conventions this phase depends on.
Verify implementation produces expected outputs.
Test Template
Use the test template: index.test.ts, copied
to src/__tests__/stepExamples.test.ts. Canonical references:
packages/web/sources/browser/src/__tests__/stepExamples.test.tspackages/server/sources/express/src/__tests__/stepExamples.test.ts
Key Test Patterns
it.each(Object.entries(examples.step))is mandatory - one iteration per step example. Do not write per-feature tests with hand-rolled payloads.- Drive via
createTrigger- construct the trigger withstartFlowconfig, then dispatch each example'strigger.type+incontent. - A typed context helper for a test that calls the source factory directly:
a small function in the test file, as
packages/server/sources/express/src/__tests__/index.test.tsdefinescreateSourceContext. - Zero hardcoded payloads - every trigger type, request body, DOM HTML, and
expected event comes from
examples.steporexamples.env. If you need something new, add it to examples first. - Test error paths - verify graceful error handling for invalid input (add
an error example to
examples.stepif needed), and that a push that fails for a retryable reason is answered with a retry, a deterministic one with an ack. - Conformance - run the source with
conformance('source', source, options)from@walkeros/collector/dev(profilesource,createTriggerfor how to feed it); it runs on a real collector. A web source may passpagefrom@walkeros/web-core/dev; the kit requires it only for the browser profilesweb-sdkandvendor-global.
Gate: Tests Pass
-
npm run verify:touched -- <platform>-source-<name>passes (L1) - Tests iterate via
it.each(Object.entries(examples.step)) - Tests contain no hardcoded payloads, requests, or expected events
- Every assertion reads from
examples.step[...].out - Invalid input handled gracefully (no crashes)
Phase 9: Document
Follow the writing-documentation skill for:
- README structure and templates
- Example validation against
apps/quickstart/ - Quality checklist before publishing
Key requirements for source documentation:
- Input format table documenting expected fields
- Event name mapping table (source format → walkerOS format)
- Configuration options table
- Working code example with imports
- Installation instructions
Validation Checklist
Beyond
understanding-development
requirements (build, test, lint, no any):
-
dev.tsexportsschemas(withsettings;z.object({})when the step has none) andexamples - Multi-export package (two or more
walkerOS.exports):exportExamplesandexportSchemasindev.tslist exactly those exports, the default included, and everyexportSchemasentry hassettings; the build fails otherwise (see using-step-examples) - Examples include edge cases (minimal, invalid input)
- Invalid input returns gracefully (no crashes, clear error)
- Tests use examples for assertions (not hardcoded values)
-
walkerOS.jsongenerated at build time -
walkerOSfield in package.json
Reference Files
| What | Where |
|---|---|
| Web template | packages/web/sources/dataLayer/ |
| Server template | packages/server/sources/fetch/ |
| Source types | packages/core/src/types/source.ts |
| Test events | packages/core/src/eventGenerator.ts |
Related Skills
- understanding-sources - Source interface and push pattern
- understanding-events - Event structure
- using-step-examples -
Authoritative
Flow.StepExample+createTriggerpattern, Three Type Zones - testing-strategy - Testing with env mocking and dev-examples-as-fixtures conventions
- writing-documentation - Documentation standards
Create a New Transformer
Prerequisites
Before starting, read these skills:
- understanding-flow - How transformers fit in architecture
- understanding-transformers - Transformer interface
- understanding-events - Event structure
- testing-strategy - How to test
- writing-documentation - Documentation standards (for Phase 7)
Transformer Categories
| Category | Purpose | Example |
|---|---|---|
| Validate | Check event structure/content | JSON Schema, required fields |
| Enrich | Add data to events | Server timestamps, geo data |
| Redact | Remove/mask sensitive data | Strip PII, anonymize IPs |
Process Overview
1. Research → Understand use case (validate/enrich/redact)
2. Examples → Create event before/after examples FIRST
3. Scaffold → Copy template, configure package.json
4. Convention → Add walkerOS.json metadata and buildDev
5. Implement → Build transformer with TDD
6. Test → Verify against example transformations
7. Document → Write README
Supporting Files
This skill includes reference files you can copy:
- templates/validation/ - Complete redact
transformer template
- index.ts - Implementation with context pattern
- types.ts - Type definitions
- schemas.ts - Zod settings schema
- index.test.ts - Test suite with helpers, one test per step example
- conformance.test.ts - The conformance kit on a real collector
- examples/ - The template's examples:
step.ts,events.ts(which its unit tests read) andconfig.ts(sample configs)
Copy them into a package as Phase 3 lays it out: index.ts to
src/transformer.ts, types.ts to src/types.ts, schemas.ts to
src/schemas/index.ts, index.test.ts to src/__tests__/stepExamples.test.ts,
conformance.test.ts to src/__tests__/ and examples/ to src/examples/.
Phase 1: Research
Goal: Understand what the transformer needs to do.
1.1 Define Use Case
- Category: Validate, Enrich, or Redact?
- Input: What events will this process?
- Output: What should change? What should be blocked?
- Configuration: What settings does user need?
1.2 Check Existing Patterns
# Reference implementation
ls packages/server/transformers/fingerprint/
# Transformer types
cat packages/core/src/types/transformer.ts
Gate: Research Complete
- Category identified (validate/enrich/redact)
- Input/output transformation defined
- Configuration options listed
Phase 2: Create Examples (BEFORE Implementation)
Goal: Define event transformations in dev entry FIRST.
2.1 Scaffold Directory Structure
mkdir -p packages/server/transformers/[name]/src/{examples,schemas}
2.2 Step Examples
Write one step example per case in src/examples/step.ts: events that pass
through (modified or not), the expected output after processing, and events that
are blocked (for validators). Each export is one Flow.StepExample:
// examples/step.ts
import type { Flow } from '@walkeros/core';
export const orderPasses: Flow.StepExample = {
title: 'Order passes',
description: 'An order event passes through unchanged.',
in: { name: 'order complete', data: { id: 'ORD-123' } },
out: [
['return', { event: { name: 'order complete', data: { id: 'ORD-123' } } }],
],
};
export const debugFiltered: Flow.StepExample = {
public: false,
description: 'A debug event is dropped.',
in: { name: 'debug test', data: { message: 'noise' } },
out: [['return', false]], // the transformer drops this event
};
in is a walkerOS event; out is the ['return', value] effect: the result
the transformer returns ({ event }), or false for an event it drops. A
failure example sets outcome instead (see Phase 6). Set title +
description for public examples; mark test-only fixtures with public: false.
src/examples/index.ts exports them with export * as step from './step';. See
using-step-examples for the Three
Type Zones.
2.3 Export via dev.ts
export * as schemas from './schemas';
export * as examples from './examples';
Gate: Examples Valid
- All example files compile (
npm run build) - Can trace: input event → expected output for each example
Phase 3: Scaffold
Template transformer: packages/server/transformers/fingerprint/ (a server
transformer). A platform-neutral transformer lives in packages/transformers/
(for example validate, ga4).
cp -r packages/server/transformers/fingerprint packages/server/transformers/[name]
cd packages/server/transformers/[name]
# Update package.json: name, description, repository.directory
Directory structure:
packages/server/transformers/[name]/
├── src/
│ ├── index.ts # Main export
│ ├── transformer.ts # Transformer implementation
│ ├── types.ts # Settings, Types interfaces
│ ├── dev.ts # Exports schemas and examples
│ ├── examples/
│ │ ├── index.ts # export * as step from './step'
│ │ └── step.ts # Step examples
│ ├── schemas/
│ │ └── index.ts # Zod schemas for settings
│ └── __tests__/ # stepExamples.test.ts, conformance.test.ts
├── package.json
├── tsconfig.json
├── tsup.config.ts
├── jest.config.mjs
└── README.md
Phase 4: walkerOS.json Convention
Every walkerOS package ships a walkerOS.json file for CDN-based schema
discovery.
Add walkerOS field to package.json
{
"walkerOS": {
"type": "transformer"
},
"keywords": ["walkerOS", "walkerOS-transformer", ...]
}
Use buildDev() in tsup.config.ts
Keep buildDev() in the defineConfig array (the template package's
tsup.config.ts has it):
import { buildDev } from '@walkeros/config/tsup';
// In defineConfig array:
buildDev(),
It builds src/dev.ts and writes dist/walkerOS.json from the schemas,
examples and hints it exports.
Hints (Optional)
If your transformer has capabilities, behaviors, or troubleshooting patterns not
obvious from schemas alone, add hints. See walkeros-writing-documentation
skill for full guidelines.
Create src/hints.ts:
import type { Hint } from '@walkeros/core';
export const hints: Hint.Hints = {
'validation-behavior': {
text: 'Describes how validation works. See settings schema for options.',
code: [{ lang: 'json', code: '{ "settings": { ... } }' }],
},
};
Export from src/dev.ts:
export * as schemas from './schemas';
export * as examples from './examples';
export { hints } from './hints';
Guidelines:
- Expand awareness: describe capabilities ("supports X, Y, Z"), don't prescribe one path
- Reference schemas and examples, don't duplicate them
- Verify every claim against actual implementation before publishing
- Key naming: kebab-case, group with prefixes (validation-*, enrichment-*, troubleshoot-*)
- Most transformers don't need hints; schemas and examples cover the common case
Gate: Convention Met
-
walkerOSfield in package.json withtype: "transformer" -
buildDev()in tsup.config.ts - Build generates
dist/walkerOS.json - Keywords include
walkerOSandwalkerOS-transformer
Runtime-only npm dependencies
A third-party npm dependency that cannot be ESM-bundled (uses __dirname,
ships a .node binary, etc.) needs no declaration: server bundles are traced
with nft, which copies every file the bundle reaches at runtime. Do not add a
walkerOS.bundle field to package.json: the CLI ignores it and warns
still declares walkerOS.bundle.<keys>; this is ignored. See
walkeros-using-cli: server bundles use nft tracing.
Phase 5: Implement
Now write code to transform examples as expected.
5.1 Define Types
See templates/validation/types.ts for the
pattern. Define Settings and Types interfaces.
5.2 Implement Transformer (Context Pattern)
Transformers use the context pattern - they receive a single context
object containing config, env, logger, id, and collector.
See templates/validation/index.ts for a complete implementation example.
Key patterns:
- Context destructuring: Extract
config,logger,idfrom init context - Settings schema, defaults inline: the Zod schema (see
templates/validation/schemas.ts) describes
the settings for
walkeros validate,walkerOS.jsonand the docs; the transformer applies its defaults inline and does not parse settings at runtime - Push receives pushContext: The
pushfunction gets event + push context - Return values:
{ event }(continue),void(passthrough),false(cancel on purpose: a deliberate drop, not a failure; the push answersok: true), or an array of results to fork the event (see understanding-transformers) - A failure is a throw: throw a
StepErrorfrom@walkeros/corewith acode.missing-inputis the code for an event or an ingest that lacks the input the transformer needs (do not drop it silently);configfor a missing setting.initthrows instead of returningfalseor returning early, and a transformer never logs a failure itself: the collector turns the throw intook: falsewith the transformer named and writes the one ERROR line. A transformer that throws a plain error fails with the codetransformer.
5.3 Export
src/index.ts:
export { transformerRedact } from './transformer';
export type { Settings, Types } from './types';
Gate: Implementation Compiles
-
npm run buildpasses -
npm run verify:touched -- <platform>-transformer-<name>passes (L1: typecheck + lint + test; the argument is the package name after@walkeros/,transformer-<name>for a platform-neutral one)
Phase 6: Test Against Examples
Verify implementation produces expected outputs.
See templates/validation/index.test.ts for a complete test suite showing:
createTransformerContext()helper - a typed init context for unit tests, defined in the test filecreatePushContext()helper - the same for the push context- Examples for test data - Don't hardcode test values
- Return value testing - Verify
event,void, orfalsereturns - Conformance -
conformance('transformer', transformer, options)from@walkeros/collector/dev(profiletransformer) runs it on a real collector, seepackages/server/transformers/fingerprint/src/__tests__/conformance.test.ts. A transformer needs nopage. A transformer whose input is the ingest and not an event declares its failure with amissing-inputexample (an event asin,out: [], anoutcomeand a description).
Gate: Tests Pass
-
npm run verify:touched -- <platform>-transformer-<name>passes (L1) - Tests verify against example outputs
Phase 7: Document
Follow the writing-documentation skill for:
- README structure and templates
- Quality checklist before publishing
Key requirements for transformer documentation:
- Use case description (validate/enrich/redact)
- Configuration options table
- Working code example with imports
- Installation instructions
- Link to website docs
Transformer-Specific Validation
Beyond
understanding-development
requirements (build, test, lint, no any):
-
dev.tsexportsschemas(withsettings;z.object({})when the step has none) andexamples - Multi-export package (two or more
walkerOS.exports):exportExamplesandexportSchemasindev.tslist exactly those exports, the default included, and everyexportSchemasentry hassettings; the build fails otherwise (see using-step-examples) - Examples include before/after event pairs
- Return values handle all cases (event, void, false)
- A failure is a thrown
StepError(missing-inputfor absent input),initthrows instead of returningfalse, conformance test passes - Tests use examples for assertions (not hardcoded values)
-
walkerOS.jsongenerated at build time -
walkerOSfield in package.json
Reference Files
| What | Where |
|---|---|
| Template | packages/server/transformers/fingerprint/ |
| Types | packages/core/src/types/transformer.ts |
| Chaining logic | packages/collector/src/transformer.ts |
Related Skills
- walkeros-understanding-transformers - Transformer interface and chaining
- walkeros-testing-strategy - Testing patterns and env mocking
- walkeros-writing-documentation - Documentation standards
Debugging walkerOS Events
Quick Diagnosis
| Symptom | Likely Cause | Check |
|---|---|---|
| No events at all | Source not initialized | Console for errors, verify startFlow() |
| Events fire but destination silent | Mapping mismatch | Event name matches mapping? |
| Partial data missing | Path doesn't exist | Log event structure, check nested paths |
| Consent blocking | Required consent not granted | Check consent config, grant consent |
| Destination error | Vendor API issue | Check network tab, vendor console |
Debugging Strategies
1. Console Logging
Log all events at collector level:
import { startFlow } from '@walkeros/collector';
const { elb } = await startFlow({
destinations: {
debug: {
code: {
type: 'debug',
config: {},
push: async (event) => {
console.log('[walkerOS Event]', {
name: event.name,
data: event.data,
context: event.context,
consent: event.consent,
timestamp: event.timestamp,
});
},
},
},
// ... other destinations
},
});
2. Network Tab Inspection
For destinations that make HTTP calls:
- Open DevTools → Network tab
- Filter by destination domain (e.g.,
google-analytics.com,facebook.com) - Trigger event
- Inspect request payload
What to look for:
- Request being made at all?
- Correct endpoint URL?
- Payload structure matches vendor spec?
3. Vendor Debug Tools
| Vendor | Debug Tool |
|---|---|
| GA4 | GA4 DebugView |
| Meta | Facebook Pixel Helper |
| Plausible | Plausible Dashboard real-time |
4. Simulate or Mock the Destination
To see what a destination's mapping produces without calling the vendor, simulate it: the vendor calls are recorded on a mock env instead of sent.
walkeros push flow.json -e event.json --simulate destination.ga4
config.mock ("mock": true in the destination's config) replaces only the
destination's push call with that value: the mapping still runs, init still
runs (it may load the vendor script and make its init calls), and the push
answers with the mock value instead of calling the vendor.
Common Issues
Event Name Mismatch
Problem: Event fires but destination doesn't receive it.
// Event pushed
elb({ name: 'product view', data: { id: 'P123' } });
// Mapping expects a different name
const mapping = {
Product: {
// Wrong: capital P
View: { name: 'view_item' }, // Wrong: capital V
},
};
Fix: Event names are case-sensitive. Use exact match:
const mapping = {
product: {
view: { name: 'view_item' },
},
};
Missing Nested Data
Problem: items array is empty in destination.
// Event structure
const event = {
name: 'order complete',
data: { value: 100 },
nested: [{ entity: 'product', data: { id: 'P1' } }],
};
// Mapping tries wrong path
const rule = {
data: {
map: {
items: {
loop: [
'data.items', // Wrong: nested is at root, not in data
{ map: { id: 'data.id' } },
],
},
},
},
};
Fix: Use correct path to nested array:
const items = {
loop: [
'nested', // Correct: root-level nested
{ map: { item_id: 'data.id' } },
],
};
Consent Blocking Events
Problem: Events not reaching destination.
Check 1: Does destination require consent?
// Destination config
const config = {
consent: { marketing: true }, // Requires marketing consent
};
Check 2: Is consent granted?
// Check current consent state
console.log(event.consent);
// Grant consent
elb('walker consent', { marketing: true });
Simulated Destination Recorded Nothing: Pending vs Consent Skip
walkeros push flow.json -e event.json --simulate destination.X prints why a
destination sent nothing, from the collector's own records (--json and MCP:
skipped on the result):
Line (skipped) | Meaning | Fix in simulate |
|---|---|---|
pending: waits for consent (require) ({ reason: 'pending', require: ['consent'] }) | require: ["consent"]: never started, no consent state exists | --consent '{"functional":true}' (MCP state.consent) |
skipped: consent (requires marketing; granted functional) ({ reason: 'consent', required, granted }) | Started, but config.consent denied the event (collector + event consent) | Grant the key in --consent or the event's consent |
mapping: <key>, no calls, skipped: consent (...) | That rule's own consent denied the event (on top of config.consent) | Grant the key in --consent or the event's consent |
mapping: none (skipped before mapping), no skipped line | Not consent: an event without name, a filter, a before chain stop, or an error | Check the event has a name, the --verbose logs, the before chain and the mapping rules |
require checks that consent state is present at all; config.consent checks
each event's keys. --consent is the collector's starting state: any value
clears the consent require, and its granted keys feed the consent check. A
require other than consent (e.g. user) cannot be seeded by simulate.
An event without name sends nothing and the simulation still answers
success: true with mapping: none (skipped before mapping) and no calls,
while a real push of it answers ok: false with Event name is required.
Does the CMP Source Send Consent?
Simulate the CMP source with its own example (in plus trigger as
{ content, trigger }). The consent command it issues is recorded as an elb
call:
success: true
source.usercentrics
call elb("walker consent",{"essential":true,"functional":true,"marketing":true})
no events
No call elb("walker consent",...) means the CMP source published nothing for
that input (for example explicitOnly with only implicit defaults), so
destinations waiting for consent stay pending in the real flow. --consent does
not help here: it is the collector's starting state and is never recorded as the
source's call. For a consent-gated source (the session source with
settings.consent), no events without --consent is expected; pass
--consent '{"functional":true}' (MCP state.consent) to see what it emits.
startFlow Throws Cannot read properties of undefined (reading 'config')
A destinations entry without code (for example { push, config } written
directly) makes startFlow throw this TypeError, which names neither the
destination nor code. Put the destination instance under code, as in
"Console Logging" above.
Vendor SDK Not Loaded
Problem: the push fails with vendor-not-loaded
(Vendor script failed to load), vendor-not-ready
(Vendor not ready: its global is missing) or vendor-timeout, or the vendor
receives nothing although the push answers done.
Cause: the vendor script was blocked, did not load in time, or was never
injected: a web destination that loads its vendor script (gtag, Meta, Plausible
and others) injects it only with config.loadScript: true. Without it nothing
is injected, and gtag's calls wait in dataLayer while the push answers done.
Fix: set "loadScript": true in the destination's config, or keep the
vendor's own tag on the page. A destination package loads its script through
loadVendor from @walkeros/web-core, which reports the codes above instead of
losing events (see
create-destination, Phase 7, "Scripts
through loadVendor").
Function Mapping Errors
Problem: the log shows
ERROR mapping fn failed { step: 'mapping.product view', code: 'mapping', ... error: "Cannot read properties of undefined (reading 'price')" },
and the destination receives the event without that field.
{
"data": {
"map": {
"value": {
"fn": "$code:(value, { event }) => event.data.product.price * 100"
}
}
}
}
The function fails when the event has no data.product. A function that returns
NaN instead (for example event.data.price * 100 on an event without a price)
drops the field with no log line at all.
Fix: Read the event through optional chaining and a fallback, which avoids both:
{
"data": {
"map": {
"value": {
"fn": "$code:(value, { event }) => (event.data?.product?.price ?? 0) * 100"
}
}
}
}
Debugging Checklist
When events aren't working:
- Console errors? Check browser console for exceptions
- Event pushed? Add debug destination to log all events
- Mapping matched? Verify entity/action names exactly match
- Data paths correct? Log full event structure, verify paths exist
- Consent granted? Check consent requirements and state
- SDK loaded? Verify vendor script loaded before push
- Network request? Check DevTools network tab
- Vendor receiving? Use vendor debug tools
Testing in Isolation
Run one destination on a real collector and record its vendor calls through
env instead of loading the vendor script:
import { startFlow } from '@walkeros/collector';
import { getEvent } from '@walkeros/core';
import { destinationGtag } from '@walkeros/web-destination-gtag';
// Record every gtag() call instead of loading gtag.js
const calls: unknown[][] = [];
const { elb } = await startFlow({
destinations: {
ga4: {
code: destinationGtag,
config: {
settings: { ga4: { measurementId: 'G-XXXXXXXXXX' } },
mapping: {
product: {
view: { name: 'view_item', data: { map: { value: 'data.price' } } },
},
},
},
env: {
window: {
dataLayer: [],
gtag: (...args: unknown[]) => calls.push(args),
},
},
},
},
});
const result = await elb(getEvent('product view'));
console.log(result.ok, calls);
// true [['config', 'G-XXXXXXXXXX', {}], ['event', 'view_item', { value: 420, send_to: 'G-XXXXXXXXXX' }]]
The CLI does the same for a flow file:
walkeros push flow.json -e event.json --simulate destination.ga4.
MCP Tool Debugging
When using walkerOS MCP tools, check _hints.warnings in tool responses for
diagnostic information:
flow_simulatewarns when 0 destinations exist or none received the event; a destination that sent nothing carriesskipped(pending on itsrequire, or a consent skip) and the warning says which. Passstate.consentto start a destination that waits for consent.stepis required (e.g."destination.gtag"). Source steps take a{ content, trigger? }event wherecontentis{ name, data }; sources, including@walkeros/source-demo, can be simulated this way. A source result shows the source's own walker commands aselbcalls (noverboseneeded) and its summary counts them (Source captured 0 events and 1 command);state.consentapplies to source steps too.flow_bundlewithstats: falsestill writes the bundle, but its result readssuccess: falseandBundle produced no outputflow_exampleswarns when no examples are found in the configpackage_searchbrowses the published step packages whose walkerOS metadata it could read. When the npm search fails, the catalog is empty and a warning says so; a second warning counts the npm packages without readable walkerOS metadata, which includes every package that is not a step, so it appears on every browse- Error responses carry
errorand, where the tool knows a recovery, ahint
From a terminal, walkeros diagnostics reports the CLI version, the app URL
(and whether it came from WALKEROS_APP_URL, the config file or the default),
and whether the app answers its health route.
Related Skills
- walkeros-understanding-flow - Event flow architecture
- walkeros-understanding-destinations - Destination interface
- walkeros-mapping-configuration - Mapping recipes
Mapping Configuration Recipes
Prerequisites
Read understanding-mapping first for core concepts.
Quick Reference
| I want to... | Use this pattern |
|---|---|
| Rename event | { name: 'new_name' } |
| Extract nested value | 'data.nested.value' |
| Set static value | { value: 'USD' } |
| Transform value | { fn: (value, { event }) => transform(event) } |
| Build object | { map: { key: 'source' } } |
| Process array | { loop: ['source', { map: {...} }] } |
| Gate by consent | { key: 'data.email', consent: { marketing: true } } |
| Gate one event type by consent | rule consent: { marketing: true } (on top of config.consent) |
| First defined value (fallback) | [{ key: 'data.sku' }, { key: 'data.id' }] |
| Add a field to a default rule | { extend: { data: { map: { x: 'path' } } } } |
| Drop a field from output | { remove: ['field.path'] } |
Common Recipes
GA4 / gtag
Product view → view_item:
product: {
view: {
name: 'view_item',
data: {
map: {
currency: { key: 'data.currency', value: 'EUR' },
value: 'data.price',
items: {
loop: [
'this', // the product is the event itself
{
map: {
item_id: 'data.id',
item_name: 'data.name',
item_category: 'data.category',
price: 'data.price',
quantity: { key: 'data.quantity', value: 1 },
},
},
],
},
},
},
},
}
Order complete → purchase:
order: {
complete: {
name: 'purchase',
data: {
map: {
transaction_id: 'data.id',
value: 'data.value',
currency: { key: 'data.currency', value: 'EUR' },
items: {
loop: [
'nested',
{
// only the products, not other nested entities
condition: (entity) =>
typeof entity === 'object' &&
entity !== null &&
'entity' in entity &&
entity.entity === 'product',
map: {
item_id: 'data.id',
item_name: 'data.name',
price: 'data.price',
quantity: { key: 'data.quantity', value: 1 },
},
},
],
},
},
},
},
}
Meta Pixel
Product view → ViewContent:
product: {
view: {
name: 'ViewContent',
data: {
map: {
content_ids: { set: ['data.id'] },
content_type: { value: 'product' },
content_name: 'data.name',
value: 'data.price',
currency: { key: 'data.currency', value: 'EUR' },
},
},
},
}
Order complete → Purchase:
order: {
complete: {
name: 'Purchase',
data: {
map: {
content_ids: {
loop: [
'nested',
{
condition: (entity) =>
typeof entity === 'object' &&
entity !== null &&
'entity' in entity &&
entity.entity === 'product',
key: 'data.id',
},
],
},
content_type: { value: 'product' },
value: 'data.value',
currency: { key: 'data.currency', value: 'EUR' },
num_items: {
fn: (value, { event }) =>
event.nested?.filter((entity) => entity?.entity === 'product')
.length ?? 0,
},
},
},
},
}
Custom API Destination
Transform to REST API format:
'*': {
'*': {
// A computed name goes in policy: a rule's name is a fixed string
policy: {
name: { fn: (value, { event }) => `${event.entity}_${event.action}` }, // page_view
},
data: {
map: {
eventName: 'name',
eventData: 'data',
userId: 'user.id',
sessionId: 'user.session',
timestamp: 'timestamp',
metadata: {
map: {
consent: 'consent',
globals: 'globals',
},
},
},
},
},
}
Conditional Mapping
Different mapping based on event data:
order: {
complete: [
// High-value orders get extra tracking
{
condition: (value, { event }) =>
typeof event.data?.value === 'number' && event.data.value > 500,
name: 'high_value_purchase',
data: {
map: {
value: 'data.value',
priority: { value: 'high' },
notify: { value: true },
},
},
},
// Standard orders
{
name: 'purchase',
data: { map: { value: 'data.value' } },
},
],
}
Patching Package-Shipped Rules (extend / remove)
@walkeros/transformer-ga4 ships default mapping rules in its
settings.mapping. Normally a user rule at the same key replaces the default in
full. Use extend or remove to patch instead.
Two layers, two keywords:
extendruns at init (config layer): deep-merges a partial rule onto the shipped default. Anullvalue clears an inherited field. Only a package that ships defaults merges it; on any other rulewalkeros validateand the bundle reject it (MAPPING_NO_OP).removeruns after evaluation (output layer): strips dotted paths from the final data payload. Applied last, always wins. It also works in a destination's own rules.
A rule with neither keyword replaces the default in full.
{
"settings": {
"mapping": {
"purchase": {
"extend": {
"data": { "map": { "affiliation": "params.ep.affiliation" } }
},
"remove": ["currency"]
}
}
}
}
This keeps all fields the transformer ships for purchase, adds affiliation,
and strips currency from the output.
For the full reference and the two-layer model, see Mapping.Rule docs.
Value[] Fallback Chains
At any value position, an array of values is a fallback chain: the first entry that resolves to a defined value wins.
{
"item_id": [
{ "key": "data.sku" },
{ "key": "data.id" },
{ "value": "unknown" }
]
}
Use this to try multiple source fields before falling back to a constant.
Consent-Gated Fields
Only include PII if consent granted:
user: {
login: {
name: 'login',
data: {
map: {
method: 'data.method',
// Only include email if marketing consent
email: {
key: 'user.email',
consent: { marketing: true },
},
// Only include user ID if functional consent
userId: {
key: 'user.id',
consent: { functional: true },
},
},
},
},
}
Wildcard Patterns
Catch-all for unmatched events:
// Any product action (a computed name goes in policy)
product: {
'*': {
policy: { name: { fn: (value, { event }) => `product_${event.action}` } },
data: 'data',
},
}
// A click on any entity without its own rule for it: with the product rule
// above, a product click matches product '*' instead
'*': {
click: {
name: 'element_click',
data: {
map: {
element_type: 'entity',
element_id: 'data.id',
},
},
},
}
Source-Side Mapping
A source's mapping uses the same entity-action rules on the events the source
pushes, before they reach the collector, for example to rename, edit (policy)
or drop them. Turning a raw request format into events is the source's own job
(see create-source).
// In a source's config
{
mapping: {
product: { click: { name: 'product view' } }, // rename
test: { '*': { ignore: true } }, // drop every test event
},
}
Debugging Tips
- Event not mapping? Check entity/action match exactly (case-sensitive)
- Data missing? Verify source path exists:
'data.nested.field' - Function errors? Add null checks:
e.data?.price ?? 0 - Array empty? Confirm
nestedarray exists and has items
Related Skills
- walkeros-understanding-mapping - Core mapping concepts
Reference:
- packages/core/src/mapping.ts - Implementation
- apps/quickstart/src/ - Validated examples
Calling walkerOS MCP Tools from Code
When to Use This Skill
You are an AI agent running inside a code-execution host, not issuing tool calls one at a time through the model. Examples:
- Claude Code in code-execution / "Code Mode"
- a Cloudflare Worker or other serverless sandbox
- an app's server-side sandbox that binds the walkerOS MCP transport
The pattern: bind the MCP transport once, call only the tools you need from
code, filter the verbose structuredContent payload in code, and return
only the distilled answer to the model. This suppresses tokens that the model
never needs to see (full catalogs, per-destination simulation dumps) and avoids
a chain of one-by-one tool calls.
This is the "Code Execution with MCP" / "Code Mode" pattern: treat MCP tools as functions you call in a sandbox, not as round-trips through the model context.
The Seven Pure Tools
These tools of the local walkerOS MCP server (@walkeros/mcp) are pure and
composable: same input, same output, no auth, no side effects in any account.
Safe to call freely from code.
| Tool (fully-qualified) | Purpose | Verbose output worth filtering |
|---|---|---|
walkeros-flow:flow_validate | Validate a flow or step config; return issues | small |
walkeros-flow:flow_simulate | Run a flow against an event, mock destination calls | yes (per-destination calls) |
walkeros-flow:flow_bundle | Bundle a flow config to a JS artifact | medium (bundle/meta) |
walkeros-flow:flow_examples | Fetch canonical flow/step examples | yes (many examples) |
walkeros-flow:flow_load | Load a flow config, or start a new one | medium (the whole config) |
walkeros-flow:package_search | Find packages by type/platform/keyword | yes (catalog) |
walkeros-flow:package_get | Read a package's schemas, hints, examples | yes (full schemas) |
To test an event against a flow, use walkeros-flow:flow_simulate: it runs the
event through the flow with destination calls mocked, so nothing leaves the
sandbox. Do not use walkeros-flow:flow_push for this (see below).
Always use the fully-qualified <server>:<tool> name (Agent Skills
best-practice) so the call is unambiguous when multiple MCP servers are bound.
walkeros-flow is the name the walkerOS plugin and the documented MCP config
give the local server; the hosted walkerOS MCP server is bound as walkeros.
Out of scope for this pattern
walkeros-flow:flow_push sends a real event to real destinations (real API
calls to live endpoints), so calling it "freely" would produce real, duplicated
sends; use walkeros-flow:flow_simulate to test without sending. The tools for
what the walkerOS app stores (projects, saved flows, deployments, secrets,
Observe) live on the hosted walkerOS MCP server and act with the person's
authorization there. Do not drive these tools from this code-execution
filtering pattern; they belong in an interactive, authorized session.
The Recommended Pattern
- Bind the transport once. Connect to the walkerOS MCP server at the top of your script, reuse the client for every call.
- Call only the tools you need. Don't enumerate; pick the minimal set.
- Read
structuredContent. Every tool returns an MCP envelope; the typed payload lives instructuredContent(thecontenttext block is for humans). - Filter in code. Reduce verbose payloads to the few fields that answer the question.
- Return the distilled answer. Hand the model one line, not the full dump.
// Pseudo-code in a code-execution host. `mcp` is the bound walkerOS client:
// call the raw tools and read `structuredContent` yourself.
const flow = await readFile('flow.json', 'utf8');
// 1. Validate first; bail with one line if the config is broken. Branch on
// `code`, never on `message`. `strict: true` turns warnings and skipped
// checks (`details.skipped`) into failures: `valid` is false, as CLI --strict.
const v = await mcp.call('walkeros-flow:flow_validate', {
type: 'flow',
input: flow,
strict: true,
});
const { valid, errors, warnings, details } = v.structuredContent;
if (!valid) {
const codes = [...errors, ...warnings].map((issue) => issue.code);
const skipped = (details.skipped ?? []).map((skip) => skip.code);
return `invalid flow: ${[...codes, ...skipped].join(', ')}`;
}
// 2. Simulate one event per destination: `step` names one target step
// ("destination.<name>"), and each answer is verbose (every vendor call).
const destinations = Object.keys(
JSON.parse(flow).flows.default.destinations ?? {},
);
const failed = [];
for (const name of destinations) {
const sim = await mcp.call('walkeros-flow:flow_simulate', {
configPath: 'flow.json',
step: `destination.${name}`,
event: { name: 'product add', data: { id: 'ers', price: 420 } },
});
// 3. Filter in code: keep only destinations that received nothing. The
// model never sees the per-destination call dump, only the verdict.
const received = Object.values(sim.structuredContent.destinations ?? {}).some(
(d) => d.received,
);
if (!received) failed.push(name);
}
// 4. Return one line.
return failed.length
? `simulate failed for: ${failed.join(', ')}`
: 'simulate ok for all destinations';
The same shape applies to walkeros-flow:package_search /
walkeros-flow:package_get (filter a catalog or a schema down to the one field
the user asked about) and walkeros-flow:flow_examples (pick the one matching
example).
Responsibility Boundary
PII handling, secret redaction, and sandbox isolation are the execution host's responsibility, not walkerOS's: walkerOS MCP tools are pure functions over the config and event you pass them.
Related Skills
- walkeros-using-cli - The validate / simulate / bundle verbs these tools wrap
- walkeros-understanding-flow - Flow architecture you are validating and simulating
- walkeros-mapping-configuration -
Mapping recipes for the configs you inspect with
package_get
walkerOS Testing Strategy
Overview
walkerOS uses a layered testing approach with built-in patterns for mocking and documentation sync. This skill ensures tests are reliable, efficient, and maintainable.
Core principle: Test real behavior using the env pattern, link to dev
examples, verify before claiming complete.
The Rules
Rule 1: Use env for Mocking, Not Jest
walkerOS has a built-in dependency injection pattern via env in context. This
is lighter than Jest mocks, enables documentation generation, and keeps tests in
sync with examples.
Wrong:
jest.mock('../ga4', () => ({ initGA4: jest.fn() }));
expect(initGA4).toHaveBeenCalledWith(...);
Right:
import { examples } from '../dev';
import { mockEnv } from '@walkeros/core';
const calls: Array<{ path: string[]; args: unknown[] }> = [];
const testEnv = mockEnv(examples.env.push, (path, args) => {
calls.push({ path, args });
});
await destination.push(event, { ...context, env: testEnv });
expect(calls).toContainEqual({
path: ['window', 'gtag'],
args: ['event', 'page_view', { page_title: 'Home' }],
});
Rule 2: Link Tests to dev Examples
The dev.ts export provides examples.env (the mock env and its simulation
paths) and examples.step (the step examples), plus examples.minimal for a
destination and examples.createTrigger for a source. Using these in tests
ensures documentation stays in sync.
Step Examples on a Real Collector
A destination's step examples run through runExamples, which starts a real
collector for each one and compares its out and outcome:
import type { StandardEventName } from '@walkeros/core';
import { runExamples } from '@walkeros/collector/dev';
import { page } from '@walkeros/web-core/dev';
import { destinationMeta } from '..';
import { examples } from '../dev';
// The standard events the destination maps, each with a public example
const claims: StandardEventName[] = [
'page view',
'product view',
'product add',
'order complete',
];
// The init example takes a config as `in`, not an event
const { init, ...events } = examples.step;
it('runs the step examples', async () => {
await runExamples('destination', destinationMeta, events, {
settings: examples.minimal.settings,
env: { push: examples.env.push, simulation: examples.env.simulation },
page,
claims,
});
});
(packages/web/destinations/meta/src/__tests__/conformance.test.ts runs this
test.)
A source's step examples run one test per example with
it.each(Object.entries(examples.step)): the test feeds each example's
trigger and in through examples.createTrigger and compares what the source
hands the collector with out (see
packages/server/sources/express/src/__tests__/stepExamples.test.ts).
See using-step-examples for the full lifecycle including the Three Type Zones and naming conventions.
Rule 3: Test Real Behavior, Not Mock Behavior
If you're asserting that a mock was called, you're testing the mock works, not the code.
Red flags:
expect(mockFn).toHaveBeenCalled()without verifying the mock produces real effects- Assertions on
*-mocktest IDs - Tests that pass when mock is present, fail when removed
Fix: Test what the code actually does. If external APIs must be mocked, verify the real API would receive correct data.
Rule 3b: Prove Delivery, Loss and Failures on a Real Collector
A test or repro that claims an event was delivered, lost or reported as failed
starts a real collector with startFlow and mocks only the network or the
vendor global (through env), never collector.push. A suite that mocks the
collector cannot see loss: a destination that returns early, swallows an error
or forgets an await passes it. Assert on what a user sees at default settings:
the push result (ok: false, the destination under failed with its reason),
the status counters, or the ERROR line the collector wrote.
Failure paths are examples too. A step package's failure modes (a missing
required setting, a failing transport, a vendor script that is not loaded yet)
are public: false step examples with an outcome (and respond for a vendor
answer), not only the happy path. A silent rule and a consent revocation get
examples as well.
Use the conformance kit. conformance(kind, step, options) from
@walkeros/collector/dev runs a destination, source, transformer or store on a
real collector against its step examples and checks the failure contract in one
call. The profile states what the step talks to: for destinations http,
sdk-env, web-sdk (a web SDK started in init), vendor-global or socket;
transformer and source for those kinds. A web step passes page from
@walkeros/web-core/dev, the browser it runs on; the browser profiles fail
without it. A destination also passes minimal: examples.minimal (from
src/examples/minimal.ts, only the settings the schema requires or init
refuses to start without): the kit starts the destination with the settings its
docs quickstart shows and requires one public example delivered. A package that
cannot have a minimal example passes withoutMinimal: '<reason>' instead; with
neither, or with an option that disagrees with the file, the test fails.
runExamples runs the examples alone.
Examples are the tests. A destination's public examples (one per standard
event it maps, from getEvent(name)) are asserted by runExamples with
claims; do not restate an example's input and assertion in a second test.
Every example runs as published: each function as source text, revived outside
its module, the way walkerOS.json ships it (the runner reads that text from
the code the test run compiled, not from the built dist/walkerOS.json). A
condition that calls an imported helper therefore fails the test, as it would
fail in a user's flow. Keep the tests for boundaries, failures and ordering. An
example's out is a literal taken from the vendor's documented request, never
the output of the code under test, otherwise the test passes by construction.
See using-step-examples.
Rule 4: Test First, Watch It Fail
If you didn't see the test fail, you don't know it tests the right thing.
Process:
- Write failing test
- Verify it fails for expected reason (missing feature, not typo)
- Write minimal code to pass
- Verify it passes
- Refactor if needed
Red flags:
- Test passes immediately when written
- Can't explain why test failed
- "I'll add tests later"
Rule 5: No Test-Only Methods in Production Code
Production classes shouldn't have methods only tests use.
Wrong:
class Session {
destroy() {
/* only used in tests */
}
}
Right:
// In test-utils/
export function cleanupSession(session: Session) { ... }
Rule 6: Verify Before Claiming Complete
"Should pass now" is not verification.
Process:
- Run the actual test command
- Read the output
- Confirm pass/fail count
- Only then claim status
When to Use Each Test Type
| Type | When to Add | Example |
|---|---|---|
| Integration | New usage pattern, new external API interaction, new data flow path | Collector → Destination → gtag() |
| Unit | Combinatorics, edge cases, pure function logic | Mapping variations, core utilities |
| Contract | Boundary validation | Destination output matches vendor API, source input validation |
Guideline: Integration tests prove things work when stuck together. Unit tests efficiently cover variations. Contract tests catch API drift.
Simulation Testing
Simulation testing uses the CLI push command with --simulate flags. The
collector does not export a simulate() function: simulation is a CLI concern
that maps to mock/disabled config properties at runtime.
CLI usage:
# Simulate a destination (mocks its push, captures API calls)
walkeros push flow.json -e '{"name":"page view"}' --simulate destination.ga4
# Simulate a source (captures events it pushes, disables all destinations)
walkeros push flow.json -e '{"content":"","trigger":{"type":"load"}}' --simulate source.browser
# Mock a destination with a specific return value
walkeros push flow.json -e event.json --mock destination.ga4='{"status":"ok"}'
The event needs a name: a destination simulation of an event without one sends
nothing and still answers success: true
(mapping: none (skipped before mapping), no calls), while a real push of it
answers ok: false with Event name is required.
Programmatic usage:
import { simulateDestination, simulateSource } from '@walkeros/cli';
// Simulate a destination: returns one Simulation.Result
const dest = await simulateDestination(
'flow.json',
{ name: 'page view' },
{ destinationId: 'ga4', ingest: { userAgent: 'Mozilla/5.0' } },
);
// dest.calls = recorded vendor calls ({ fn, args, ts })
// dest.mappingKey = matched rule, e.g. "page view"
// Simulate a source: input is { content, trigger? }
const src = await simulateSource(
'flow.json',
{ content: { name: 'page view' } },
{ sourceId: 'browser' },
);
// src.events = events captured from source env.push
simulateTransformer and simulateCollector follow the same shape. The CLI
push --simulate path collects these results into PushResult.simulations and
prints them with formatPushResult; programmatic push() does not simulate.
Key points:
--simulate destination.Xstarts only the target destination (no source, no other destination); the flow's transformers still start--simulate source.Xstarts only that source and captures its events at the collector boundary; they reach no destination. A source package declaringexamples.env.simulation(SQS, Pub/Sub pull) runs on its mock client, calls recorded- Simulate options:
--consent(starting collector consent),--command(destination runs a collector command instead of a push),--page-url(web source page),--ingest(request context). Programmatic:simulateDestination(config, event, { destinationId, consent, command }),simulateSource(config, input, { sourceId, pageUrl }) - Destination
/devenv.push is auto-loaded to provide mock globals (fakewindow.gtag, a mock SDK client, a mocksendServer), resolved per export (exportExamples). Without one, simulate refuses a package destination before the flow starts instead of calling the real vendor (an inlinecodestep has no package and runs as given). A flow store's mock env is injected when its package ships one; a store without one runs for real. - Returns
PushResultwithsimulations(oneSimulation.Resultper simulated step:events,calls,mappingKey?,skipped?,error?;skippedsays why a destination sent nothing:pendingon itsrequire, or aconsentskip) - Trace-mode limit: simulate injects the mock env before
init, so a client built ininitis recorded. Runtime trace mode wraps the env per push only (collector/src/destination.ts), so calls through a client built ininitare not recorded there. - The
mockEnv()and env pattern examples above remain correct for unit testing individual step functions directly
Package-Specific Approaches
| Package | Approach |
|---|---|
| core | Unit tests only - pure functions, no env needed |
| collector | Integration tests critical - input/output consistency is paramount |
| browser source | Maintain walker algorithm coverage |
| web destinations | Integration tests per unique pattern + unit tests for mappings, use env pattern |
| server destinations | Same as web destinations |
| cli/docker | Integration tests for spawn behavior, explore dev pattern to reduce duplication |
| sources | Contract tests for input validation, integration tests for event capture |
The env Pattern Deep Dive
How env Works
Each destination/source defines an env type that specifies external
dependencies:
// Destination-specific env type
export interface Env extends DestinationWeb.Env {
window: {
gtag: Gtag.Gtag;
dataLayer: unknown[];
};
document: {
createElement: (tagName: string) => HTMLElement;
head: { appendChild: (node: unknown) => void };
};
}
mockEnv() Function
The mockEnv() function from @walkeros/core creates a Proxy that intercepts
all function calls:
import { mockEnv } from '@walkeros/core';
const calls: Array<{ path: string[]; args: unknown[] }> = [];
const testEnv = mockEnv(examples.env.push, (path, args) => {
calls.push({ path, args });
// Optionally return a value
});
// Now use testEnv in your destination context
await destination.push(event, { ...context, env: testEnv });
// Assert on captured calls
expect(calls).toContainEqual({
path: ['window', 'gtag'],
args: ['event', 'purchase', expect.objectContaining({ value: 99.99 })],
});
Rules for a package's env
initkeeps an injected dependency. Every external dependency (SDK client, constructor or module,fetch,sendServer, auth client) is reached throughenv, andinitcreates one only when none is injected:env?.x ?? (await createX(...)). Overwriting an injected client means tests and simulate hit the real vendor.- Every mock env key is read. Each key a package's
examples.env.pushdeclares must be read asenv.<key>somewhere in the package'ssrc; otherwise the mock is dead and the real module runs (jest may pass only through__mocks__). simulationnames the request call and is pinned withexpectSimulationResolves(examples.env)from@walkeros/core/dev(see using-step-examples).
dev.ts Structure
Each package with external dependencies should have:
// src/dev.ts
export * as schemas from './schemas';
export * as examples from './examples';
// src/examples/index.ts (a destination)
export * as env from './env';
export * as step from './step'; // Step examples { in, out }
export { minimal } from './minimal';
Testing Sources with Injected env
Sources accept platform dependencies via env. Mock window, document, or
library imports by passing them through env instead of mocking globals.
Type the mock against the source's own Env type, not against the global
Window. A source that narrows Env.window to only the members it touches, as
the Cookiebot source does with CookiebotWindow, takes a mock of that shape
directly with no as unknown as Window cast:
import type { Elb } from '@walkeros/core';
import { createMockLogger } from '@walkeros/core';
import type { CookiebotWindow, Types } from '../types';
// Instead of mocking the global window:
const mockWindow: CookiebotWindow = {
Cookiebot: {
consent: { marketing: true, method: 'explicit' },
hasResponse: true,
},
addEventListener: jest.fn(),
removeEventListener: jest.fn(),
};
const elb: Elb.Fn = jest.fn();
const env: Types['env'] = {
push: elb,
command: elb,
elb,
window: mockWindow,
logger: createMockLogger(),
};
This pattern avoids global state pollution between tests and enables simulation
in non-browser environments. Most sources type Env.window as the full
Window & typeof globalThis; the mock stays cast-free only when the source
narrows it to the members it uses (the same declare-global + narrowed-Env
approach destinations use, see
create-destination, Phase 7, Key
Patterns 3). Server sources take their platform modules the same way
(env.express ?? express).
Red Flags - Stop and Fix
- Using
jest.mock()for internal modules whenenvpattern is available - Tests that don't import from
../dev - Assertions only checking mock call counts
- Proving delivery or loss with a mocked
collector.pushinstead ofstartFlow - A destination without a conformance test, a
silentexample or a failure example, or a failure example without adescription - Tests with extensive mock setup (>50% of test is setup)
- Test-only methods added to production classes
- Claiming tests pass without running them
Commands
Verification tiers (run from the repository root):
# L1, the default during a task: typecheck, lint, test for the touched package
npm run verify:touched -- core
npm run verify:touched -- web-destination-gtag
# L2, at plan completion: only packages affected since origin/main
git fetch origin main --depth=1
npm run verify:affected
# L3, before pushing or marking PR-ready: critical path + affected
npm run test:smoke
# Single test file (still useful while iterating)
cd packages/<name> && npm run test -- path/to/file.test.ts
# Watch mode (single package)
cd packages/<name> && npm run test -- --watch
Avoid bare npm run test at root inside per-task steps. That is L4 (full suite,
10-15 min) and is reserved for plan completion when the plan touched shared
infra, or for explicit user request.
Reading a Step Back from the Collector
When a test calls a step's raw push directly through the collector bag
(collector.sources.X.push, collector.destinations.X.push, etc.), use the
typed accessors from @walkeros/core instead of casting:
import { Source } from '@walkeros/core';
const src = Source.getSource<TestSourceTypes>(collector, 'testSource');
await src.push({ method: 'GET', path: '/api/data' });
Available helpers: Source.getSource, Destination.getDestination,
Transformer.getTransformer, Store.getStore. Each accepts an optional type
parameter to recover the per-step generic that the bag's index signature erases
on read. Each throws <Kind> not found: <id> when the id is unknown.
Do not write collector.sources.X.push as any or
collector.sources.X.push as (rawData: ...) => Promise<...>. The accessor
exists exactly to remove that boundary cast.
Shared Destination Test Harness
The conformance kit (conformance and runExamples from
@walkeros/collector/dev) is the shared harness for destination tests: the
failure contract, missing settings, the minimal example and every step example.
When writing a new destination test:
- Keep destination-specific behavior (vendor-specific batch shapes, ordering, edge cases the examples cannot state) in the test file.
- Mirror the shape of an existing destination of the same family (web vs
server), for example
packages/web/destinations/meta/src/__tests__/orpackages/server/destinations/pinterest/src/__tests__/. - Do NOT invent a new private harness inside the destination. If the kit does not cover a case, flag it.
Related Skills
- walkeros-understanding-development - Development conventions and workflow
Reference:
- AGENT.md - Development guide
Understanding walkerOS Destinations
Overview
Destinations receive processed events from the collector and deliver them to third-party tools (analytics, marketing, data warehouses).
Core principle: Destinations transform and deliver. They don't capture or process: that's sources and collector.
Destination Interface
See packages/core/src/types/destination.ts for canonical interface.
| Method | Purpose | Required |
|---|---|---|
init(context) | Load scripts, authenticate | Optional |
push(event, context) | Transform and send event | Required |
pushBatch(batch, context) | Batch processing | Optional |
destroy(context) | Cleanup on shutdown | Optional |
config | Settings, mapping, consent | Required |
destroy?: DestroyFn: optional cleanup method. Called during
command('shutdown'). Use to close DB connections, flush buffers, or release
SDK clients. Receives { id, config, env, logger }.
The env Pattern
Destinations use dependency injection via env for external APIs. This enables
testing without mocking.
// Destination defines its env type
export interface Env extends DestinationWeb.Env {
window: {
gtag: Gtag.Gtag;
dataLayer: unknown[];
};
}
// Destination uses env, not globals
async function push(event, context) {
const { env } = context;
env.window.gtag('event', mappedName, mappedData);
}
Testing with env
REQUIRED SKILL: See testing-strategy for full testing patterns.
import { mockEnv } from '@walkeros/core';
import { examples } from '../dev';
const calls: Array<{ path: string[]; args: unknown[] }> = [];
const testEnv = mockEnv(examples.env.push, (path, args) => {
calls.push({ path, args });
});
await destination.push(event, { ...context, env: testEnv });
expect(calls).toContainEqual({
path: ['window', 'gtag'],
args: ['event', 'purchase', expect.any(Object)],
});
The env as observability seam
The env pattern is also the interception seam for trace-level vendor-call
capture. At trace level the collector may attach an observe recorder under
the observe key of the env, and @walkeros/web-core's getEnv wraps the
callables the destination declared as observable, recording each call. A
destination that reaches its vendor through getEnv(env) is automatically
observable; one that reaches a global directly, bypassing env, is not. This is
one more reason to route every external call through env.
Rules that keep the seam intact:
initkeeps an injected env. A client built ininit(SDK client, auth client, writer) is taken fromenvwhen present and created only otherwise (env?.BigQuery ?? BigQuery). Every key the mock env declares must be read asenv.<key>insrc.simulationnames the request call (sendServer,call:PubSub.topic.publishMessage,call:JSONWriter.appendRows), not a constructor. The one recorder isobserveEnvin@walkeros/core; it navigates constructors, factories and Promises, and records only the leaf.- Trace-mode limit. CLI and MCP simulate inject the mock env before
init, so a client built ininitis recorded. Runtime trace mode wraps the env per push only, so calls through a client built ininitare not recorded there. - Client IP and user agent. Server conversion-API destinations (Meta,
TikTok, Snapchat, Pinterest, Bing, Reddit, Criteo, X, Data Manager, Piwik PRO)
expose
ipanduserAgentsettings (Mapping.Value | false), defaulting to['ingest.ip', 'event.user.ip']and['ingest.userAgent', 'event.user.userAgent'], resolved withgetMappingValue(createMappingRoot(ingest, event), setting, { collector }).falsedisables, an explicit mapped vendor field wins, each package carries its own few lines (no shared helper).
Destination Config
config: {
settings: { /* destination-specific */ },
mapping: { /* event transformation rules */ },
data: { /* global data mapping */ },
consent: { /* required consent states */ },
policy: { /* processing rules */ },
queue: boolean, // when added later, receive the run's earlier events (default true)
timeout: number, // delivery timeout in ms (default 10000)
queueMax: number, // consent-queued events cap (default 1000)
dlqMax: number, // dead-letter queue cap (default 100)
}
Buffer bounds
Each destination keeps two internal buffers: queuePush (consent-denied events)
and dlq (failed pushes). Both are size-bounded with FIFO drop-oldest eviction.
Defaults: queueMax: 1000, dlqMax: 100. Set either on a destination's config
to override per destination. Operators read drop counts from
collector.status.dropped[stepId('destination', id)]?.queue (consent-denied
evictions) and ?.dlq (DLQ evictions). Build the key with stepId() from
@walkeros/core. Point-in-time sizes stay on
collector.status.destinations[id].queuePushSize / dlqSize.
queuePush drains back into the destination once consent allows delivery. The
DLQ is never replayed: an event that reaches it is gone once it is evicted.
Delivery timeout
config.timeout bounds one delivery. A push or pushBatch that has not
settled within it is abandoned, its events are routed to the DLQ and failed is
incremented. The default is 10000 ms, applied whenever timeout is 0 or
unset, so one slow destination never wedges the collector push.
A destination's own client timeout has to fit inside it, retry delays
included. Many SDK defaults are longer (30 s in several HTTP clients). With a
longer one, the collector stops waiting at 10 s, dead-letters the batch and
counts it lost, then the request lands anyway a few seconds later. Those events
are both counted lost and actually written. Derive the client's per-attempt
timeout from config.timeout minus what the retry delays can take, split across
the attempts, so the last attempt finishes before the race does.
The same bound applies at shutdown, where the collector races each pending batch flush against a fixed 5000 ms step timeout before destroying the destination. A destination whose retries need longer than that loses them on shutdown.
Outcomes and failure reasons
A push ends in one place per destination in the PushResult: done, queued
(reason consent, batch or held) or failed. A failed entry carries a
reason { code, message, retryable, status? } that the collector derives from
the thrown value: a StepError keeps its code; a TypeError, RangeError or
ReferenceError is runtime (not retryable), except Node's fetch failed with
a cause.code; any other throw is transport. A failed browser fetch is a
TypeError too, so throw network failures as a StepError (sendWeb and
sendServer do). Codes the collector assigns itself: timeout, not-ready (a
destination whose init could not start or returned false), breaker-open,
blocked (held past one retry window), queue-full, mapping (a rule
condition threw), transformer. Codes a destination throws: config,
missing-input, vendor-not-loaded, vendor-not-ready, vendor-timeout,
transport. retryable decides whether a server source asks the producer to
resend (acknowledge(result) from @walkeros/core: ack, retry or
reject); it never reads message text.
Unmet consent is never a failure. A destination waiting for consent answers
queued with the reason consent; a server flow built by the CLI skips a
consent-denied event instead (consentDenied: 'skip', counted in
status.destinations[id].skipped, a debug line). A rule with ignore: true
is a skip as well, not a delivery.
Failed means thrown. A destination reports a failure only by throwing a
StepError; it never returns early, never returns false from init, never
logs the failure itself, and awaits every call it makes. The collector writes
one ERROR line per (step, code, status) and minute. Every destination honours
rule.silent (no event of its own, side effects still run) and ships a silent
example.
The conformance kit. conformance('destination', destination, options) from
@walkeros/collector/dev runs a destination on a real collector against its
step examples and fails with every broken item of the contract at once. Pick the
profile by what the destination talks to: http, sdk-env, web-sdk (a web
SDK started in init), vendor-global, socket. A web destination also passes
page from @walkeros/web-core/dev. Every destination test passes minimal,
the package's examples.minimal published from src/examples/minimal.ts (a
package with several exports passes exportName and exportExamples, one
src/<export>/examples/minimal.ts each), or withoutMinimal: '<reason>' when
the package publishes none. The kit composes the minimal example into the
quickstart the docs page shows and requires it delivered. A test with neither
option fails, and so does one whose option disagrees with the published file.
See create-destination.
Batch scheduling
Set config.batch on a destination (with pushBatch implemented) to batch
all of its events into one shared buffer. No '* *' wildcard mapping rule
is needed. The configuration shape is batch?: number | { wait?, size?, age? }
at both the destination-config layer and the mapping-rule layer.
pushBatch is the gate. The collector engages batching only when
destination.pushBatch exists. Set config.batch on a destination that lacks
it and every event is sent one at a time; the collector logs one ERROR at
registration naming the setting, and walkeros validate and the bundle fail
(BATCH_WITHOUT_PUSH_BATCH) for inline code. When adding batching to a
destination, implementing pushBatch is the change; config.batch alone does
not batch. A batched event answers queued with the reason batch, so a server
source acknowledges the producer at queue time; a failed flush is visible (ERROR
line, failed counter, DLQ) but is not redelivered. walker flush flushes every
pending batch.
wait(ms): debounce window. The timer resets on every push. A bare number,batch: 1000, is shorthand for{ wait: 1000 }.size: hard count cap. Default1000. Flushes immediately when reached.age(ms): hard age cap since the first entry of the current window. Default30000. Prevents debounce starvation under sustained load.
A mapping rule's own batch splits that entity-action into its own buffer and
overrides config.batch per field (rule ?? config ?? default). To batch only
specific events, omit config.batch and set batch on those rules. Pending
batches flush on shutdown.
Per-event metadata
Batch.entries[] carries per-event
{ event, ingest?, respond?, rule?, data? }. Destinations that need per-event
request IDs or HTTP responses (BigQuery, mParticle, HubSpot) should read
entries instead of assuming all events in a batch share one ingest.
batch.events and batch.data are derived views of entries.
Failure handling
pushBatch has three return outcomes:
- Resolves
void: the whole batch succeeded; every entry counts as delivered. - Throws or rejects: the whole batch failed. Every entry is routed to the
destination's
dlqandstatus.destinations[id].failedis incremented by the batch size. - Resolves a
BatchOutcome: partial failure.{ failed: [{ index, error? }] }names the entries that did not make it, by index intobatch.entries. Only those are DLQ'd; the rest count as delivered. Use this when the vendor reports per-row results (BigQuery'srowErrorsis the reference case), so a later DLQ retry does not rewrite rows that already landed.
A BatchOutcome is for sinks that report per row, and for nothing else. A
sink that succeeds or fails as a whole (one SQL INSERT, one file write, one
publish) must never report an all-failed outcome in place of throwing. Only a
throw can count as a failure on the breaker (a retryable one, such as a network
error, a timeout or a 429 or 5xx answer), and only a flush with at least one
delivered entry records a success, so an outcome where nothing succeeded records
neither: a total outage passes the breaker unseen and it never opens. A
whole-batch sink has exactly two answers: resolve void or throw.
Batching, the delivery timeout, the DLQ and the breaker are the collector's;
per-call retry is the vendor SDK's (BigQuery, Kafka, HubSpot and the AWS SDK
each have their own backoff semantics). Where a destination relies on that SDK
retry, it keeps the attempts and their delays inside config.timeout, and makes
a repeat safe through an idempotency key or a deduplication token computed once
and reused across attempts. Without that, a retry of a delivery the vendor had
already accepted duplicates data. Counters (count, out) are bumped only
after a successful flush.
Three rules every destination follows. First, no retry loops, backoff or DLQ
code of its own: each request is sent once through the vendor SDK, whose
built-in retry stays at its defaults, because a standard retry belongs in the
collector, not in each package. Second, a server destination's init is
offline: it validates the config and builds clients, but makes no network call,
loads no credentials and creates no remote resources; connections (Kafka,
BigQuery, S3, Sheets, SQLite) open at the first use, memoized and retried after
a failure, and provisioning belongs to setup. Third, every failure, thrown or
reported in a BatchOutcome, is a StepError that names the cause (and, for a
configuration error, the fix) and carries code (the vendor error name or a
package code), status (the HTTP status when the vendor answered) and
retryable. Take retryable from the vendor SDK's own public classifier where
one exists, else from documented codes or statuses, and keep the SDK error
reachable as cause. @walkeros/server-destination-aws is the reference
implementation.
Operators also see status.destinations[id].inFlightBatch: the number of events
buffered but not yet delivered.
Out-of-band errors from an EventEmitter SDK
tryCatchAsync only catches errors on the awaited push/pushBatch path. If a
destination owns an SDK object that is an EventEmitter (a BigQuery
StreamConnection, a Redis client, a Kafka producer), that object can emit
'error' on a detached tick with no awaiter. An EventEmitter that emits
'error' with zero listeners throws synchronously and crashes the process.
A destination that owns such an object MUST attach an 'error' listener and
route the error through context.reportError, which is available on every step
context:
context.reportError(err)(no event): a connection-level error between pushes. Logs (redacted) and bumpsstatus.connectionErrors[stepId]. Does not count as a failed event.context.reportError(err, event): a specific event was lost. Routes it to the DLQ and bumpsfailed, exactly like an in-band push failure.
reportError is guarded and must never be called in a way that throws back into
the emitter tick. The Pub/Sub pull source (sources/gcp/src/pubsub/pull) is the
reference for attaching an 'error' listener. The runner's process guards are a
backstop, but the listener is what gives clean DLQ routing, attribution, and
redaction. Optionally pair this with the breaker config so a persistently
broken transport stops retrying every event.
Require vs Consent
Two separate mechanisms control when destinations receive events:
| Mechanism | Purpose | Scope | Effect |
|---|---|---|---|
require | Delay initialization | Whole destination | Destination stays in pending until all required events fire (e.g., walker consent) |
consent | Filter events | Per-event | Events without matching consent are queued (skipped in a server flow built by the CLI) |
Require gates the destination lifecycle. A destination with
require: ["consent"] does not exist in the collector until consent state is
present: a "walker consent" event, or starting consent (startFlow's
consent, which walkeros push --simulate sets with --consent). Until then,
events are queued internally. require takes effect only in the step
definition's config (destinations.<id>.config in flow.json,
{ code, config } in startFlow); a require in the destination code's own
config is ignored today, and the destination receives events at once.
Consent gates individual event delivery. A destination with
consent: { marketing: true } only receives events where the collector's
consent state (or the event's own consent field) includes
{ marketing: true }.
State on flush (consent queue): When consent is granted, the consent-queued
events are delivered in their original order. Each one gets the current
consent, its user is merged with the collector's (the event's own values
win), and it keeps the globals it was captured with. Events that still lack
the required consent stay queued. See
How queued events are processed.
Events replayed from the run's buffer, to a require-pending destination when
it activates or to a destination added later, behave differently today: they
carry the user set after they were captured, and only an event that brought its
own user keeps its values.
Both can be combined:
{
"config": {
"require": ["consent"],
"consent": { "marketing": true }
}
}
This means: don't initialize until consent fires, then only accept events with
marketing consent. The state that satisfies require reaches on() before the
destination's queued events are pushed.
Simulation impact: a destination whose require is not met is reported as
skipped with the reason pending by flow_simulate and
walkeros push --simulate. Pass the starting consent to simulate it:
state.consent in flow_simulate, --consent on the CLI.
Policy
Policy shapes the event before its data is mapped; the rule was selected earlier, as the event entered the destination. Defined at config level (applies to all events) or rule level (applies to specific events):
{
"config": {
"policy": {
"user_data.email": {
"key": "user.email",
"consent": { "marketing": true }
}
}
}
}
Policy supports consent-gated field injection: fields are only added when the required consent is present in the event.
Destination Paths
| Type | Path | Examples |
|---|---|---|
| Web | packages/web/destinations/ | gtag, meta, api, piwikpro, plausible |
| Server | packages/server/destinations/ | aws, gcp, meta |
Template Destination
Use as starting point: packages/web/destinations/plausible/
Transformer Wiring
Destinations can wire to their own transformer chain via the before property.
It runs for this destination only, after the collector chain (collector.next,
shared by every destination); a stop in it skips only this destination:
destinations: {
gtag: {
code: destinationGtag,
before: 'redact' // First transformer to run before this destination
}
}
The transformer chain runs after collector enrichment, before the destination receives events. Each destination can have its own chain. See understanding-transformers for chain details.
Consent: Two-Layer Pattern
Destinations that integrate vendor SDKs typically need two consent layers:
Layer 1: config.consent gates walkerOS event delivery. If consent is not
granted, events don't reach the destination. This is the primary barrier.
Layer 2: on('consent') controls vendor SDK internals. Even when walkerOS
stops sending events, the vendor SDK may still run its own behaviors (DOM
capture, polling, fetching configs). Use on('consent') to pause/resume these.
import { getGrantedConsent } from '@walkeros/core';
on(type, { config, collector }) {
if (type !== 'consent' || !config.consent) return;
// The rule of the delivery gate: one granted required key is enough.
// Derive from config.consent, don't hardcode consent names.
const granted =
getGrantedConsent(config.consent, collector.consent) !== false;
vendorSdk.setOptOut(!granted);
}
Complete consent compliance needs both layers: config.consent prevents data
flow, on('consent') prevents vendor SDK side effects.
on('consent') runs before the first push on every activation path (require,
walker destination at runtime, state passed to walker run, state set before
run) and is delivered exactly once per state change. It may be async: the
collector awaits it, bounded by config.timeout, and events pushed meanwhile
answer queued with the reason held. A handler that rejects or does not
settle in that time writes an ERROR line (consent not applied) and holds the
destination's events for one more config.timeout. A push after that window
retries a failed delivery; once a delivery succeeds, the held events are
delivered. If none has succeeded by the first push after the window, the held
events and every later event with the required consent fail with the reason
blocked until one does: the push answers ok: false, they are counted in
failed and kept in the DLQ, an ERROR Push failed line is written, and they
are not sent when the destination recovers. A handler that never settles keeps
the destination blocked. Events still held at the next walker run are
discarded. Lifecycle handlers such as on('run') are awaited too, bounded by
the same timeout, and never hold events. elb commands and startFlow resolve
only after the destinations' on() handlers for that command have settled or
timed out. See
Gate a destination with consent.
Response Delegation (env.respond)
Destinations can customize HTTP responses by calling
context.env.respond?.({ body, status?, headers? }). This is useful for
destinations that need to signal success/failure back to the HTTP caller. First
call wins (idempotent). The respond function is optional, only present when the
source provides one.
Setup (optional)
Destinations can implement an optional setup() lifecycle to provision external
resources, for example a BigQuery dataset and table, a Pub/Sub topic, or a
warehouse schema. Setup is never invoked by the runtime, push, init, or
deploy. It runs only when an operator explicitly types
walkeros setup destination.<name>.
The signature is
(ctx: LifecycleContext<Config<T>, Env<T>>) => Promise<unknown>, where
LifecycleContext carries { id, config, env, logger }. Idempotency is the
package's responsibility: the framework adds no opinion. Use
resolveSetup(ctx.config.setup, DEFAULTS) from @walkeros/core to normalize
the boolean | object shape into a concrete options object.
See walkeros-create-destination,
walkeros-understanding-sources,
walkeros-understanding-stores, and
the walkeros setup CLI documentation for the authoring template and operator
workflow.
Related Skills
- walkeros-understanding-flow - How destinations fit in architecture
- walkeros-understanding-mapping - Configure transformations
- walkeros-create-destination - Create new destination
Source Files:
- packages/core/src/types/destination.ts - Interface
Package READMEs:
- packages/web/destinations/gtag/README.md - gtag example
- packages/web/destinations/plausible/README.md - Plausible (template)
Documentation:
- Website: Destinations - Overview
- Website: Create Your Own - Guide
Understanding walkerOS Development
Overview
walkerOS follows extreme programming principles with strict conventions. Read this skill before writing any code.
Core principle: DRY, KISS, YAGNI. Test first. Verify before claiming complete.
Commands
| Command | Purpose |
|---|---|
npm install | Install all dependencies |
npm run dev | Watch mode for all packages |
npm run build | Build all packages |
npm run format | Prettier formatting |
Verification cheatsheet:
- While iterating (L1, touched package):
npm run verify:touched -- <pkg> - At plan completion (L2, affected since
origin/main):npm run verify:affected
XP Principles (Non-Negotiable)
| Principle | In Practice |
|---|---|
| DRY | Use @walkeros/core utilities, don't reimplement |
| KISS | Minimal code to solve the problem |
| YAGNI | Only implement what's requested |
| TDD | Test first, watch it fail, then implement |
No any | Never use any in production code (tests are exception) |
Folder Structure
packages/
├── core/ # Platform-agnostic types, utilities, schemas
├── collector/ # Central event processing engine
├── config/ # Shared config (eslint, jest, tsconfig, tsup)
├── cli/ # walkeros CLI (validate, bundle, push, setup)
├── runner/ # Runs prebuilt flow bundles
├── mcps/ # MCP servers (mcp, source-browser)
├── transformers/ # Neutral transformers: validate, ga4, demo
├── web/
│ ├── core/ # Web-specific utilities
│ ├── sources/ # browser, dataLayer, session, cmps/
│ └── destinations/ # gtag, meta, api, piwikpro, plausible, ...
└── server/
├── core/ # Server-specific utilities
├── sources/ # express, fetch, aws, gcp
├── destinations/ # aws, gcp, meta, ...
├── transformers/ # bot, fingerprint, file
└── stores/ # fs, s3, gcs, sheets
apps/
├── walkerjs/ # Zero-config browser file for GTM
├── quickstart/ # Code examples (source of truth for patterns)
├── explorer/ # React components for the docs and exploration
├── cli/ # The walkeros npm package (CLI entry)
├── storybook-addon/ # walkerOS Storybook addon
├── scripts/ # Repository scripts (docs and skills checks)
└── demos/ # Demo applications
Core Package Usage
Always import from @walkeros/core:
// Types
import type { WalkerOS } from '@walkeros/core';
// Utilities
import {
getEvent,
createEvent, // Event creation
getMappingEvent,
getMappingValue, // Transformations
isString,
isObject,
isDefined, // Type checking
assign,
clone, // Object operations
tryCatch,
tryCatchAsync, // Error handling
} from '@walkeros/core';
Config package for shared tooling:
- ESLint config:
@walkeros/config/eslint - Jest config:
@walkeros/config/jest - TSConfig presets:
@walkeros/config/tsconfig/{base,web,node}.json - Tsup config:
@walkeros/config/tsup
Editing core Config types
Core component configs live in two places:
- TS interface:
packages/core/src/types/{destination,source,transformer,store,collector}.ts - Zod schema:
packages/core/src/schemas/{destination,source,transformer,store,collector}.ts
Both are hand-written and mirror each other. TS stays authoritative because
Zod's inferencer collapses recursive types (Routes, MatchExpression,
Value) to unknown. Zod drives runtime validation, JSON Schema emission, and
website Configuration reference tables.
When adding, renaming, or removing a Config field, update BOTH files. A
compile-time drift guard at
packages/core/src/schemas/__tests__/config-drift.test-d.ts fails tsc if the
key sets diverge. The guard checks keys only; value types may differ (recursion,
generic slots). Run npm run verify:touched -- core to verify.
Error visibility contract
Top-level boundaries in the collector (createPush in push.ts,
createCommand in command.ts) wrap their inner pipeline in tryCatchAsync.
The onError callback MUST do two things:
- log one structured ERROR line with the context an operator needs to find the cause, and no payload: the event name for push, the command name for command, never the event, the ingest or the command's data, which can carry personal data,
- increment
collector.status.failed.
An empty onError is a defect: it swallows the exception, returns
{ ok: false } silently, and leaves the operator blind. Use
packages/collector/src/push.ts and packages/collector/src/command.ts as the
canonical pattern.
Two categories of caught error:
- Internal walkerOS pipeline failures (push, command, a throwing rule
conditioninpush.ts, source factory / init / queueOn flush insource.ts, transformer init intransformer.ts, destination init indestination.ts): log ANDstatus.failed++. - User-supplied callbacks (a mapping value's
condition/fn/validate,onsubscriptions inon.ts): log only.status.failedstays a pipeline-health signal; user-code visibility goes via logs.
For invariant violations or operator-initiated aborts that must crash the host
process, throw FatalError (exported from @walkeros/core). FatalError
bypasses every boundary catch in both categories so a supervisor can terminate
cleanly. Standard Error is absorbed, logged, and (for category 1) counted.
The log message verb identifies the site: 'mapping condition failed',
'source factory failed', 'transformer init failed', 'on callback failed',
etc. Operators grep for the verb. No kind field is required except in on.ts,
where seven sites share one verb and disambiguate via a typed kind field on
the structured payload.
Platform boundaries
Every package is one platform: web, server, or neutral (usable on both, like
@walkeros/core and @walkeros/collector). Neutral code ships inside walker.js
and runs on a server too, so it uses neither platform's APIs.
- Where it is declared. A step package sets
walkerOS.platformin itspackage.json:["web"],["server"], or both for neutral. Every other package and published app is in thedeclaredtable ofpackages/config/scripts/check-platform.mjs. A package declared nowhere fails the check. - Presets.
base.jsonis neutral (lib: es2020),web.jsonaddsdom,node.jsonises2022. All settypestonodeandjest, so no other@typespackage can bring the DOM back in. A package tsconfig may add only an allowlisted ambient vendor type (gtag.js,facebook-pixel). - What each may use. Web: browser globals, no Node globals or builtins
(
Buffer,process,fs,node:*). Server: Node, no browser globals. Neutral: neither, and noglobalThis, nojsdomimport and no/// <reference lib or types>in any file, tests included. ESLint holds these lines where TypeScript cannot:@walkeros/config/eslint/web,/node(itsnodePlatformblock) and/neutral. - API floor. Neutral and web code may use ES2020 APIs at most; ES2021 and
later (for example the
causeargument ofnew Error) fail typecheck, so errors assigncausethemselves. - Needing the other platform. A neutral package that needs something only
the web has defines a small interface in
@walkeros/core(a port), web-core implements it, and the web caller passes it in explicitly. Examples:ObserveSlot(pageObserveSlot()from@walkeros/web-core) andFlow.StepPage(pagefrom@walkeros/web-core/dev). Node-only code of core lives in@walkeros/core/node. - Check.
npm run check:platformfrom the walkerOS root fails for any drift in a preset,lib,types, eslint config or declaration. CI runs it on every run, outside the turbo cache.
Testing
REQUIRED SKILL: Use testing-strategy for detailed testing patterns.
Quick reference:
- Use
envpattern for mocking (not Jest mocks) - Import from
dev.tsfor examples - Test first, watch it fail
- Verify before claiming complete
Related Skills
- walkeros-testing-strategy - Testing patterns and env mocking
Source Files:
- packages/core/ - Core utilities
- packages/config/ - Shared configuration
- apps/quickstart/ - Validated examples
Understanding walkerOS Events
Overview
walkerOS events are self-describing, stateless, vendor-agnostic data structures. They capture user interactions in a standardized format that can be transformed for any destination.
Core principle: Events describe what happened, not where it goes. They are stateless, self-describing and industry-agnostic.
Entity-Action Naming (Critical)
STRICT REQUIREMENT: All events use "entity action" format with space separation.
// Correct
'page view';
'product add';
'order complete';
'button click';
// Wrong
'page_view'; // underscore
'pageview'; // no separator
'purchase'; // no entity
'add_to_cart'; // wrong format
Parsing: const [entity, action] = event.split(' ')
- Entity: Noun (page, product, user, order, button)
- Action: Verb (view, add, complete, click, login)
Event Properties
See
packages/core/src/types/walkeros.ts
for canonical types (Event interface, plus event helpers in event.ts).
| Property | Type | Purpose | Example |
|---|---|---|---|
name | string | "entity action" format | "product view" |
data | object | Entity-specific properties | { id: "P123", price: 99 } |
context | object | State/environment info (optional) | { stage: ["checkout", 1] } |
globals | object | Global properties | { language: "en" } |
user | object | User identification | { id: "user123" } |
nested | array | Related entities (optional) | [{ entity: "product", data: {...} }] |
consent | object | Consent states | { marketing: true } |
id | string | W3C span_id, 16 lowercase hex chars, generated by collector | "0123456789abcdef" |
timestamp | number | Auto-generated Unix ms | 1647261462000 |
entity | string | Parsed from name | "product" |
action | string | Parsed from name | "view" |
source.type | string | Source kind (browser, dataLayer, session, ...) | "browser" |
source.platform | string | Runtime platform (web, server) | "web" |
source.schema | string | Event-model version, set by the collector when no source | "4" |
source.trace | string | Run-scoped W3C trace_id, shared by every event of a run | "0123...cdef" (32 hex) |
source.count | number | Per-run emission sequence (1, 2, 3, ...) | 1 |
source.release | object | Per-flow config release map, accumulates across crossings | { web: "3", server: "5" } |
data Property
Entity-specific properties. Schema-free but consistent within entity type.
// product entity
data: { id: "P123", name: "Laptop", price: 999, currency: "USD" }
// page entity
data: { title: "Home", path: "/", referrer: "https://..." }
context Property
Hierarchical state information. Format: { name: [value, order] }. Optional.
context: {
stage: ["checkout", 1], // checkout stage, first step
test: ["variant-A", 0], // A/B test variant
group: ["premium", 2] // user segment
}
globals Property
Properties the collector merges into every event it creates.
globals: {
language: "en",
currency: "USD",
environment: "production"
}
nested Property
Related entities captured together. Optional.
// Order with line items
nested: [
{ entity: 'product', data: { id: 'P1', quantity: 2 } },
{ entity: 'product', data: { id: 'P2', quantity: 1 } },
];
user Property
User identification across sessions.
user: {
id: "user123", // Your user ID
device: "device456", // Device ID
session: "sess789", // Session ID
hash: "a1b2c3...", // Hashed identifier (for example a hashed email)
optout: false, // User opted out of tracking (boolean)
segment: "vip" // Custom keys are allowed, each holds a property value
}
The declared fields keep their types: email must be a valid address, so a
hashed email goes to hash, never to email. Any other key is a custom user
attribute with a property value and passes the validate transformer's
format: true check.
source Property
Where the event came from. The collector enriches the event with source data
based on which source emitted it.
source: {
type: 'browser', // source kind (browser, dataLayer, session, ...)
platform: 'web', // runtime: 'web' or 'server'
version: '3', // external source-emitter version (the collector does not set it)
schema: '4', // event-model version (the collector sets it on an event without a source)
trace: '0123...cdef', // run-scoped W3C trace_id, groups all events of a run
count: 1, // per-run emission sequence (1, 2, 3, ...)
release: { web: '3' }, // per-flow config release map, accumulates web to server
url: 'https://...', // page URL (set by the browser, dataLayer and session sources)
referrer: '...', // page referrer (same sources)
tool: 'cli', // tool that produced the event (optional)
command: 'simulate', // command name (optional)
}
Steps may add their own keys to source (the validate transformer writes
valid, GA4 decoding adds pageLoadId and hitSequence); each holds a
property value.
The browser, dataLayer and session sources set source.url and
source.referrer from the page. CMP sources emit walker consent commands, not
events.
source.trace is the run grouping key: the collector mints a fresh run-scoped
trace_id on each run and stamps it (plus a per-run source.count) on every
event when absent. It is preserved unchanged when an event is forwarded from web
to server, so the whole pipeline shares one trace.
source.release records which config handled the event, keyed by flow name.
Each flow the event passes through adds its own entry (that flow's config
release), so an event captured on the web and processed on the server carries
both flow releases, and a warehouse row shows exactly which config version
touched it. This is distinct from source.version, the source package version
set only by external source emitters; the collector does not stamp it.
Upgrading v3 events or configs: see
Migrating from v3 to v4. Nested
entities name their entity in entity, where v3 used type:
nested: [{ entity, data }].
Design Principles
Statelessness
Events are immutable snapshots. They don't reference previous events or maintain state.
Self-Describing
Events contain all context needed to understand them, without external lookups.
Vendor-Agnostic
Events use generic concepts (product, order) not vendor-specific (GA4 item, FB content).
Transformation to vendor formats happens in mapping, not in event creation.
Creating Events
elb takes a partial event: name is required, every other field is optional.
The collector adds id, timestamp, timing, entity, action and source,
and merges its own globals, user and consent key by key (the event's
values win).
import { startFlow } from '@walkeros/collector';
const { elb } = await startFlow({
/* destinations */
});
const result = await elb({
name: 'product add',
data: { id: 'P123', price: 99 },
context: { stage: ['cart', 1] },
globals: { currency: 'USD' },
});
// A failed destination is in result.failed (with its reason); a rejected
// event reaches no destination and sets result.error, e.g. 'Event name is invalid'
if (!result.ok) console.error(result.failed ?? result.error);
Every event uses this partial event form. walker commands keep their string
name: elb('walker consent', { marketing: true }).
Related Skills
- walkeros-understanding-mapping - Transform events for destinations
- walkeros-understanding-flow - Data flow architecture
Source Files:
- packages/core/src/types/walkeros.ts - Event types
- packages/core/src/schemas/ - Event schemas
Documentation:
- Website: Event Model - User-facing docs
- walkeros.io/docs - Public documentation
Understanding walkerOS Flow
Overview
walkerOS follows a Source → Collector → Destination(s) architecture for composable, modular event processing.
Core principle: Separation of concerns. Each component has one job. Components are composable and replaceable.
The Flow Pattern
[Source.before] → Sources → [Source.next] → Collector → [Collector.next] → [Dest.before] → Destinations → [Dest.next]
(Preprocessing) (Capture) (Source chain) (Processing) (Collector chain) (Dest. chain) (Delivery) (Post-push)
Consent-exempt: Post-consent:
- Decode - Validation - Event creation - Validation - Redaction - Push - Audit logging
- Validate format - Enrichment - Enrichment - Bot scoring - Per-vendor - Send - Response parsing
- Authenticate - Redaction - Consent check - Drop for all - Drop for one - Store - Webhooks
collector.next runs once per event before the destination fan-out, so its work
(and a stop) applies to every destination. destination.before runs per
destination.
Key Concepts
Composability
A Flow combines components. You can:
- Use multiple sources feeding one collector
- Route events to multiple destinations
- Swap components without changing others
The Flow Type
See packages/core/src/types/flow.ts for the canonical interface.
// Conceptual structure (see source for full type)
interface Flow {
stores?: Record<string, Store>;
sources?: Record<string, Source>;
transformers?: Record<string, Transformer>;
destinations?: Record<string, Destination>;
collector?: Collector.InitConfig;
}
Universal Push Interface
All components communicate via push functions:
| Component | Push Signature | Purpose |
|---|---|---|
| Source | push(input) → events | Capture external data |
| Collector | push(event) → PushResult | Process and route |
| Destination | push(event, context) → response? | Transform and deliver |
A destination's return value is the response a destination.next chain reads at
ingest._response.
The collector's elb() takes a partial event
(elb({ name: 'page view', data })) or a walker command; collector.push
takes event objects only. startFlow returns the primary source's push as
elb, or the collector's elb when no source is configured. The browser
source's push takes the same partial event; a server source's push is its
request handler.
startFlow Helper
See packages/collector/src/flow.ts for
the startFlow function.
import { startFlow } from '@walkeros/collector';
const { collector, elb } = await startFlow({
stores: {
/* key-value storage, init first, destroy last */
},
sources: {
/* ... */
},
transformers: {
/* ... */
},
destinations: {
/* ... */
},
});
Ingest: Mutable Pipeline Context
walkerOS uses a two-layer data model:
- Event: strict schema, structured analytics data (name, data, context, etc.)
- Ingest: free-form mutable context that flows alongside events
Any step can read and write arbitrary keys on ingest. The runtime manages
_meta:
_meta.hops: increments per step (safety valve at 256)_meta.path: ordered list of step IDs visited (path[0] = source ID)
// In a transformer
push: async (event, context) => {
context.ingest.botScore = 0.95; // Write freely
context.ingest.geo = { country: 'DE' };
console.log(context.ingest._meta.path); // ['express', 'validate', 'enrich']
return { event };
};
Ingest is cloned per destination to prevent cross-contamination in parallel
processing. After a destination push, the response is available at
ingest._response.
Stores
Three pipeline components (Source / Transformer / Destination) plus Stores as
passive infrastructure. Stores are not a fourth pipeline stage: they are
key-value storage that other components consume via env. They have no push,
no next, no before; they sit alongside the pipeline rather than inside it.
- Referenced via
$store.storeIdinenvvalues (bundled mode) or by passing the same store definition object used instores(integrated mode) - Init first, destroy last: stores are available before any source, transformer, or destination starts, and outlive them on shutdown
- No chains: stores don't participate in the event pipeline. Components
access them through their
env. - Implementations:
@walkeros/server-store-fs(async, filesystem),@walkeros/server-store-s3(async, S3-compatible),@walkeros/server-store-gcs,@walkeros/server-store-sheets,@walkeros/server-store-api. The collector ships a built-in in-memory cache tier; enable it on any store viaFlow.Store.cacheinstead of declaring a separate memory store.
{
"stores": {
"assets": {
"package": "@walkeros/server-store-fs",
"config": { "file": true, "settings": { "basePath": "./public" } }
}
},
"transformers": {
"file": {
"package": "@walkeros/server-transformer-file",
"config": { "settings": { "prefix": "/static" } },
"env": { "store": "$store.assets" }
}
}
}
See walkeros-understanding-stores for the full store interface and lifecycle.
Separation of Concerns
| Concern | Handled By | NOT Handled By |
|---|---|---|
| Event capture | Sources | Collector, Destinations |
| Event structure | Event model | Components |
| Consent checking | Collector | Sources, Destinations |
| Transformation | Mapping system | Raw push calls |
| Delivery | Destinations | Sources, Collector |
Step-Level Primitives
Next to its package wiring, every step (source, transformer, destination)
carries cache and state, and a transformer also mapping. A source's or
destination's mapping and consent sit in its config.
Event shapes are not a step-level primitive. They live in the top-level
contract block (a sibling of flows) as named JSON Schemas, and enforcement
is an explicit @walkeros/transformer-validate step that references a contract
via $contract.<name>. See
Website: Contract for
the contract shape and
Website: Validate for
runtime enforcement.
How a step references its implementation
Every step picks exactly one of three forms to point at its implementation:
package: "<npm-package>"alone loads the package's default export. The common case for sources and destinations.package: "<npm-package>"plusimport: "<exportName>"loads a specific named export from that package. Use when a package ships multiple named exports or has no default export.code: { push, type?, init? }is inline implementation, no package wiring. Useful for one-off custom logic in TypeScript flows.
There is no string form of code. A bare step with no package, import, or
code is a valid no-op for all four step kinds (handy as a chain-only or
mapping-only transformer).
Transformer Chains
Transformers run at three points around the collector, configured via
source.next, collector.next, and destination.before (plus source.before,
transformer.before, and destination.next):
Source Chain (source.next)
Runs after source captures event, before collector processing:
Bundled mode (flow.json):
{
"sources": {
"browser": {
"package": "@walkeros/web-source-browser",
"next": "enrich"
}
},
"transformers": {
"enrich": {
"code": {
"push": "$code:(event) => ({ event: { ...event, data: { ...event.data, enrichedAt: Date.now() } } })"
},
"next": "redact"
},
"redact": {
"mapping": { "policy": { "user.email": { "value": "[redacted]" } } }
}
}
}
enrich is inline code and redact a mapping-only step; any transformer
package fits the same slots.
Integrated mode (TypeScript):
sources: {
browser: {
code: sourceBrowser,
next: 'enrich'
}
},
transformers: {
enrich: {
code: transformerEnrich,
next: 'redact'
},
redact: {
code: transformerRedact
}
}
In both modes next sits on the step entry (config.next works too).
Transformer before chain
Each transformer can have its own before chain that runs before its push
function:
{
"transformers": {
"fingerprint": {
"before": "bot",
"package": "@walkeros/server-transformer-fingerprint",
"config": { "settings": { "salt": "$secret.FINGERPRINT_SALT" } }
},
"bot": {
"package": "@walkeros/server-transformer-bot"
}
}
}
Collector Chain (collector.next)
Runs once per event after the collector completed it, before the destination
fan-out. Every destination receives its output; a stop here reaches no
destination:
{
"collector": {
"next": [
{
"match": {
"key": "event.name",
"operator": "eq",
"value": "product impression"
},
"stop": true
},
"bot",
"validate"
]
}
}
In TypeScript it is the top-level next of startFlow({ next, ... }).
Destination Chain (destination.before)
Runs per destination, after the collector chain, before that destination
receives the event. A stop here skips only this destination:
Bundled mode (flow.json):
{
"destinations": {
"gtag": {
"package": "@walkeros/web-destination-gtag",
"config": { "settings": { "ga4": { "measurementId": "G-DEMO123456" } } },
"before": "redact"
}
},
"transformers": {
"redact": {
"mapping": { "policy": { "user.email": { "value": "[redacted]" } } }
}
}
}
Integrated mode (TypeScript):
destinations: {
gtag: {
code: destinationGtag,
before: 'redact'
}
},
transformers: {
redact: {
code: transformerRedact
}
}
Post-push chain (destination.next)
Runs after destination push completes. The push response is available at
context.ingest._response:
Bundled mode (flow.json):
{
"destinations": {
"api": {
"package": "@walkeros/server-destination-api",
"config": { "settings": { "url": "https://collect.example.com/events" } },
"next": "auditLog"
}
},
"transformers": {
"auditLog": {
"code": {
"push": "$code:(event, context) => { context.logger.info('delivered', { id: event.id, response: context.ingest._response }); }"
}
}
}
}
Integrated mode (TypeScript):
destinations: {
api: {
code: destinationApi,
next: 'auditLog'
}
},
transformers: {
auditLog: {
code: {
push: (event, context) => {
context.logger.info('delivered', {
id: event.id,
response: context.ingest._response,
});
},
},
},
}
auditLog is inline code: there is no audit transformer package.
Chain Resolution
source.before→ consent-exempt preprocessing chainsource.next→ source chain, before the collectortransformer.before→ pre-transform enrichment chaintransformer.next(flow.json) ortransformer.config.next(runtime) → links transformerscollector.next→ collector chain, once per event, for all destinationsdestination.before→ destination chain, for that destination onlydestination.next→ post-push processing chain
Cross-Flow References ($flow)
When a single flow.json defines multiple flows, any flow can pull values from
another flow's config block via $flow.<name>(.<path>)?. The most common case
is wiring a web flow's API destination to a server flow's deployed URL, so the
two stay in sync without duplication.
{
"version": 4,
"flows": {
"server": {
"config": {
"platform": "server",
"url": "https://collect.example.com"
},
"sources": {
"http": { "package": "@walkeros/server-source-express" }
}
},
"web": {
"config": { "platform": "web" },
"destinations": {
"api": {
"package": "@walkeros/web-destination-api",
"config": { "settings": { "url": "$flow.server.url" } }
}
}
}
}
}
An unknown flow, a missing key or an empty value is an error in
walkeros validate (UNRESOLVED_REFERENCE) and in bundle, so a build never
ships with an empty cross-flow value.
Step Examples
Each step in a flow (source, transformer, destination) can ship step
examples: structured { in, out } pairs that define expected input/output
behavior.
The Three Type Zones
Steps sit at boundaries between arbitrary formats and walkerOS events:
- Source: arbitrary
in(HTTP request, DOM event) → walkerOS eventout - Transformer: walkerOS event
in→ walkerOS eventout(orfalse) - Destination: walkerOS event
in→ arbitraryout(vendor API call)
See using-step-examples for the full ASCII diagram and detailed explanation.
Example: Step with Examples in Flow Config
{
"destinations": {
"gtag": {
"package": "@walkeros/web-destination-gtag",
"config": { "settings": { "ga4": { "measurementId": "G-DEMO123456" } } },
"examples": {
"purchase": {
"in": {
"name": "order complete",
"data": { "id": "ORD-123", "value": 149.97 }
},
"mapping": {
"name": "purchase",
"data": {
"map": { "transaction_id": "data.id", "value": "data.value" }
}
},
"out": [
[
"window.gtag",
"event",
"purchase",
{
"transaction_id": "ORD-123",
"value": 149.97,
"send_to": "G-DEMO123456"
}
]
]
}
}
}
}
}
Step examples enable it.each testing and cross-step checks in
walkeros validate. See
using-step-examples for the complete
lifecycle.
Flow Graph Connection Rules
This section defines which components can connect to which, and how chains are resolved at runtime. Use it as the canonical reference for building flow graphs, validating configurations, and rendering UI visualizations.
Every chain and route shape below has a tested instance in
packages/cli/examples/flow-complete.json. Look them up by feature id in the
manifest (flowCompleteFeatures from @walkeros/cli/examples, each with a JSON
Pointer and a note), taught in the guide chapter chains-routing
(packages/cli/examples/flow-complete.md):
| Shape | Feature ids |
|---|---|
| Source chains | source-before, source-next |
| Collector chain | collector-next, web-collector-next |
| Destination chains | destination-before, pubsub-before |
one, conditional next, sequence | route-one, route-next-match, route-sequence |
stop with and without match | route-stop, event-filter, stop-unconditional |
| Code-less path step | code-free-hop |
and, or, not | match-and, match-or, match-not |
many, transformer.before and destination.next have no genuine use in that
file; the route fixture routeCases in @walkeros/core/dev covers them.
Valid connection matrix
| From | To | Via Field | Valid? |
|---|---|---|---|
| Source | Transformer | source.before | Yes (consent-exempt) |
| Source | Transformer | source.next | Yes (source chain) |
| Source | Collector | (implicit, no next) | Yes |
| Source | Source | - | No |
| Source | Destination | - | No |
| Transformer | Transformer | transformer.before | Yes (pre-transform) |
| Transformer | Transformer | transformer.next | Yes (chain continues) |
| Transformer | Collector | (implicit, source chain ends) | Yes |
| Transformer | Destination | (implicit, chain ends) | Yes |
| Collector | Destination | (implicit, no next/before) | Yes |
| Collector | Transformer | collector.next | Yes (all destinations) |
| Collector | Transformer | destination.before | Yes (one destination) |
| Destination | Transformer | destination.next | Yes (post-push) |
| Collector | Source | - | No |
Source chains (source.next)
- Entry:
source.next: "transformerId"orsource.next: ["t1", "t2"] - Exit: chain ends, event reaches collector
- Multiple sources can reference the same transformer (fan-in)
- No
next= source connects directly to collector - Resolved per event after the source's
state, so it can route on a loaded value
Collector chain (collector.next)
- Entry:
collector.next(flow.json) or top-levelnextinstartFlow - Runs once per completed event, before the destination fan-out
- Exit: every finished copy is delivered to the destinations; a
stopor a transformer returningfalsedrops the event for all destinations - Consent replay, late destinations, and pre-run replay receive its output; it never runs twice for one event
- No
next= collector connects directly to the destinations
Destination chains (destination.before)
- Entry:
destination.before: "transformerId"ordestination.before: ["t1", "t2"] - Exit: chain ends, event reaches destination; a
stopskips only this destination - Multiple destinations can share the same transformer
- No
before= the collector (or its chain) connects directly to destination - Per-destination filtering belongs here, not in
collector.next
Chain resolution (one model for every chain field)
One runner executes every position (runTransformerChain in
packages/collector/src/transformer.ts),
on top of the pure continuation stack startChain / advanceChain in
packages/core/src/chain.ts. Routes are
resolved per hop by getNextSteps(spec, root) in
packages/core/src/route.ts, which returns
NextSteps (ids plus an optional continuation, a stop, or forks) and needs the
root { ingest, event }.
- String start: runs the transformer, then its own
next, recursively - Array start: the array is the backbone. A member's own
next(static or conditional) is inserted right after that member, then the array continues.["bot", "validate", "session"]withbot.next = "foo"runs bot, foo, validate, session. Insertion is depth-first. - Lazy per hop: a route is evaluated only when the event reaches it, against
{ ingest, event }as the previous step left them. A sequence["a", { "match": ..., "stop": true }, "b"]evaluates the stop afteraran. - Repeated steps: a step listed twice runs twice (no dedup, no warning)
- Cycles: a member
nextthat leads back to a step already on its insertion path is skipped; each copy is capped at 256 steps - Unknown transformer id: logged as an ERROR line and skipped, the chain
continues;
walkeros validatereportsUNKNOWN_ROUTE_TARGETas an error - Result
{ next }: replaces the member's ownnextfor that event, then the array continues
For tooling without an event, getRouteGraph(spec, transformers?) enumerates
every branch of the same compiled form (both sides of each gate, every one and
many entry, stops marked), so canvases and validators draw exactly what can
run.
Conditional routing (one operator)
The next and before properties accept a Route
(string | Route[] | RouteConfig). A RouteConfig is a disjoint union:
each config sets at most one of next (gated link), one (first-match
dispatch), many (all-match fan-out), or stop (drop), never more than one.
Every match reads { ingest, event }. The one operator picks the first
entry whose match succeeds:
"next": {
"one": [
{ "match": { "key": "ingest.path", "operator": "prefix", "value": "/api" }, "next": "apiHandler" },
{ "next": "default" }
]
}
oneentries are evaluated in order, first match wins- Besides
_meta,ingestholds only what the source'sconfig.ingestmaps and what earlier steps write into it: without{ "map": { "path": "path" } }in that config,ingest.pathis empty and the/apientry never matches - An entry without
matchalways matches, use it as the fallback - No matching entry means the event passes through unchanged
- Works on all chain positions:
source.before,source.next,transformer.before,transformer.next,collector.next,destination.before, anddestination.next - An array made only of route configs (no id) is an implicit
one: first match wins.walkeros validatehints at it; write{ "one": [...] }explicitly - Routes are compiled once (cached) and resolved per event by
getNextStepsin packages/core/src/route.ts
All-match dispatch (many operator)
Use many when every matching entry should produce an independent copy of the
event (audit-while-process, multi-decoder fan-out). Allowed in every chain
field, at any depth. Each copy finishes the entire rest of the path on its own
(rest of the array, every enclosing chain, collector, destinations); copies are
never merged. Each copy gets its own event.id, derived deterministically from
the parent id and branch position (deriveSpanId); trace stays shared. One
matching entry continues in place without a copy. A transformer returning
Result[] forks the same way.
"next": {
"many": [
{ "match": { "key": "event.consent.analytics", "operator": "eq", "value": "true" }, "next": "ga4Pipeline" },
{ "next": "auditLog" }
]
}
A consent value is a boolean, and eq compares it as text, so a granted consent
matches "true".
Dropping an event (stop)
{ "stop": true } ends the running copy (optionally gated by match; a failing
match falls through). In a many, a stop ends only its own copy. Meaning per
position:
| Position | A resolved stop means |
|---|---|
source.before, source.next | never reaches the collector; push result { ok: true, dropped: true } |
transformer.before, transformer.next, result { next } | the copy ends; the enclosing position applies its meaning |
collector.next | no destination receives the event (runs before the fan-out) |
destination.before | this destination skips it; others unaffected |
destination.next | post-push chain ends; delivery already happened |
"Drop impressions everywhere" is a stop in collector.next; "drop impressions
for vendor destinations only" is a stop in those destinations' before. A drop
is observable as a skip with skipReason: 'dropped'. walkeros validate
warns about entries after an unconditional stop (dead code).
Paths and pass-through steps (code-less transformer entries)
A path is the multi-step chain through a flow's transformers section. A
pass-through step (short: pass) is a single step inside a path that
declares no code and no package; the runtime synthesizes its push from the
operative fields the step does declare.
Pass-through steps come in three variants:
- Chain-only: only
beforeand/ornextset. A named hop that shares a chain across multiple call sites (avoids duplicating arrays). - Cache-only: only
cacheset. A dedup or short-circuit step.cache.stop: truein a source chain orcollector.nexthalts the event for all destinations. - Mapping-only: only
mapping: Mapping.Configset. A declarative event-to-event transform that mutates the event in-flight.
{
"transformers": {
"validateThenEnrich": {
"before": ["validate", "enrich"]
}
},
"destinations": {
"gtag": {
"package": "@walkeros/web-destination-gtag",
"config": { "settings": { "ga4": { "measurementId": "G-DEMO123456" } } },
"before": "validateThenEnrich"
},
"meta": {
"package": "@walkeros/web-destination-meta",
"config": { "settings": { "pixelId": "1234567890" } },
"before": "validateThenEnrich"
}
}
}
Transformer step entries follow a closed schema: unknown top-level keys are
validation errors (UNKNOWN_KEY). An entry with no field at all is a valid
no-op.
See
walkeros-understanding-transformers
for full depth on the three variants, the closed-schema rule, and the dual
semantic of mapping at the transformer position versus the destination
position.
Transformer sharing
A single transformer can appear in source chains (source.next), the collector
chain (collector.next), and destination chains (destination.before). The
same transformer pool is shared; role depends on which chain references it. Work
every destination needs belongs in collector.next, not repeated in each
destination.before.
Deferred activation (require)
source.config.require: ["consent"]: the source'son()deliveries wait until consent state is recorded; itsinitruns at oncedestination.config.require: ["user"]: destination deferred until "user" event fires- Multiple requirements: all must be fulfilled (each fires independently)
Mapping and consent gating
- Source-level:
source.config.mappingandsource.config.consent: applied before the source chain; blocks event entirely - Destination-level:
destination.config.consentis checked first, before the destination chain (destination.before): it skips only that destination and queues denied events. The queue sits aftercollector.next, so a replay of a queued event never re-runscollector.next; it does rundestination.before.destination.config.mappingis applied after the destination chain, so a rename in the chain changes which rule matches
Canvas rendering rules (for UI graph visualization)
- Shared transformers (pre+post): duplicate visually with a link indicator; editing one updates the other
- Router fan-out: keep graph planar; trace edges individually
- Orphan transformers (not in any chain): render grey/muted; gain color when connected
- Diamond patterns (fan-in + fan-out): expected and valid
- Overlapping
destination.beforechains: intentional (e.g., shared validator for monitoring) - Collector chain: draw
collector.nextbetween the collector and the destination fan-out, distinct from per-destinationbeforechains; draw astopas an end of its branch
Related Skills
- walkeros-understanding-events - Event model
- walkeros-understanding-sources - Source interface
- walkeros-understanding-destinations - Destination interface
Package READMEs:
- packages/collector/README.md - Collector details
Source Files:
- packages/collector/src/flow.ts - startFlow implementation
Documentation:
- Website: Flow - Flow concept
- Website: Collector - Collector docs
- Website: Routing -
Connections, chain resolution, route operators,
stop
Understanding walkerOS Mapping
Overview
Mapping transforms data at multiple points in the walkerOS flow:
- Source → Collector: Transform raw input (HTTP requests, dataLayer pushes) into walkerOS events
- Collector → Destination: Transform walkerOS events into vendor-specific formats
Core principle: Mapping is the universal transformation layer. Same strategies work everywhere in the flow.
Core Functions
See packages/core/src/mapping.ts for implementation.
| Function | Purpose |
|---|---|
getMappingEvent(event, rules) | Find mapping rule for an event |
getMappingValue(value, data, { collector }) | Transform a value using mapping config |
processEventMapping(event, config, collector) | Unified processing for sources/destinations |
processEventMapping Flow
1. Apply config.policy (modifies event)
2. Find matching rule via getMappingEvent()
3. Apply rule.policy (modifies event)
4. Transform config.data (global)
5. Check rule.ignore (short-circuits if true)
5b. Read rule.silent (informational - destination honors it)
6. Override event.name if rule.name
7. Transform rule.data (event-specific)
8. Add the include sections (rule.include ?? config.include; data wins)
9. Strip rule.remove paths from the produced data
Configuration Hierarchy
Mapping.Config (Top Level)
interface Config {
consent?: Consent; // Required consent for ALL events
data?: Value; // Global data transformation
include?: string[]; // Event sections to flatten into context.data
policy?: Policy; // Pre-processing for ALL events
mapping?: Rules; // Event-specific rules
}
Mapping.Config.include
include lists event sections whose fields are added to the data the
destination receives (PushContext.data), after the rule's own data ran. Each
field becomes a flat key named <section>_<key>: nested objects are joined with
_, arrays stay whole, a context entry contributes its label. Sections:
data, globals, context, user, source, event (entity, action,
id, timestamp, name, trigger, timing), or all for every one of them.
On a key conflict the produced data (the config-level data merged with the
rule's) wins; remove runs after it. The runtime helper is
flattenIncludeSections (in packages/core/src/include.ts).
Use this to forward whole sections without mapping each field. include is also
available on Mapping.Rule (rule-level replaces config-level). user includes
every user field, email among them, unhashed.
{
include: ['globals'],
mapping: {
page: {
view: {
// data: { page_title, globals_language, globals_currency, ... }
data: { map: { page_title: 'data.title' } },
},
},
},
}
Mapping.Rule (Per Event)
interface Rule {
name?: string; // Override event name
data?: Value; // Event-specific data transformation
include?: string[]; // Event sections to flatten into context.data (rule-level)
ignore?: boolean; // Skip this event entirely (no processing, no push)
silent?: boolean; // Process settings side effects, skip destination default push
policy?: Policy; // Event-specific pre-processing
condition?: Function; // Match condition (for arrays)
consent?: Consent; // Required in addition to config.consent for events this rule matches
settings?: unknown; // Custom event configuration
batch?: number | BatchOptions; // Own batch buffer (a number is the wait in ms)
extend?: RulePatch; // Config-layer merge onto a package-shipped default rule
remove?: string[]; // Output-layer: dotted paths stripped from the final payload
}
extend and remove (patching package-shipped rules)
@walkeros/transformer-ga4 ships its own default mapping rules. A user rule at
the same key normally replaces the default in full. Two keywords change that:
extend(config layer): a partial rule deep-merged onto the package-shipped default at init, before any event is evaluated. Anullvalue clears an inherited field. Lets you add or override one field while keeping the rest. It has an effect only in@walkeros/transformer-ga4'ssettings.mapping; in any other rulewalkeros validateand the bundle reject it (MAPPING_NO_OP).remove(output layer): dotted paths stripped from the produced payload after evaluation, applied last. Useful for dropping PII or vendor-reserved fields without rewriting the full rule. Works in a destination's rules and in@walkeros/transformer-ga4'ssettings.mapping.
A rule with neither keyword keeps the standard replace behavior.
See the website mapping docs for the authoritative reference and the GA4 example.
{
"settings": {
"mapping": {
"purchase": {
"extend": {
"data": { "map": { "affiliation": "params.ep.affiliation" } }
},
"remove": ["currency"]
}
}
}
}
Silent vs Ignore
Both flags control rule behavior but have different semantics:
ignore: true: the rule matched but nothing happens. No data transform, no destination call, no side effects. Use for suppression.silent: true: the rule matched and the destinationpush()is called.settings.identify,settings.revenue,settings.group, etc. still run. Only the destination's default forwarding call (e.g.track(),capture(),event()) is suppressed. Use for "identify without an event" style flows.
If both flags are set on the same rule, ignore wins.
Common use case: a user login event that should call amplitude.identify()
but should not create a separate track("user login") event in Amplitude.
Mapping.ValueConfig (Value Extraction)
interface ValueConfig {
key?: string; // Extract from path
value?: Primitive; // Static fallback value
fn?: Function; // Compute the value from the source
map?: Record; // Object transformation
loop?: [path, config]; // Array transformation
set?: Value[]; // Create array from values
condition?: Function; // Conditional extraction
consent?: Consent; // Consent-gated extraction
validate?: Function; // Value validation
}
Exactly one producer runs per value, in this order: loop, map, set, key,
fn. The rest never run, so a fn next to a key is ignored. value is the
fallback when the producer yields nothing, including a loop over something
that is not a list. To transform a field, use fn alone (it receives the
source), or rewrite it with policy and read it with key.
Event Matching
Match events to transformation rules by entity and action.
const mapping = {
// Exact match: "product view" → view_item
product: {
view: { name: 'view_item' },
add: { name: 'add_to_cart' },
},
// Wildcard action: "foo *" → foo_interaction
foo: {
'*': { name: 'foo_interaction' },
},
// Wildcard entity: "* click" → generic_click
'*': {
click: { name: 'generic_click' },
},
};
Conditional Mapping (Array)
In an array of rules, the first matching condition wins:
order: {
complete: [
{
condition: (event) => event.data?.value > 100,
name: 'high_value_purchase',
},
{ name: 'purchase' }, // Fallback (no condition)
],
}
JSON with $code:
{
"order": {
"complete": [
{
"condition": "$code:(event) => event.data?.value > 100",
"name": "high_value_purchase"
},
{ "name": "purchase" }
]
}
}
Value Mapping Strategies
Common patterns shown below. For detailed examples of all 12 strategies, see value-strategies.md.
Essential Patterns
// Key extraction (string shorthand)
'data.price' // → event.data.price
// Key with fallback
{ key: 'data.currency', value: 'USD' } // Use USD if missing
// Static value
{ value: 'USD' }
// Function transform
{ fn: (event) => event.data.price * 100 } // Convert to cents
// Object map
{ map: { item_id: 'data.id', item_name: 'data.name' } }
// Array loop
{ loop: ['nested', { map: { item_id: 'data.id' } }] }
// Loop with "this" (single item as array)
{ loop: ['this', { map: { item_id: 'data.id' } }] }
// Set (create array)
{ set: ['data.id'] } // → ["SKU-123"]
// Fallback chain: Value[] at any value position (first defined value wins)
[{ key: 'data.sku' }, { key: 'data.id' }, { value: 'unknown' }]
// Consent-gated (the event's consent decides, in a loop too: a `consent`
// key on a loop item is plain data)
{ key: 'user.email', consent: { marketing: true } }
// Validate
{ key: 'data.email', validate: (v) => v.includes('@') }
Policy (Pre-Processing)
Policy shapes the event before its data is mapped. At a destination the rule was already selected when the event entered it; at a source or transformer, policy runs before matching. Use for:
- Adding computed fields
- Normalizing data structure
- Consent-gated field injection
Config-Level Policy
Applied to all events:
config: {
policy: {
'user_data.external_id': 'user.id',
'custom_data.server_processed': { value: true },
},
mapping: { /* ... */ }
}
Event-Level Policy
Applied after config policy, only for specific event:
mapping: {
order: {
complete: {
policy: {
'enriched.total_cents': {
fn: (event) => Math.round(event.data.value * 100)
}
},
name: 'purchase',
data: { /* ... */ }
}
}
}
Policy with Consent
A policy entry that resolves to nothing removes the field, so a denied consent gate redacts it:
{
"policy": {
"user_data.em": {
"key": "user.email",
"consent": { "marketing": true }
}
}
}
Rule Features
Ignore Events
mapping: {
test: { '*': { ignore: true } }, // Ignore all test events
}
Batch Processing
To batch all of a destination's events, set config.batch (no mapping rule
needed):
config: {
batch: { size: 5 }, // Flush after every 5 events (a bare number is the wait window in ms)
}
A rule-level batch batches only that entity-action into its own buffer and
overrides config.batch per field:
mapping: {
order: {
complete: {
batch: { wait: 1000 }, // Only order complete batches, with a 1s debounce
}
}
}
Custom Settings
mapping: {
order: {
complete: {
name: 'purchase',
settings: { priority: 'high', retryCount: 3 }
}
}
}
$code: Prefix (JSON Configs)
The $code: prefix enables JavaScript functions in JSON configurations:
{
"fn": "$code:(event) => event.data.price * 100",
"condition": "$code:(event) => event.data?.value > 100",
"validate": "$code:(value) => value > 0"
}
Important: The $code: prefix is processed by the CLI bundler. It converts
JSON strings to actual JavaScript functions during build.
Function Signatures
All mapping callbacks share (value, context). The second argument is a
Mapping.Context object (see "Context object" below).
| Context | Signature |
|---|---|
fn | (value, context) => result |
condition (value) | (value, context) => boolean |
condition (rule) | (event, context) => boolean |
validate | (value, context) => boolean |
loop condition | (value, context) => boolean |
One-arg signatures like (value) => value.toUpperCase() continue to work,
TypeScript ignores the unused second arg.
Context object
| Field | Type | Required | Description |
|---|---|---|---|
event | WalkerOS.DeepPartialEvent | yes | The root event being mapped |
mapping | Mapping.Value | Mapping.Rule | yes | The surrounding mapping config (or rule) |
collector | Collector.Instance | yes | Active collector, use for push and queue |
logger | Logger.Instance | yes | Use for info/warn/error/debug |
consent | WalkerOS.Consent (optional) | no | Resolved consent at this evaluation point |
Quick Reference
Value Extraction Cheatsheet
| Pattern | Result |
|---|---|
"data.id" | Extract event.data.id |
{ value: "USD" } | Static "USD" |
{ key: "x", value: "y" } | Extract x, fallback to "y" |
{ fn: (e) => ... } | Custom function |
{ map: {...} } | Object transformation |
{ loop: ["nested", {...}] } | Array transformation |
{ loop: ["this", {...}] } | Single-item as array |
{ set: ["a", "b"] } | Create array [valA, valB] |
[m1, m2, m3] | Fallback chain |
{ consent: {...} } | Consent-gated |
{ condition: fn } | Conditional |
{ validate: fn } | Validated |
Rule Features Cheatsheet
| Feature | Purpose |
|---|---|
name | Override event name |
data | Transform event data |
ignore | Skip event entirely (no processing, no push) |
silent | Run settings side effects, skip default forwarding |
policy | Pre-process event |
condition | Match condition (arrays) |
consent | Required in addition to config.consent for the events it matches |
settings | Custom configuration |
batch | Own batch buffer for this event (destination needs pushBatch) |
extend | Config-layer merge onto a package-shipped default (null clears a field) |
remove | Output-layer: dotted paths stripped from the final payload |
Config Features Cheatsheet
| Feature | Purpose |
|---|---|
consent | Required consent (all events) |
data | Global data transformation |
include | Event sections added to data |
policy | Global pre-processing |
mapping | Event-specific rules |
Complete Examples
For full destination configuration examples (TypeScript + JSON), see complete-examples.md.
Mapping at the Transformer Position
Mapping.Config accepts the same shape at three positions in the flow, but the
semantic differs by position:
| Position | What the mapping produces |
|---|---|
| Source | A walkerOS event from raw input |
| Transformer | A mutated walkerOS event that continues through the chain |
| Destination | A vendor-shaped payload that the destination consumes |
When a transformer step declares only a mapping (no code, no package), the
collector synthesizes a push that runs processEventMapping against each event.
A config.mapping wins over the step-level mapping. Next to code or
package, a mapping never runs. walkeros validate and the bundle reject both
cases (MAPPING_NO_OP), and the collector logs one ERROR line for them when the
transformer registers. It is the same keyword as the destination field, with a
different semantic at this step position. See
walkeros-understanding-transformers
for the pass-through-step model.
Which fields apply at the transformer position
Only event-mutating fields run; every other field is a no-op that
walkeros validate and the bundle reject (MAPPING_NO_OP) and the collector
names in one ERROR line at registration (rules in array form included):
| Field | Transformer position |
|---|---|
policy | Applies, pre-processes the event before rule matching |
mapping[].condition | Applies, picks the rule (awaited in order, first match wins) |
mapping[].policy | Applies, per-event policy |
mapping[].name | Applies, renames the event (mutation is observable downstream); in a destination's own before chain it does not change that destination's rule (ERROR rule-not-applied when it lands on another rule) |
mapping[].ignore | Applies, drops the event from the chain entirely (no downstream step sees it) |
consent, include, data | No-op (no vendor payload, no destination to gate) |
mapping[].consent, include, remove, batch, settings, extend, data, silent | No-op (destination concepts; a rule's consent also applies at a source) |
Note the ignore: true semantic shift: at a destination it skips delivery
to that destination only; at a transformer step it drops the event from the
chain so no downstream step (transformer or destination) sees it.
{
"transformers": {
"redactPII": {
"mapping": {
"policy": {
"user.email": { "value": "[redacted]" }
},
"mapping": {
"test": {
"*": { "ignore": true }
},
"order": {
"complete": { "name": "purchase" }
}
}
}
}
}
}
Where Mapping Lives
| Location | Purpose |
|---|---|
| Source config | Transform raw input → walkerOS events |
| Transformer step config | Mutate walkerOS events in-flight |
| Destination config | Transform walkerOS events → vendor format |
state.mapping | Shape a store read or write (paths relative to the value read) |
packages/core/src/mapping.ts | Core mapping functions |
packages/core/src/types/mapping.ts | Type definitions |
packages/cli/examples/flow-complete.json | Tested example, features by id |
Related Skills
- walkeros-understanding-events - Event structure
- walkeros-understanding-destinations - Destination-side mapping
- walkeros-understanding-transformers - Transformer-position mapping and pass-through-step variants
- walkeros-mapping-configuration - Recipes and patterns
Source Files:
- packages/core/src/mapping.ts - Implementation
- packages/core/src/types/mapping.ts - Types
Detailed References:
- value-strategies.md - All 12 value extraction strategies with examples
- complete-examples.md - Full destination config examples
Examples:
- packages/cli/examples/flow-complete.json -
Tested example; mapping features by id (for example
ga4-purchase,value-loop,rule-policy) are listed in complete-examples.md - packages/cli/examples/flow-complete.md -
Guide, chapters
mappingandconsent-privacy
Understanding walkerOS Sources
Overview
Sources capture events from the external world (browser DOM, dataLayer, HTTP requests, cloud functions) and feed them to the collector.
Core principle: Sources capture. They don't process or deliver: that's collector and destinations.
Source Interface
See packages/core/src/types/source.ts for canonical interface.
Init Function (Context Pattern)
Sources use a context-based initialization pattern:
import type { Source } from '@walkeros/core';
export const sourceMySource: Source.Init<Types> = async (context) => {
const { config = {}, env, logger, id } = context;
// ...
};
Context contains:
| Property | Type | Purpose |
|---|---|---|
config | Source.Config<T> | Settings, mapping, options |
env | Types['env'] | Environment (push, logger) |
logger | Logger | Logging functions |
id | string | Source identifier |
collector | Collector.Instance | Reference to collector |
withScope | (scope, respond, body) => Promise<R> | Bind ingest + respond to a single scope (server sources). Takes a normalized Source.Scope |
Push Method
| Method | Purpose |
|---|---|
push(input) | Receive external input, emit events |
Init Method
init?: () => void | Promise<void>: optional eager-startup hook on the returned
Source.Instance. The factory must be side-effect-free: build the instance
and return it. The collector calls init() on every source eagerly after all
factories register, regardless of config.require. Use init for the side
effects: adopting a pre-init window queue (e.g., window.elbLayer), attaching
DOM listeners, opening sockets, intercepting window.dataLayer. After init
runs the collector flips Source.Config.init to true.
queueOn Buffer
queueOn?: Array<{ type: On.Types; data: unknown }>: optional buffer on the
Source.Instance for lifecycle events delivered before the source is
started (started ≡ config.init === true && !config.require?.length). The
collector pushes { type, data } here when it would otherwise call
source.on(type, data). Once the source becomes started, the collector replays
each entry via source.on(...) and clears the queue.
Destroy Method
destroy?: DestroyFn: optional cleanup method. Called during
command('shutdown'). Use to close HTTP servers, timers, or connections.
Receives { id, config, env, logger }.
Source Lifecycle
The collector and every source agree on three lifecycle markers, all on
Source.Config:
| Field | Purpose |
|---|---|
init?: boolean | Set by the collector to true after Instance.init() resolves. Authors do not write to it. Reflects "init has run", not "is started". |
require?: string[] | Author-supplied timing hint. Lists collector state a source needs before its first on() delivery. Satisfied by the collector's current recorded state (level), not only by a future event, so order does not matter: the gate clears whether the required state was recorded before or after this source registered. It is not a correctness dependency: a source reacts to state correctly whether or not it declares require. |
disabled?: boolean | Accepted by walkeros validate, but the collector does not read it for sources: a source with disabled: true still registers, runs init and captures events. Remove the source from the flow to switch it off. |
The flow is:
- Register: collector invokes the factory. Factory returns a fresh
Source.Instancewith no side effects. - Init pass: collector calls
Instance.init()on every registered source, then setsconfig.init = true. - Lifecycle delivery: for each collector event (
consent,user,session,run, …) the collector decrements every source'srequirein place AND reconciles every still-pending source/destination against current state, so a gate also clears from state that was already recorded. If a source is started, it callssource.on(type, data)directly. Otherwise it pushes{ type, data }intoInstance.queueOn. - Replay: when a source becomes started (require empties), the collector
replays its
queueOnviasource.on(...)and clears the queue.
require therefore gates the timing of on() delivery, not code execution
and not correctness. Instance.init() always runs eagerly. There is no
collector.pending.sources map: per-source state lives entirely on
Source.Instance and Source.Config.
Exactly-once state delivery is a collector guarantee
State commands (consent, user, globals, custom) are recorded by the
collector immediately, even before run, and delivered to each source's on
handler exactly once per change. The collector tracks what each subscriber has
already received, so re-running or re-registering never double-fires a state
reaction. Sources should not hand-roll their own deduplication for state
deliveries: the collector enforces exactly-once, and delivery is
order-independent (it does not depend on source init order or on whether the
state arrived before or after run).
This mirrors the destination model: Destination.Instance.init handles one-time
bootstrap, Destination.Config.init is the collector-managed "init has run"
flag, and Destination.Config.require gates event delivery the same way.
Destinations receive the same exactly-once state delivery through their on()
handler, and receive the current state before their first push. See
walkeros-understanding-destinations.
Pre-run events are held and replayed by the collector
Events a source emits before the collector runs are held in a bounded FIFO
buffer (collector.preRunQueue, capped by config.queueMax) and replayed at
run, with the pipeline running once at replay time. A source therefore does
not need bespoke on('run') deferral to avoid losing its startup events.
Deferring emission to on('run') is fine and often clearer for ordering
(the session source does this so session start lands in the post-run
pipeline), not for loss protection.
The one case that needs an explicit run gate is a queue consumer
(pubsub-pull, sqs): starting to consume means acknowledging messages to a
broker, and an ack must never precede delivery. These sources start their
subscription or poll loop in on('run') and never ack a message the pipeline
did not accept (result.ok === false routes to the source's onPushError
disposition, nack by default).
Push Signatures by Type
| Source Type | Signature | Example |
|---|---|---|
| Cloud Function | push(req, res) → Promise<void> | HTTP handler |
| Browser | push(event) → Promise<PushResult> | A partial event or a walker command |
| DataLayer | push is the collector's elb | Captures window.dataLayer.push instead |
A source's push is the handler itself, with no wrapper:
// Direct deployment
http('handler', source.push);
Do not confuse Instance.push with env.push. They point opposite ways:
Instance.pushis inward: the handler callers hand input to (an HTTP request, awindow.elb(...)call, a synthetic test message).startFlowexports the primary source'sInstance.pushas itselb.env.pushis outward: where the source sends the events it produced.
A source can legitimately have both, and they need not be related: the browser
source's Instance.push is a BrowserPush that accepts partial events and
walker commands, while its events leave via env.push. The dataLayer and
session sources keep Instance.push: env.elb, because callers do not feed them
events at all: they capture from dataLayer.push and from session detection
respectively.
When a test or integration code needs to invoke a source's push through the
collector bag, collector.sources erases the per-source generic on read. Use
Source.getSource<T>(collector, id) to recover the narrow signature without a
cast. See the testing-strategy skill for the full pattern (symmetric helpers
exist for destinations, transformers, stores).
Source Paths
| Type | Path | Examples |
|---|---|---|
| Web | packages/web/sources/ | browser, dataLayer |
| Server | packages/server/sources/ | gcp |
Browser Source
The browser source captures events from DOM using data attributes.
<button data-elb="product" data-elb-product="id:P123;name:Laptop">
<span data-elbaction="click">Add to Cart</span>
</button>
See packages/web/sources/browser/ for implementation.
Parallel sources
Several browser sources can run on one page. Trigger, element, and visibility
state is per source, and the DOM is shared: any number of sources may scan the
same node, each emitting into its own pipeline. Only two resources are single
slots, window[elb] and the named elbLayer array, and each belongs to the
first source that takes it. A source refused one of them logs an error and keeps
capturing without it. Give each parallel flow its own elb and elbLayer name,
or elb: false and elbLayer: false for an embedded source with no page-window
footprint.
Full contract, recipes, and known limits: Multiple sources on one page.
DataLayer Source
Captures events from a GTM-style dataLayer array.
window.dataLayer.push({
event: 'view_item',
ecommerce: { items: [{ item_id: 'P123' }] },
});
// arrives as the event 'dataLayer view_item' (settings.prefix, default
// 'dataLayer'), the other keys as its data; a source mapping renames it
See packages/web/sources/dataLayer/ for implementation.
Server Sources
Handle HTTP requests in cloud functions. Server sources use the context pattern:
import type { Source } from '@walkeros/core';
import { createRespond, toEventList } from '@walkeros/core';
export const sourceCloudFunction: Source.Init<Types> = async (context) => {
const { config = {} } = context;
// Apply defaults inline: flow.json is developer-controlled, so no
// runtime validation. Shape checks live in ./schemas and are used by
// `walkeros validate` and dev tooling, never at runtime.
const userSettings = config.settings || {};
const settings = {
...userSettings,
// example default: port: userSettings.port ?? 3000,
};
const push = async (req: Request, res: Response): Promise<void> => {
const respond = createRespond((options) => {
res.status(options.status ?? 200).json(options.body);
});
const scope = buildScope(req); // see Per-Scope Context below
await context.withScope(scope, respond, async (env) => {
for (const event of toEventList(scope.body)) await env.push(event);
});
respond({ body: { success: true } }); // a no-op if a step responded
};
return { type: 'cloudfunction', config: { ...config, settings }, push };
};
See packages/server/sources/gcp/ for implementation.
Forward the whole body. A server source hands the collector the full
DeepPartialEvent it received, renaming only its wire-level event-name key. It
must not project the body through a field allowlist, or it strips source
provenance (release, trace) and other legitimate fields. express and
fetch are the reference pass-through implementations.
Env Pattern (Dependency Injection)
Platform dependencies go through env with fallback to globals or direct
imports. This enables testing and simulation without touching globals.
// Express source: env.express ?? express (import fallback)
const expressLib = env.express ?? express;
const app = expressLib();
// Web sources: env.window ?? window (global fallback)
const win = env.window ?? window;
const doc = env.document ?? document;
Every source's Env interface extends Source.BaseEnv with optional platform
deps:
export interface Env extends Source.BaseEnv {
window?: Window & typeof globalThis; // web sources
document?: Document; // web sources
express?: typeof express; // express source
cors?: typeof cors; // express source
}
Tests inject mocks via env instead of mocking globals. See
testing-strategy.
env.push for events, env.elb for commands
BaseEnv gives every source two outward exits, and they are not
interchangeable:
| Exit | Use for | What it is |
|---|---|---|
env.push | Events (event objects only) | The collector's per-source pipeline (wrappedPush) |
env.elb | walker * commands | The raw collector adapter |
Emit events through env.push. That is what makes the source's own
next/before chains, mapping (including ignore and consent), cache,
state, per-source Ingest, and collector.status.sources.<id> apply. A
source that emits events through env.elb bypasses all of it, so those config
fields become silently dead: accepted, never executed.
Commands must keep using env.elb. env.push takes WalkerOS.DeepPartialEvent
only, so a command string has no meaning there.
Both exits, plus env.command, env.logger and env.sources, are
runtime-owned: the collector writes these five capabilities into the env last,
so an author value for any of them is ignored, uniformly. env stays the
author's bag for platform and vendor dependencies only. To end the pipeline
somewhere else, declare terminus on the source registration (a sibling of
code); it is explicit and total, and stored flow configs cannot carry it
(validation rejects unknown source keys).
// Event → pipeline
await env.push({ name: 'page view', data: { title } });
// Command → raw collector
await env.elb('walker consent', { marketing: true });
Call env.push with the event alone. The collector supplies id, ingest,
mapping and preChain itself; passing your own options is for server
sources threading a per-request scope via context.withScope.
Two consequences for testing a source:
- The pipeline's end is captured when the source is constructed. Replacing
collector.pushafterstartFlowresolves is invisible to pipeline pushes, so assert on a spy destination instead of acollector.pushmock. For a deterministic raw capture at the source→collector boundary, declareterminuson the registration: it receives the event exactly as the source emitted it, and the entire pipeline is skipped (nobefore/nextchains, nocache, nostate, nomapping, no minted span id, no ingest, norespond, nostatus.sourcescounting, no Observe records). - A
next/beforechain addsawaithops between the emit and delivery. For a fire-and-forget emit, settle microtasks before asserting.
Transformer Wiring
Sources can wire to pre-collector transformer chains via the next property:
sources: {
browser: {
code: sourceBrowser,
next: 'validate' // First transformer to run after this source
}
}
The transformer chain runs before events reach the collector. See understanding-transformers for chain details.
Per-Scope Context (server sources)
A single source factory instance handles many concurrent invocations: Express
processes overlapping requests, Lambda reuses one handler across calls, queue
consumers loop over messages. Each logical unit of work is a scope. Server
sources MUST wrap each invocation with
context.withScope(rawScope, respond, body):
A server source's job is to adapt its platform's request into the shared
Source.Scope shape, so everything downstream is source-independent. Build the
scope first, then pass it:
const push = async (req, res) => {
const respond = createRespond((options) => {
/* wire options into res */
});
const scope = buildScope(req); // normalize: method, url, path, query,
// headers, body, ip, raw
await context.withScope(scope, respond, async (env) => {
for (const event of toEventList(scope.body)) {
await env.push(event);
}
});
};
normalizeHeaders, normalizeQuery and normalizeBody from @walkeros/core
do the per-field work; toEventList and isBatchBody decide how many events
the body carries. Never pass the platform object straight to withScope: the
parameter is typed Source.Scope | undefined, and a Headers instance or a
WHATWG Request cannot be descended by a mapping path at all.
Inside body, env.push carries that scope's ingest (extracted from the
scope via config.ingest mapping) and respond end to end through the
pipeline. Concurrent scopes never share ingest or respond.
ingest is request scoped, not event scoped. With N events in one scope, a
transformer writing per-event annotations into ingest records only the last.
Browser sources skip withScope. A browser tab is a single logical scope
for its lifetime; calling env.push directly is correct.
Response Delegation (env.respond)
When a server source passes a respond to withScope, every transformer and
destination in the pipeline can call
env.respond?.({ body, status?, headers? }) to customize the HTTP response.
First call wins (createRespond is idempotent), so the source's default
response is a no-op if a step already responded.
See @walkeros/server-source-express for the reference implementation.
createTrigger Pattern
Every source exports a createTrigger factory from its examples (dev entry)
that follows the unified Trigger.CreateFn interface:
type CreateFn<TContent, TResult> = (
config: Collector.InitConfig,
options?: unknown,
) => Promise<Trigger.Instance<TContent, TResult>>;
createTrigger simulates real-world invocations from the outside: full
blackbox, no source instance access. Each package implements it differently:
| Source | Content | Trigger type | Mechanism |
|---|---|---|---|
| Browser | HTML string | click, load... | DOM injection + events |
| Express | HTTP req shape | POST, GET | Real fetch() requests |
The trigger lazily calls startFlow(config) on first invocation. Tests capture
events via spy destinations. See
using-step-examples for testing
patterns.
Setup (optional)
Sources can implement an optional setup() lifecycle to provision external
resources, for example registering a webhook with a third-party provider,
creating a Pub/Sub subscription, or pre-allocating queue resources. Setup is
never invoked by the runtime, push, init, or deploy. It runs only when an
operator explicitly types walkeros setup source.<name>.
Today the command cannot run a source's setup. It reads the package's export
as an object with a setup method, while a source exports its init function, so
it stops with Package <name> has no default export (or
has no export "<name>" when the step sets import) and creates nothing.
Provision a source's resources with the platform's own tooling;
walkeros-create-source
names the packages affected.
The signature is
(ctx: LifecycleContext<Config<T>, Env<T>>) => Promise<unknown>, where
LifecycleContext carries { id, config, env, logger }. Idempotency is the
package's responsibility: the framework adds no opinion. Use
resolveSetup(ctx.config.setup, DEFAULTS) from @walkeros/core to normalize
the boolean | object shape into a concrete options object.
See walkeros-create-source,
walkeros-understanding-destinations,
walkeros-understanding-stores, and
the walkeros setup CLI documentation for the authoring template and operator
workflow.
Related Skills
- walkeros-understanding-flow - How sources fit in architecture
- walkeros-understanding-events - Events that sources emit
- walkeros-create-source - Create new source
Source Files:
- packages/core/src/types/source.ts - Interface
Package READMEs:
- packages/web/sources/browser/README.md - Browser source
- packages/web/sources/dataLayer/README.md - DataLayer source
Documentation:
- Website: Sources - Overview
- Website: Browser Source - Browser docs
- Website: Create Your Own - Guide
Understanding walkerOS Stores
Overview
Stores provide key-value storage that other components (sources, transformers, destinations) consume via environment injection. They are the 4th component type in Flow.Json alongside sources, transformers, and destinations.
Core principle: Stores are passive infrastructure. They don't process events or participate in chains; they provide state that other components read and write.
Store interface
See packages/core/src/types/store.ts for the canonical interface.
Instance
| Property | Type | Purpose | Required |
|---|---|---|---|
type | string | Store type identifier | Required |
config | Store.Config | Settings and env | Required |
get | (key) => StoreValue | undefined | Read a value | Required |
set | (key, value, ttl?) => void | Write a value (see below) | Required |
delete | (key) => void | Remove a value | Required |
destroy | DestroyFn | Cleanup on shutdown | Optional |
All methods can be sync or async (return Promise).
Value type and file mode
Stores hold one canonical value type: structured data (StoreValue), with
binary (Uint8Array) as a first-class leaf. StoreValue is
string | number | boolean | null | Uint8Array | StoreValue[] | { [key: string]: StoreValue }
(undefined is reserved as the "miss" sentinel and is never a stored value). A
shared codec in @walkeros/core/node (serializeStoreValue /
deserializeStoreValue, Node only, StoreCodecError on a bad value)
round-trips that value to and from each backing.
Store.Config.file?: boolean (default false) picks the mode, decided once at
init:
- Structured (default): values are
StoreValuedata, serialized by the shared codec. - File (
file: true): a byte-native backend (fs, S3, GCS) persists raw bytes byte-exact.set()accepts aUint8Arrayorstringand stores it untouched;get()hands the exact bytes back. Use for serving assets such as walker.js. The Sheets store is structured-only and rejectsfile: trueat init.
TTL is owned by the cache layer, not the store. The store persists values; a
cache wrapper manages expiry. The ttl argument of set is only a hint for
TTL-native backends: the fs, S3, GCS and Sheets stores ignore it and keep the
value. flow_validate warns when a store sets both file: true and cache,
and when a @walkeros/server-transformer-file is wired to a byte-native store
that does not set file: true.
Init function (context pattern)
Stores use the same context-based init pattern as other components:
import type { Store } from '@walkeros/core';
export const storeMyStore: Store.Init = (context) => {
const { config, env, logger, id } = context;
const settings = config.settings || {};
return {
type: 'my-store',
config,
get(key) {
/* ... */
},
set(key, value, ttl) {
/* ... */
},
delete(key) {
/* ... */
},
destroy() {
/* cleanup */
},
};
};
Context contains:
| Property | Type | Purpose |
|---|---|---|
config | Store.Config | Settings from flow config |
env | Store.Env | Environment dependencies |
logger | Logger.Instance | Scoped logger |
id | string | Store identifier |
collector | Collector.Instance | Reference to collector |
Lifecycle
Stores have the simplest lifecycle of all component types:
Startup: Stores → Destinations → Transformers → Sources
Shutdown: Sources → Destinations → Transformers → Stores
- Init first: Stores init before all other components so they're available when sources/transformers/destinations start
- Destroy last: Stores destroy after all other components so transformers can flush during their own destroy
- No lazy init: Unlike destinations, stores don't support
requireor deferred activation; they are always eager
Wiring stores via $store.
Bundled mode (flow.json)
Use $store.storeId in a component's env to inject a store instance:
{
"stores": {
"assets": {
"package": "@walkeros/server-store-fs",
"config": { "file": true, "settings": { "basePath": "./public" } }
}
},
"transformers": {
"file": {
"package": "@walkeros/server-transformer-file",
"config": { "settings": { "prefix": "/static" } },
"env": { "store": "$store.assets" }
}
}
}
The bundler resolves $store.assets to a runtime reference. Invalid references
are caught at build time. walkeros validate also catches typos at validation
time, including unknown store names and the colon-instead-of-dot mistake (e.g.
$store:assets is flagged with the suggested form $store.assets).
Integrated mode (TypeScript)
Pass the store definition object itself, no $store. prefix:
import { startFlow } from '@walkeros/collector';
import { storeFsInit } from '@walkeros/server-store-fs';
import { transformerFile } from '@walkeros/server-transformer-file';
const assets = {
code: storeFsInit,
config: { file: true, settings: { basePath: './public' } },
};
await startFlow({
stores: { assets },
transformers: {
file: {
code: transformerFile,
env: { store: assets }, // same object as stores.assets
},
},
});
Note: In integrated mode, put the same store definition object in stores and
in env. The collector replaces it with the initialized instance at startup,
matched by object identity (a copy does not match), and only for top-level env
keys. The $store. string prefix is a bundler feature: in startFlow it stays
a plain string.
Available stores
Built-in cache tier (__cache)
The collector ships a built-in in-memory cache with LRU eviction, TTL, and
entry/byte caps. Enable it on any store by setting Flow.Store.cache. No
separate package import needed:
{
"stores": {
"files": {
"package": "@walkeros/server-store-fs",
"config": { "settings": { "basePath": "./data" } },
"cache": { "rules": [{ "ttl": 60 }] }
}
}
}
Use this for the common "cache in front of a slow backing store" pattern (API, GCS, Sheets, etc.).
@walkeros/server-store-fs (filesystem)
File-based store for serving static assets. Server-only.
import { storeFsInit } from '@walkeros/server-store-fs';
@walkeros/server-store-s3 (S3-compatible object storage)
S3-compatible store using s3mini (~26 KB minified, zero dependencies). Works
with AWS S3, Cloudflare R2, Scaleway, DigitalOcean Spaces, Backblaze B2, MinIO,
and any S3-compatible provider. Structured by default (stored as
application/json); set file: true to serve raw bytes byte-exact with the
real mime. Server-only.
import { storeS3Init } from '@walkeros/server-store-s3';
Settings:
| Setting | Type | Required | Default | Purpose |
|---|---|---|---|---|
bucket | string | Yes | - | S3 bucket name |
endpoint | string | Yes | - | S3-compatible endpoint URL |
accessKeyId | string | Yes | - | S3 access key ID |
secretAccessKey | string | Yes | - | S3 secret access key |
region | string | No | "auto" | AWS region (SigV4 signing) |
prefix | string | No | - | Key prefix for scoping |
Primary use case: Serving static files in managed deployments where files live in a bucket rather than being baked into a Docker image.
@walkeros/server-store-gcs (Google Cloud Storage)
Zero-dependency GCS store using raw fetch + GCS JSON API. Built-in auth: ADC
on Cloud Run / GKE, or explicit service account JWT. Server-only.
import { storeGcsInit } from '@walkeros/server-store-gcs';
Settings:
| Setting | Type | Required | Default | Purpose |
|---|---|---|---|---|
bucket | string | Yes | - | GCS bucket name |
prefix | string | No | - | Key prefix for scoping |
Credentials live at config.credentials (sibling of settings): a
service-account JSON (string | object) for non-GCP environments, set as
$secret.<NAME> (an unset secret stops the flow at startup); omit it for ADC. A
value missing client_email or private_key is ignored without a log line, and
the store falls back to ADC, the runtime's default credentials.
Primary use case: Serving static files on GCP infrastructure (Cloud Run, GKE) where ADC handles authentication.
@walkeros/server-store-sheets (Google Sheets)
Zero-dependency Google Sheets store using raw fetch + Sheets v4 REST API. One
row per key, one cell per value (JSON-serialized). Built-in auth shared with the
GCS store. Server-only. Structured-only: cells hold structured JSON, so it
rejects file: true at init and rejects values carrying a binary (Uint8Array)
leaf. Use fs, S3, or GCS for byte-exact serving.
import { storeSheetsInit } from '@walkeros/server-store-sheets';
Settings:
| Setting | Type | Required | Default | Purpose |
|---|---|---|---|---|
id | string | Yes | - | Spreadsheet ID (segment in URL) |
sheet | string | No | 'Sheet1' | Sheet (tab) name |
key | string | No | 'A' | Column letter for keys |
value | string | No | 'B' | Column letter for JSON values |
headerRows | number | No | 1 | Header rows to skip when reading |
Credentials live at config.credentials (sibling of settings): a
service-account JSON (string | object) for non-GCP environments, set as
$secret.<NAME> (an unset secret stops the flow at startup); omit it for ADC. A
value that is not valid JSON, or is missing client_email or private_key, is
ignored without a log line, and the store falls back to ADC, the runtime's
default credentials.
Primary use case: Demos and small prototypes where the spreadsheet is the
operator-facing UI for tweaking lookup data. Quota: 60 read and 60 write
requests per minute per user per project. Enable the built-in cache via
Flow.Store.cache on the store declaration to absorb the quota, otherwise quota
burns in seconds. Not a production CRM substitute. See
Website: Sheets Store for the
cache-wiring example.
@walkeros/server-store-api (third-party HTTP API)
Read-only store for a third-party API, such as a CRM. settings.url is an
absolute URL with one {key} placeholder after the host; each get is one GET
with the encoded key. A 404, 204 or JSON null is a miss; any other failure
throws a StepError. Pass an optional bearer token as config.credentials
({ "token": "$secret.NAME" }). set, delete and file: true are refused.
Misses are not cached by the cache block.
import { storeApiInit } from '@walkeros/server-store-api';
Stores in flow config
Declare any store consumed by one or more components in the stores section of
the flow config and wire it via $store.<id> in component env. The built-in
cache tier (Flow.Store.cache) covers the "fast in-memory cache in front of a
slow backing store" pattern without a separate memory store.
Accessing stores at runtime
After startFlow(), stores are available on the collector instance:
const { collector } = await startFlow({
stores: {
files: { code: storeFsInit, config: { settings: { basePath: './data' } } },
},
});
// Read/write
await collector.stores.files.set('key', 'value'); // fs keeps it, no expiry
const value = await collector.stores.files.get('key');
await collector.stores.files.delete('key');
Hooks
Store operations (get, set, delete) are wrapped with useHooks during
initialization, enabling pre/post interception via the collector's hooks system.
Available hook names:
| Hook name | Pre hook | Post hook |
|---|---|---|
StoreGet | preStoreGet | postStoreGet |
StoreSet | preStoreSet | postStoreSet |
StoreDelete | preStoreDelete | postStoreDelete |
Hooks fire on every store operation regardless of which component triggered it
(cache system, transformer via env, destination via env, direct access on
collector.stores).
const { collector } = await startFlow({
stores: {
files: { code: storeFsInit, config: { settings: { basePath: './data' } } },
},
});
// Intercept all store reads
collector.hooks.preStoreGet = ({ fn }, key) => {
console.log('Reading key:', key);
return fn(key);
};
Key differences from other components
| Aspect | Sources/Transformers/Destinations | Stores |
|---|---|---|
| Event flow | Participate in push chain | No push, no chain |
next/before | Chain connection fields | None (passive) |
| Lifecycle | Init after stores | Init first, destroy last |
require | Deferred activation supported | Always eager |
| Interface | push(event, context) | get/set/delete(key) |
Dev exports (dev.ts)
Every store package must export schemas and examples via dev.ts, matching the
convention used by sources and destinations:
Required files
src/
├── schemas/
│ ├── settings.ts # Zod schema for store settings
│ └── index.ts # Re-export + zodToSchema() conversion
├── examples/
│ └── index.ts # Example Store.Config objects
└── dev.ts # export * as schemas; export * as examples
Schema pattern
// src/schemas/settings.ts
import { z } from '@walkeros/core/dev';
export const SettingsSchema = z.object({
mySetting: z.string().describe('Setting description for docs and MCP'),
});
export type Settings = z.infer<typeof SettingsSchema>;
// src/schemas/index.ts
import { zodToSchema } from '@walkeros/core/dev';
import { SettingsSchema } from './settings';
export { SettingsSchema, type Settings } from './settings';
export const settings = zodToSchema(SettingsSchema);
tsup + package.json
// tsup.config.ts
import { defineConfig, buildModules, buildDev } from '@walkeros/config/tsup';
export default defineConfig([buildModules(), buildDev()]);
Add ./dev export to package.json:
"./dev": {
"types": "./dist/dev.d.ts",
"import": "./dist/dev.mjs",
"require": "./dist/dev.js"
}
Why this matters
buildDev()generatesdist/walkerOS.jsonat build time- MCP tools fetch
walkerOS.jsonfrom CDN for package discovery and schema validation - Website docs import that
walkerOS.jsonand render the settings with<Configuration type="store" data={data} />instead of hardcoded markdown tables - Without
dev.ts, a store is invisible to MCP and docs tables rot
Hints (Optional)
Stores can optionally export hints: lightweight, actionable context for AI
agents beyond what schemas and examples already provide. Create src/hints.ts:
import type { Hint } from '@walkeros/core';
export const hints: Hint.Hints = {
'persistence-behavior': {
text: 'Describes persistence guarantees. See settings schema for options.',
},
};
Export from src/dev.ts alongside schemas and examples:
export * as schemas from './schemas';
export * as examples from './examples';
export { hints } from './hints';
Most stores don't need hints; only add them for non-obvious behaviors, prerequisites, or troubleshooting patterns.
Setup (optional)
Stores can implement an optional setup() lifecycle to provision external
resources, for example creating a SQLite table, initializing an S3 bucket, or
running a one-off schema migration. Setup is never invoked by the runtime,
push, init, or deploy. It runs only when an operator explicitly types
walkeros setup store.<name>.
The signature is
(ctx: LifecycleContext<Config<T>, Env<T>>) => Promise<unknown>, where
LifecycleContext carries { id, config, env, logger }. Idempotency is the
package's responsibility: the framework adds no opinion. Use
resolveSetup(ctx.config.setup, DEFAULTS) from @walkeros/core to normalize
the boolean | object shape into a concrete options object.
See walkeros-create-destination,
walkeros-create-source,
walkeros-understanding-destinations,
walkeros-understanding-sources,
and the walkeros setup CLI documentation for the authoring template and
operator workflow.
Related Skills
- walkeros-understanding-flow - Architecture and Flow structure
- walkeros-understanding-transformers - Transformer interface and env pattern
- walkeros-using-store-cache -
Store-level cache and the declarative
stateblock for fetch/stash without$code:, includingstate.mappingto pick and shape what a lookup returns (paths relative to the value read, result merged into the target, declared fallbacks applied on a miss).statenever uses afile: truestore. Full reference: Website: State - walkeros-using-cli - Bundling flows with stores
Source files:
- packages/core/src/types/store.ts - Store types
- packages/collector/src/store.ts - initStores implementation
- packages/server/stores/fs/src/ - Filesystem store package
- packages/server/stores/s3/src/ - S3 store package
- packages/server/stores/gcs/src/ - GCS store package
- packages/server/stores/sheets/src/ - Google Sheets store package
Documentation:
- Website: Flow - Flow configuration
- Website: Stores - Stores overview
- Website: S3 Store - S3 store documentation
- Website: FS Store - Filesystem store documentation
- Website: GCS Store - GCS store documentation
- Website: Sheets Store - Google Sheets store documentation
Understanding walkerOS Transformers
Overview
Transformers are middleware for validating, enriching, and redacting events in the walkerOS pipeline. They run in chains at configurable points between sources, collector, and destinations.
Core principle: Transformers transform events. They don't capture (sources) or deliver (destinations); they modify events in-flight.
Use Cases
| Use Case | Purpose | Example |
|---|---|---|
| Validate | Ensure events match schema contracts | JSON Schema validation |
| Enrich | Add server-side data to events | User segments, geo data |
| Redact | Remove sensitive data before destinations | Strip PII, anonymize IPs |
Available Packages
| Package | Env | Purpose |
|---|---|---|
@walkeros/transformer-validate | both | Enforce JSON Schema contracts on events at runtime |
@walkeros/transformer-ga4 | both | Decode GA4 Measurement Protocol hits into events |
@walkeros/server-transformer-bot | server | Annotate an automation score and client category |
@walkeros/server-transformer-fingerprint | server | Hash request fields into a rotating visitor id |
@walkeros/server-transformer-file | server | Serve static files from a store |
File step answers
@walkeros/server-transformer-file owns the paths inside settings.prefix
("/" owns every routed path; without a prefix it owns none). A hit answers
with the file, a miss inside the prefix answers 404, and a failing store read
answers 503. A missing store stops the flow at startup. Route only file requests
to a step that owns /, as flow-complete does with a .js case: any other path
routed to it answers 404 and its event stops there.
Contract validation
@walkeros/transformer-validate is the runtime arm of a
contract. Event shapes
live in the top-level contract block; the transformer references one via
$contract.<name> in its contract setting and validates the canonical event.
mode: "strict" drops invalid events (chain-stop); mode: "pass" (default)
annotates event.source.valid and continues so a downstream step can route on
the verdict. format: true additionally checks the canonical WalkerOS.Event
structure. Filtering is the same mechanism: an inline schema that rejects the
unwanted events plus mode: "strict"; there is no separate ignore field. See
Website: Validate transformer.
Transformer Interface
See packages/core/src/types/transformer.ts for canonical interface.
Init Function (Context Pattern)
Transformers use a context-based initialization pattern:
import type { Transformer } from '@walkeros/core';
export const transformerMyTransformer: Transformer.Init<Types> = (context) => {
const { config = {}, env, logger, id } = context;
// Apply defaults inline: flow.json is developer-controlled, so no
// runtime validation. Shape checks live in ./schemas and are used by
// `walkeros validate` and dev tooling, never at runtime.
const userSettings = config.settings || {};
const settings = {
...userSettings,
// example default: threshold: userSettings.threshold ?? 100,
};
return {
type: 'my-transformer',
config: { ...config, settings },
push(event, pushContext) {
// Process event
return { event };
},
};
};
Init Context contains:
| Property | Type | Purpose |
|---|---|---|
config | Transformer.Config | Settings, mapping, next chain |
env | Types['env'] | Environment deps (stores via $store.) |
logger | Logger | Logging functions |
id | string | Transformer identifier |
collector | Collector.Instance | Reference to collector |
ingest | Ingest | Empty (_meta only); see Push Context |
Instance Methods
| Member | Purpose | Required |
|---|---|---|
type | Transformer type identifier | Required |
config | The transformer's config | Required |
push | Process event, return modified/false | Required |
init | One-time initialization | Optional |
destroy | Cleanup resources | Optional |
Return Values
The push function controls event flow:
| Return | Behavior |
|---|---|
{ event } | Continue chain with modified event |
void | Continue chain, event unchanged |
false | Stop chain, event dropped |
{ event, next } | Route via next in place of its own next, then the chain continues |
{ event, respond } | Continue chain with wrapped respond function |
Result[] | Fork: each result is its own copy with a derived event.id |
A push that throws also stops the chain and drops the event; the collector
logs it and counts it in status.failed. Catch a library error inside push
when the event should continue.
push(event, context) {
if (!event.data?.id) {
context.logger.error('Missing required id');
return false; // Stop chain
}
event.data.enrichedAt = Date.now();
return { event }; // Continue with modified event
}
Inline Code Transformers
For simple transformations without external packages, use inline code with the
$code: string prefix in JSON configs. The $code: prefix tells the CLI
bundler to parse the following string as executable JavaScript:
{
"transformers": {
"enrich": {
"code": {
"push": "$code:(event) => { event.data.enrichedAt = Date.now(); return { event }; }"
},
"next": "validate"
}
}
}
Inline code structure:
| Property | Purpose |
|---|---|
code.init | Code run once during initialization |
code.push | Code run for each event |
Push code has access to:
event: the event being processedcontext: push context with logger, config, etc.
Return values in push code:
- Return
{ event }to continue chain with modified event - Return
undefinedto pass event unchanged - Return
falseto drop event from chain
Example: Filtering internal events
{
"transformers": {
"filter": {
"code": {
"push": "$code:(event) => { if (event.name.startsWith('internal_')) return false; return { event }; }"
}
}
}
}
Mixing inline and package transformers:
{
"transformers": {
"addTimestamp": {
"code": {
"push": "$code:(event) => { event.data.processedAt = new Date().toISOString(); return { event }; }"
},
"next": "validate"
},
"validate": {
"package": "@walkeros/transformer-validate",
"config": { "settings": { "format": true } }
}
}
}
Pipeline Integration
Transformers run at three points around the collector:
Source → [Source chain] → Collector → [Collector chain] → fan-out → [Destination chain] → Destination
(source.next) (collector.next) (destination.before)
Source Chain
Runs after source captures event, before collector enrichment:
sources: {
browser: {
code: sourceBrowser,
next: 'validate' // First transformer in the source chain
}
}
Collector Chain
Runs once per event after collector enrichment, before the destination fan-out.
Every destination receives its output; a stop here (or a transformer returning
false) drops the event for all destinations:
startFlow({
next: ['bot', 'validate'], // collector.next in flow.json
// ...
});
Destination Chain
Runs per destination, after the collector chain, before that destination
receives the event. Use it for per-destination work and filtering; a stop here
skips only this destination:
destinations: {
gtag: {
code: destinationGtag,
before: 'redact' // First transformer in the destination chain
}
}
Chain Linking
Transformers link together via next. The chain starts at the step that names
its first transformer, here the source:
sources: {
express: {
code: sourceExpress,
next: 'fingerprint' // Chain head
}
},
transformers: {
fingerprint: {
code: transformerFingerprint,
config: {
settings: { salt: process.env.FINGERPRINT_SALT }, // Required
next: 'enrich' // Chain to next transformer
}
},
enrich: {
code: transformerEnrich,
config: { next: 'redact' }
},
redact: {
code: transformerRedact
// No next = end of chain
}
}
Branching and fan-out
Transformers can redirect events to different chains using the branch()
factory from @walkeros/core. The route replaces the transformer's own next
for this event; after it, the enclosing array continues:
import { branch } from '@walkeros/core';
push(event, context) {
return branch(event, 'parser'); // Single target
return branch(event, ['a', 'b']); // Sequence: a, then b
return branch(event, { many: ['a', 'b'] }); // Fork: two copies
return branch(event, { stop: true }); // End this copy
}
Conditional routing is built into the next/before properties through the
one operator, so no router transformer is needed:
"next": {
"one": [
{ "match": { "key": "ingest.path", "operator": "prefix", "value": "/api" }, "next": "apiHandler" },
{ "next": "default" }
]
}
one entries are evaluated in order, first match wins. A RouteConfig is a
disjoint union: each config sets at most one of next (gated link), one
(first-match dispatch), many (all-match fan-out), or stop (drop), never more
than one. Every match reads { ingest, event } and is evaluated when the
event reaches it, so it sees what earlier steps wrote. Besides _meta, ingest
holds only what the source's config.ingest maps and what earlier steps write
into it: without { "map": { "path": "path" } } in that config, ingest.path
is empty and the /api entry never matches. An entry with no match always
matches (use it as a fallback). If no entry matches, the event passes through
unchanged. Use many (allowed in every chain field) when every matching branch
should run: each matching entry becomes an independent copy of the event with
its own derived event.id, and each copy finishes the rest of the path on its
own (never merged):
"next": {
"many": [
{ "match": { "key": "event.consent.analytics", "operator": "eq", "value": "true" }, "next": "ga4Pipeline" },
{ "next": "auditLog" }
]
}
A consent value is a boolean, and eq compares it as text, so a granted consent
matches "true".
Paths and pass-through steps
walkerOS uses two vocabulary terms for chain composition:
- Path: the multi-step chain that an event walks through a flow's
transformerssection. - Pass-through step (short: pass): a single step inside a path that
declares no
codeand nopackage. The runtime synthesizes the push, so the step contributes structure without shipping executable code.
A pass-through step ships in three variants. Each variant uses a different operative field; combine them on the same step when it helps.
Variant 1: chain-only (before / next)
A named hop that shares a chain across multiple call sites. Use it to avoid
duplicating arrays in before / next references:
{
"transformers": {
"validateThenEnrich": {
"before": ["validate", "enrich"]
}
},
"destinations": {
"gtag": {
"package": "@walkeros/web-destination-gtag",
"config": { "settings": { "ga4": { "measurementId": "G-DEMO123456" } } },
"before": "validateThenEnrich"
},
"meta": {
"package": "@walkeros/web-destination-meta",
"config": { "settings": { "pixelId": "1234567890" } },
"before": "validateThenEnrich"
}
}
}
Variant 2: cache-only
A step that declares only a cache block. Useful for deduplication or
short-circuit halts. cache.stop: true in a source chain or collector.next
halts the event for all destinations (not just the local chain); in a
destination.before chain it skips the rest of that chain and the destination
receives the cached event:
{
"transformers": {
"dedup": {
"cache": {
"stop": true,
"rules": [{ "key": ["event.id"], "ttl": 60 }]
}
}
}
}
Variant 3: mapping-only
A step that declares only a mapping: Mapping.Config. The runtime synthesizes a
push that calls processEventMapping and mutates the event in-flight:
{
"transformers": {
"redactPII": {
"mapping": {
"policy": {
"user.email": { "value": "[redacted]" }
}
}
}
}
}
See walkeros-understanding-mapping
for the mapping primitives (policy, data, mapping[].name, etc.) and the
"Mapping at the transformer position" section for the dual semantic.
Mapping at transformer position vs destination position
mapping is the same field shape (Mapping.Config) in both positions, but the
semantic is disambiguated by where the step sits:
| Position | What mapping produces |
|---|---|
| Destination | A vendor-shaped payload (the destination consumes data) |
| Transformer | A mutated event that continues through the chain |
At the transformer position, only event-mutating fields apply: policy, and per
rule condition, policy, name and ignore. consent, include, data
and per rule consent, include, remove, batch, settings, extend,
data, silent do nothing there. A config.mapping wins over the step-level
mapping; next to code or package, a mapping never runs.
walkeros validate and the bundle reject all of these (MAPPING_NO_OP), and
the collector logs one ERROR line for them when the transformer registers.
mapping[].ignore: true drops the event from the chain (not "skip this
destination", which is the destination-position semantic).
Closed schema (unknown keys are errors)
Transformer step entries follow a closed schema. Unknown keys at the top of
a step are validation errors (UNKNOWN_KEY; the message lists the allowed keys,
among them package, import, code, config, env, before, next,
cache, state and mapping). This catches misrouted keys (e.g.
{ rules: [], stop: true } placed at the top of a step instead of nested under
cache:) at validate time instead of letting them silently pass through at
runtime.
An empty {} is a valid no-op step. Declaring both code and package on the
same step is rejected with CONFLICT.
Chain resolution safety
Every chain position runs through one runner (runTransformerChain in
@walkeros/collector) over the core continuation stack (startChain /
advanceChain). Routes resolve per hop via getNextSteps(spec, root), which
needs the root { ingest, event } and returns NextSteps: ids (plus a
continuation for the rest of a sequence), a stop, or forks. A member next that
leads back to a step already on its insertion path is skipped, and each copy is
capped at 256 steps. A step listed twice in an array runs twice. An unknown
transformer id is logged as an ERROR line and skipped, and the chain continues;
walkeros validate reports it as an UNKNOWN_ROUTE_TARGET error. Static tools
without an event use getRouteGraph(spec, transformers?), which enumerates
every branch of the same compiled route.
Composition principle
A transformer owns its own chain. When a chain references a transformer by name,
that transformer's own before chain runs before its push, and its next chain
after, both recursively, with cycle detection. This holds inside an explicit
array too: the array is the backbone, and a member's own next is inserted
right after that member before the array continues (["bot", "validate"] with
bot.next = "foo" runs bot, foo, validate). Cache halt signals
(cache.stop: true) in a source chain or collector.next propagate to every
destination. The grammar's recursive Route shape
(string | Route[] | RouteConfig) compiles element-by-element, so sequences can
mix transformer IDs and inline one / many / next routes
(next: ["dedup", { one: [...] }] is valid). This is the model to default to
when adding new chain primitives.
See walkeros-understanding-flow for the full connection rules between sources, transformers, and destinations.
Push Context
The push function receives a context with event metadata:
| Property | Purpose |
|---|---|
config | Transformer configuration |
env | Environment dependencies |
logger | Scoped logger for output |
id | Transformer identifier |
collector | Access to collector instance |
ingest | Request metadata from source |
push(event, context) {
const { logger, id, ingest } = context;
logger.debug('Processing', { transformer: id, event: event.name });
// Access request metadata if available
if (ingest?.ip) {
event.data = { ...event.data, clientIp: ingest.ip };
}
return { event };
}
Response Delegation (env.respond)
Transformers can customize HTTP responses by calling
context.env.respond?.({ body, status?, headers? }). This is useful for
validation transformers that reject events with custom error responses, or
transformers that short-circuit the pipeline. First call wins (idempotent). The
respond function is optional, only present when the source provides one.
Transformer Paths
| Path | Description |
|---|---|
packages/transformers/ | Transformers for both platforms (validate, ga4) |
packages/server/transformers/ | Server transformers (bot, fingerprint, file) |
Related Skills
- walkeros-understanding-flow - How transformers fit in architecture
- walkeros-create-transformer - Create new transformer
- walkeros-using-store-cache -
Declarative
stateblock for fetch/stash without$code:
Source Files:
- packages/core/src/types/transformer.ts - Interface
Documentation:
- Website: Transformers - Overview
- Website: Create Your Own - Guide
Using the walkerOS CLI
Overview
The walkerOS CLI (walkeros) validates, simulates, tests and bundles event
collection flows. It does not run them: a built server flow runs with
runneros start from @walkeros/runner (the walkeros/flow image).
Core workflow:
- Configure - Write Flow.Json JSON config
- Validate -
walkeros validate flow.json - Test - Simulate events (mocked) or push (real)
- Bundle -
walkeros bundle flow.json -o dist/ - Run -
runneros start dist/flow.mjs, or deploy through the app
Quick Start
# Install (Node.js 22.12 or later)
npm install -g @walkeros/cli
# Bundle a flow into dist/
walkeros bundle flow.json -o dist/
# Test with simulated event
walkeros push flow.json -e '{"name":"page view"}' --simulate destination.demo
# Push real event
walkeros push flow.json -e '{"name":"page view"}'
An event needs name ("entity action"). One written as entity and action
only is rejected as invalid: a simulation prints
mapping: none (skipped before mapping) and no calls, a real push sends
nothing, and both still print success: true.
Commands Overview
| Command | Purpose | Safe? |
|---|---|---|
bundle | Generate JS bundle from config | ✅ |
push | Push an event: real API calls on a server flow, recorded on a web flow; --simulate mocks steps | ⚠️ |
setup | Run a component's setup() to provision external resources | ⚠️ |
validate | Validate configs/events | ✅ |
cache | Manage caching | ✅ |
auth | Check who the token in WALKEROS_TOKEN belongs to | ✅ |
api | Call the walkerOS app API, or list its operations | ⚠️ |
diagnostics | Show the CLI version, the app URL, and whether the app answers | ✅ |
telemetry | Show, enable or disable anonymous usage telemetry | ✅ |
For detailed command reference, see commands-reference.md.
Common Workflows
Development Workflow
1. Write flow.json config
2. Bundle: walkeros bundle flow.json
3. Simulate: walkeros push flow.json -e event.json --simulate destination.demo
4. Fix issues, repeat 2-3
5. Push test: walkeros push flow.json -e event.json
6. Deploy: in the walkerOS app, or
walkeros api POST /api/projects/{projectId}/flows/<flowId>/deploy
(see the CI recipe in the CLI docs)
Multi-Flow Development
# Bundle specific flow
walkeros bundle flow.json --flow myFlow
# Bundle all flows (--all needs an output directory)
walkeros bundle flow.json --all -o dist/
# Test specific flow
walkeros push flow.json --flow myFlow -e event.json --simulate destination.demo
Local Development Server
# Build, then start the artifact with the runtime (a separate package)
walkeros bundle flow.json -o dist/
npx --package=@walkeros/runner runneros start dist/flow.mjs --port 3000
Server port note: Under runneros the runtime owns the port: it listens on
--port (or PORT, default 8080), serves /health and /ready, and hands
every other request to the flow's HTTP handler. A source's own port setting is
not used there.
Flow.Json Configuration
Minimal Config
{
"version": 4,
"flows": {
"default": {
"config": {
"platform": "web",
"bundle": {
"packages": {
"@walkeros/web-destination-gtag": { "version": "^4" }
}
}
},
"destinations": {
"gtag": {
"package": "@walkeros/web-destination-gtag",
"config": {
"loadScript": true,
"settings": { "ga4": { "measurementId": "G-XXXXXX" } }
}
}
}
}
}
}
Without "loadScript": true walkerOS does not load gtag.js: unless the page
loads it itself, events go nowhere while the push still reports them as done.
Config Structure
{
"version": 4,
"flows": {
"<flowName>": {
"config": {
"platform": "web" | "server", // Platform (required)
"url": "", // Optional: public URL, read by $flow.<name>.url
"settings": {}, // Platform-specific settings (optional)
"bundle": {
"packages": {}, // NPM packages pacote will install
"overrides": {}, // Transitive dep version pins (npm-style)
"traceInclude": [] // Optional: nft escape hatch (paths/globs)
},
"observe": {} // Optional, web flows only: url + binding, level, sample
},
"sources": {}, // Event sources
"destinations": {}, // Event destinations (each with its own mapping)
"transformers": {}, // Transformer chain (optional)
"stores": {}, // Key-value stores (optional)
"collector": {}, // Collector configuration
"variables": {} // Flow-level $var values (optional)
}
}
}
You do NOT need npm install for step packages. flow.json's
config.bundle.packages is the single source of truth. Pacote installs them
transparently when you run walkeros bundle. Only @walkeros/cli belongs in
your project's package.json (as a devDependency).
config.bundle.overrides pins transitive dependency versions. Use it when a
vendor SDK's declared peer/dep range conflicts with another required version in
the same tree. Example: {"@amplitude/analytics-types": "2.11.1"} forces that
exact version everywhere in the install graph. Direct packages specs always
win over overrides; overrides only substitute transitive resolution.
Schema version stays at 4. Build-time fields live under
flow.<name>.config.bundle.{packages, overrides, traceInclude, env}. A
config.bundle.external field fails walkeros validate (Unrecognized key);
the bundle only warns and builds, so remove it.
config.observe bakes only public values. For web flows, walkeros bundle
emits the url + binding connect pair statically into the bundle; the
per-session credential arrives out-of-band at runtime (?elbObserve= URL
param), so no secret ever enters the artifact. A partial pair (only one of the
two set) warns at build time and wires nothing. A server flow ignores
config.observe: the runner reads its level from WALKEROS_OBSERVE_LEVEL
instead (see Runtime environment below).
For detailed configuration options, see flow-configuration.md.
Testing with Step Examples
Simulate with --step
Target a specific step and provide input as SourceInput
({ content, trigger? }):
# Simulate a source step with trigger metadata (a click needs its selector)
walkeros push flow.json --simulate source.browser --event '{"content":"<button data-elb=\"cta\" data-elbaction=\"click\">Sign up</button>","trigger":{"type":"click","options":"button"}}'
# Simulate a destination step with an event
walkeros push flow.json --simulate destination.gtag -e '{"name":"order complete","data":{"value":149.97}}'
A click trigger without options finds no element
(Trigger: element not found for selector "undefined") and the source emits
no events. Example output of the destination command (a destination step
prints its matched mapping key and every recorded vendor call, arguments as
compact JSON cut at 300 chars):
success: true
destination.gtag
mapping: none
call window.gtag("config","G-XXXXXX",{})
call window.gtag("event","order_complete",{"send_to":"G-XXXXXX"})
Duration: 467ms
mapping: <key>when a rule matched,mapping: nonewhen the destination pushed without a rule,mapping: none (skipped before mapping)plusno callswhen nothing was sent. Then why, when consent is the reason:pending: waits for consent (require)(never started; names the unmetrequireentries) orskipped: consent (requires marketing; granted functional)(started, event denied by itsconsentcheck; aftermapping: <key>it is that rule's ownconsent). Source, transformer and collector steps print their recorded calls, thenevent <json>orno events; a failed step printserror: <message>. A source step records the walker commands the source issues itself aselbcalls in its own call shape, e.g.call elb("walker consent",{"functional":true,"marketing":true})for a CMP,call elb("user",...)andcall elb("session",...)for the session source. Not recorded: the collector's own commands (starting consent, globals, custom, shutdown) and the wiring commandson,hook,destination.--jsonputs the same data undersimulations(one{ step, name, events, calls: [{ fn, args, ts }], mappingKey?, skipped?, error? }per simulated step, in order;skippedis{ reason: 'pending', require }or{ reason: 'consent', required, granted }; a multi-destination simulate stops at the first failure). stdout is pure JSON, logs go to stderr, so| jqworks without--silent. Values pass throughtoPrintable(@walkeros/core/node):Errorto{ name, message },Bufferto UTF-8,bigintto string,Map/Setto arrays, cycles to"[Circular]".- Text and JSON output (and MCP
flow_simulateandflow_push) are scrubbed withscrubSecrets, the same redactor the loggers use: service accounts, PEM keys,Authorization,access_token, credential-named fields and high-entropy runs show as***. The values of every$secret.NAMEthe flow references (set in the env, 6+ chars) are masked exactly, raw, JSON-escaped and URL-encoded. The step still receives the real values. With--json, a number whose printed form contains a known value becomes"***", so the JSON still parses. Simulate and a real push of a flow config route the flow's own logs through the masking CLI logger, so flow DEBUG lines need--verbose; the run's own CLI lines mask the known values too. The runner masks the secret values it fetches; values from--env-fileor the container env get the pattern rules only. - Only the target starts. Destination simulate keeps only the target
destination (no source, no other destination initializes); the flow's
transformers still start, and a store without a mock env runs for real. Source
simulate keeps only the simulated source (no browser page view or CMP decision
from another source); captured events stop at the collector. A source package
that declares
examples.env.simulation(SQS, Pub/Sub pull) runs on its mock client with those calls recorded. - No real vendor calls (destinations only). A package destination whose
export has no mock env (
examples.env.push) is refused before the flow starts (an inlinecodestep has no package and runs as given):No mock env for <package> export <exportName>: simulate would call the real vendor. Add examples.env.push to the package's dev examples.A named export of a package version withoutexportExamplesis refused with...: this package version predates export-keyed examples; use a version with exportExamples or a local path.A store whose package ships a mock env runs on it; a store without one runs for real. Sheets and GCS still need a real-format service-account key, since they sign their token request before the mockedfetchanswers (see the CLI docs, "Secrets in simulate"). --ingest <json|file|url>supplies the request context (a JSON object, e.g.{"ip":"203.0.113.7","userAgent":"Mozilla/5.0"}) fortransformer.*,collector.*anddestination.*simulation. It reaches the destinationbeforechain andcontext.ingest(orcollector.next); the CLI always sets_meta. Rejected forsource.*and a real push:--ingest applies to transformer, collector and destination simulation only. A programmaticpushCommand({ ingest })on a real push or source simulate errors the same way;simulateTransformer,simulateCollectorandsimulateDestinationtakeingestdirectly.--consent <json|file|url>(a JSON object of booleans) is the collector consent any simulated step starts from, sources included (programmaticconsent; MCP and collector:state.consent).startFlowapplies it before the flow runs, as aconsentcommand only the simulated step hears: arequire: ["consent"]destination starts, its consent check sees the granted keys, and a Consent Mode (como) target records itsgtag('consent','update',...). A consent-gated source (the session source withsettings.consent) starts from it; without it,no events. The starting consent is never recorded as a source call. Rejected for a real push only:--consent sets the collector's starting consent for a simulation; a real push uses the flow's own consent.--command <config|consent|user|run>makes a destination simulation runcollector.command(name, event)instead of a push (a step example'scommand); the event is the command's data and its calls are recorded. Starting consent applies first, then the command. Elsewhere:--command applies to destination simulation only.--page-url <url>sets the page of a simulated web source (absolutehttp/httpsonly). Precedence:--page-url, then the input trigger'soptions.url, thenhttp://localhost. Elsewhere:--page-url sets the page of a simulated web source; for request context use --ingest.
Consent mental model (what decides whether a destination receives an event):
| Setting | Effect | In simulate |
|---|---|---|
require: ["consent"] | Destination stays pending until the collector has ANY consent state | --consent starts it; else pending: waits for consent |
config.consent: { marketing } | Each event checked against collector consent plus event.consent; denied = skipped | grant via --consent or the event; else skipped |
event.consent | Per event, counted on top of collector consent | part of the -e event |
- Trace-mode limit. Simulate injects the mock env before
init, so clients built ininit(BigQuery writer, Pub/Sub client) are recorded. Runtime trace mode (an Observe session at trace level) wraps the env per push only, so calls through a client built ininitare not recorded there.
Same flow via MCP (flow_simulate)
From an AI assistant the equivalent tool is flow_simulate. A few specifics:
stepis required. Pass the target as"type.name", e.g."source.browser","collector.default","destination.gtag", or"transformer.router". There is no all-steps mode.- Four step types:
source,collector,transformer,destination. Source-stepeventshape is{ content, trigger? }, wherecontentis the walkerOS event{ name, data }andtriggeris optional{ type?, options? }. There is noenvfield. Destination and transformer steps take a plain walkerOS event{ name, data, consent? }. collectoris the enrichment step plus the collector chain. It takes a post-nextpartial event plus an optional state snapshot{ consent?, user?, globals?, timing? }, applies the collector's enrichment, then runscollector.next, and returns every event the destinations would receive (none when astopdrops it, several whenmanyforks it).transformer,collectoranddestinationsteps accept an optionalingest(a raw ingest without_meta). Supply it to test a request decoder standalone (a GA4 decoder readingctx.ingest.url: passingest: { url: "..." }), or to see the client IP and user agent a conversion-API destination sends (ingest: { ip, userAgent }). The result is scrubbed of credentials like the CLI output.- Sources are simulatable as a step, including the
@walkeros/source-demodemo source. configPathis a file path, an http(s) URL or inline JSON.- A flow saved in the walkerOS app runs locally through a config link: the
hosted walkerOS MCP server's
flow_manage(action: "config_url") returns a URL that lasts about ten minutes, andflow_simulate,flow_bundle,flow_pushandflow_loadtake it as their config input.
flow_load loads a flow from a local path, URL or inline JSON. In a returned
config, package names, platform, versions and IDs come back literally; every
other string value comes back wrapped, for example
"title": "<user_data>Home</user_data>". Remove the wrapper from each value
before writing the config back, or it becomes part of the value.
From a terminal, walkeros diagnostics (or --json) reports the CLI version,
the app URL and where it comes from, and whether the app answers its health
route:
cli 4.x.y
app https://app.walkeros.io (default)
health reachable, status ok, app 3f2a91c
Validate flow config
Validate every flow and every step in one run:
walkeros validate flow.json
What validate guarantees:
- Everything by default: schema, references (resolved per flow with the
runtime resolver; an unknown
$var, a missing store, an unresolvable$flowor a cycle is an error), routes, the rootcontract, the bundle preflight, cross-step example compatibility, the examples of@walkeros/transformer-validatesteps against their linked contract, and each step's settings against its package schema (an unknown key is an error where the package closes its schema, other findings are warnings;--offlineskips the fetch). - Narrowing is explicit:
--flow <name>narrows per-flow checks (file-level checks still run);--path <section.key>checks one entry in every flow that has it, or only in--flow. - Every skip is reported in
details.skippedand the text output, with a code. A step that imports a named export (stepimport, elsebundle.packages[pkg].imports[0]) is checked against that export's entry in the package'sexportSchemas; a single-export package usessettings. The bundler runs the step'simport, else the default export, never animportsentry, so setimporton the step to check and run the same export. A multi-export package version withoutexportSchemas, or one that does not list its exports, is a skip (NO_SETTINGS_SCHEMA). - The result states its scope: a
Scope:line anddetails.scope. --strictfails on warnings and skips (exit 2;validate()andflow_validatereturnvalid: false) and makes contract example disagreements errors (exit 1). It never changes what runs.- Stable codes: every error and warning has a
code; branch on it, never on the message. - Not checked, also under
--strict: an unknown key at the file root, on a flow or in itsconfigis dropped without a message (a misspelt"destinaitons"passes), and a mapping rule'ssettingsare not checked. A passing validate is a first filter; simulate before you bundle.
For full details on writing and testing with step examples, see using-step-examples.
$code: Prefix (Inline JavaScript)
Embed JavaScript functions in JSON configs:
{
"fn": "$code:(event) => event.data.price * 100",
"condition": "$code:(event) => event.data?.value > 100"
}
Important: The CLI bundler converts $code: strings to actual JavaScript
functions during build. This is essential for mappings in JSON configs.
For mapping patterns, see understanding-mapping.
Quick Reference
Bundle Command
walkeros bundle <config> [options]
Options:
-o, --output <path> Write to a file or directory (a server flow also to a .tar.gz/.tgz archive)
-f, --flow <name> Bundle one flow of a multi-flow config
--all Bundle all flows (needs -o)
--stats Show bundle statistics
--json JSON statistics (implies --stats); with -o the JSON replaces the bundle at that path
--no-cache Disable the package and build caches
--release <id> Stamp a config release id on event.source.release
--manifest [source] Build from a manifest (see "Manifest builds")
-v, --verbose Verbose output
-s, --silent Silent mode
Output:
- Web:
dist/walker.js(single self-contained IIFE). Every browser entry (this bundle, managed wraps, walker.js) calls the Tag Mode loadermoin()from@walkeros/web-corefirst; it stays idle unless the page was opened with?elbMoin. - Server:
dist/{flow.mjs, package.json, node_modules/}(always a directory; nft-traced) - Server archive:
flow.tar.gz/flow.tgz(the server bundle directory packed into a single gzip file)
Use -o ./dist/walker.js for web, -o ./dist/ for a server directory, or
-o ./flow.tar.gz for a server archive. Web single-file bundles do not support
archive output. Without -o the bundle is written to stdout, which for a server
flow is flow.mjs alone, without its node_modules/.
Also: --release <id> stamps a config release id on event.source.release (it
beats an authored collector.release; the default is a 12-hex content id of
config, resolved packages and CLI version, so the same input gives the same
release; env and secret values are not inputs, only their names);
--manifest [url|path] builds from a manifest (see "Manifest builds" below).
Push Command
walkeros push <config> [options]
Options:
-e, --event <json|file|url> Event to process (required; a mistyped file path is read as an event name)
-f, --flow <name> Flow to use
-p, --platform <web|server> Platform override
--simulate <step> Simulate a step (repeatable for destination.*). Format: source.NAME | destination.NAME | transformer.NAME | collector.NAME
--mock <step=value> Mock a step with a specific return value (repeatable); chain members via destination.NAME.before.ID or collector.next.ID
--ingest <json|file|url> Request context for transformer/collector/destination simulate
--consent <json|file|url> Starting collector consent for a simulation, any step
--command <name> Destination simulate runs this command (config|consent|user|run) instead of a push
--page-url <url> Page URL of a simulated web source (http/https)
--snapshot <source> JS file to eval before execution (sets global state)
-o, --output <path> Write the result to a file
--json JSON on stdout (incl. simulations); logs to stderr
-v, --verbose / -s, --silent Verbose or silent output
On a real push, a destination that writes to the console still prints there:
with @walkeros/destination-demo, push --json puts the demo's lines on stdout
before the JSON, so the output does not parse with jq.
Push takes a flow config. A file walkeros bundle wrote fails as push input
(see the CLI docs, "Input types").
Validate Command
walkeros validate <input> [options]
Options:
-t, --type <type> Validation type (default: flow). Also: event, mapping, contract
--path <path> Check one entry in every flow that has it (e.g. destinations.snowplow, stores.cache)
-f, --flow <name> Narrow per-flow checks to one flow
--strict Fail on warnings and skipped checks
--offline Do not fetch package schemas (settings not checked)
--json JSON output
-o, --output <path> Write the result to a file
Exit codes:
0 = Valid (skips alone do not change it)
1 = Errors found (contract violations count as errors under --strict)
2 = No errors, but warnings or skipped checks found (with --strict only)
3 = Validation could not run (missing file, invalid JSON, option that does not apply)
Running a built flow (runneros, not the CLI)
The CLI has no run command. The runtime is a separate package,
@walkeros/runner, with the binary runneros:
runneros start [artifact] [options]
Options:
--flow-id <id> App flow ID (enables heartbeat and secrets)
--project <id> Project ID (default WALKEROS_PROJECT_ID; required with --flow-id)
-p, --port <number> Port (default: 8080)
--env-file <path> Load a dotenv file first (existing env wins; a group- or other-readable file is refused)
--json JSON output
-v, --verbose Verbose output
-s, --silent Silent mode
[artifact] (or BUNDLE, or flow.mjs in the working directory) is a
.mjs/.js/.cjs entry, a .tar.gz/.tgz archive holding flow.mjs, or an
http(s) URL to either. The runtime does not bundle: it has no bundler
installed; a local flow config is refused and one fed by URL or stdin fails at
import. Never generate a container or command that hands flow.json to the
runtime; bundle first. The full reference is "Deploying a server flow" below.
Setup Command
# Provision external resources for one component (explicit only, never
# triggered by push, real or simulated, a deployment, or the runtime)
walkeros setup <kind>.<name> [-c ./flow.json] [-f <flow>] [--json] [--verbose] [--silent]
The target uses the same <kind>.<name> syntax as walkeros push --simulate
(source, destination, or store). Package resolution follows the flow's
pins exactly like bundle: the version from config.bundle.packages (or inline
@scope/pkg@x.y.z, bundle pin wins) is downloaded through the shared pacote
pipeline and cache, then imported from the extracted tree. Setup works via npx
without a local install; path: entries in config.bundle.packages support
local development with a built package directory. Today it cannot load the AWS
(SQS) or GCP (Pub/Sub) source, nor the GCS or Sheets store: it exits 1 and
creates nothing (see the CLI docs, "Setup command").
Bundler Gotchas
- Circular copies: Never include the output directory itself (e.g.,
include: ["./dist"]when output isdist/). The CLI detects this and errors. includeis server-only: rootinclude(default./sharedwhen that folder exists) is copied next to server bundles only. A web build copies nothing and logsinclude is ignored for web builds: a browser bundle cannot read local folders.A server build served from the build cache copiesincludetoo. A manifest (hosted) build refusesincludeonly on a server flow; on a web flow it is accepted and ignored.- Caches and temp dirs: caches live in
$TMPDIR/cache/(os.tmpdir(); on LinuxTMPDIRis often unset, read$TMPDIR/...in output as/tmp/...). Package entries are written to a temp sibling and renamed in once complete; a failed write warnsPackage cache write failed for <name>@<version>: <message>. The build continues without caching it.Entries from an older CLI miss once.walkeros cache clearalso removes interrupted writes. Build cache keys cover a local package's (path) built contents and the CLI version, so a rebuild needs nocache clear;cache clear --buildsremoves both build caches. Each run works in$TMPDIR/walkeros/<kind>/<6 chars>(push,build,bundle,wrap,archive,setup). - Runtime paths:
runnerossets CWD to the bundle directory. File paths insettingsresolve relative to the bundle, not the project root. - Component names: Source, transformer, destination, and store names must be
valid JavaScript identifiers (camelCase). Validate refuses a hyphenated name
such as
gtag-wrapper(INVALID_COMPONENT_NAME); usegtagWrapper. - Range conflicts: When two transitive consumers declare incompatible ranges
for the same dep (e.g.,
arrify@^3.0.0vsarrify@^2.0.0), the bundler resolves the chosen range to a concrete version and nests non-satisfying specs under their consumer. If a post-install warning surfaces declared-vs-installed mismatches, pin the dep inconfig.bundle.overrides. SetBUNDLER_STRICT_RANGES=0to bypass strict range validation when the npm registry is unreachable.
Server bundles use nft tracing
Server flows are bundled with @vercel/nft.
The CLI:
- Pacote installs every package declared in
flow.<name>.config.bundle.packagesinto a per-build install root. Users do not runnpm installfor step packages; only@walkeros/clilives in theirpackage.json. - esbuild stage 1 externalizes all step packages.
- esbuild stage 2 emits a small ESM
flow.mjsthat imports from those externalized packages. - nft traces
flow.mjs, finds every file actually reachable at runtime (including__dirname-loaded.protofiles and other assets), and copies only those files intodist/node_modules/.
There is no walkerOS.bundle.external annotation. nft figures it out.
After tracing, every declared bundle.packages entry the bundle actually
imports (bare imports of the built bundle, incl. imports) must be in the
trace, else the build throws nft-trace: resolved packages missing from trace.
A declared package that is neither imported nor traced (e.g. a type-only
dependency) only warns:
Package <name> is declared in bundle.packages but nothing imports it; it is not in the bundle. Remove it from bundle.packages if unused.
Bundle directory (the server flow's unpacked artifact):
dist/
├── flow.mjs # ESM entry, expects to be at /app/flow/flow.mjs in prod
├── package.json # informational sidecar (not used by the runner)
└── node_modules/ # only the files nft traced
The same directory can be packed into a .tar.gz/.tgz archive (see the Bundle
Command section), and runneros start accepts either form. Web flows are a
single dist/walker.js, served by any static host.
Canonical Dockerfile
ARG WALKEROS_VERSION
FROM node:24-alpine AS builder
ARG WALKEROS_VERSION
WORKDIR /build
RUN npm init -y && npm install --save-dev @walkeros/cli@${WALKEROS_VERSION}
COPY flow.json ./
RUN npx walkeros bundle flow.json -o dist/
FROM walkeros/flow:${WALKEROS_VERSION}
COPY --from=builder /build/dist/ /app/flow/
Build with docker build --build-arg WALKEROS_VERSION=<version> ..
Notes:
- The build stage only needs
@walkeros/cli. flow.json drives every step package install; pacote handles it. COPY --from=builder /build/dist/ /app/flow/copies the whole directory (flow.mjs + package.json + node_modules/).- The image sets
BUNDLE=/app/flow/flow.mjs,PORT=8080andCMD ["runneros", "start"]. No command and noBUNDLEneeded. - One version pins both:
walkeros/flow:<v>contains@walkeros/runner@<v>, matching@walkeros/cli@<v>. Images older than@walkeros/runnerhave norunneros; check withdocker run --rm walkeros/flow:<v> runneros --version. Do not assumelatestor a major tag carries it.
Escape hatch: traceInclude
If nft cannot statically reach a runtime asset (rare: require() of a path
constructed from a runtime variable), declare it explicitly under
flow.<name>.config.bundle.traceInclude. Paths and globs both work; both
resolve against the install root, not the project directory:
"flows": {
"default": {
"config": {
"platform": "server",
"bundle": {
"packages": { "@walkeros/server-destination-gcp": {} },
"traceInclude": [
"node_modules/some-pkg/data/*.json",
"node_modules/another-pkg/lib/runtime-loaded.js"
]
}
}
}
}
Cache (CI)
The bundler caches pacote downloads under process.env.NPM_CACHE_DIR (default
$TMPDIR/cache/npm). On CI, persist that path with actions/cache:
- uses: actions/cache@v4
with:
path: .walkeros-cache/npm
key: walkeros-${{ hashFiles('**/flow.json') }}
- run: npx walkeros bundle flow.json -o dist/
env:
NPM_CACHE_DIR: ${{ github.workspace }}/.walkeros-cache/npm
CI smoke check:
cd dist && node -e "import('./flow.mjs').then(()=>console.log('ok'))" plus
du -sh node_modules (a server flow with the BigQuery destination holds about
20 MB in about 1,300 files; use .dockerignore).
Deploying a server flow
Build, run, orchestrate
| Stage | Has | Must not have |
|---|---|---|
Build (walkeros bundle) | Registry access, esbuild, nft, a Node toolchain | Production secrets or traffic |
Run (runneros start) | The artifact, its runtime secrets, the port | A package manager, registry egress, a bundler |
| Orchestrate (you, CI, the walkerOS app) | The decision of what runs where | The ability to execute the artifact |
The runtime cannot bundle because no bundler is installed in it, not because a flag forbids it. Generate two steps (bundle, then start), never one.
Artifact shape
- Server: a DIRECTORY,
dist/flow.mjs+dist/package.json+dist/node_modules/. EveryCOPYand-vcarries the whole directory (COPY dist/ /app/flow/,-v "$PWD/dist:/app/flow:ro"), neverflow.mjsalone. Or one file:walkeros bundle flow.json -o flow.tar.gz. - Web: a single
dist/walker.jsfor any static host. The runtime has no static-file mode; to serve files from a server flow useinclude+@walkeros/server-store-fs+@walkeros/server-transformer-file.
Runtime environment
| Variable | Meaning |
|---|---|
BUNDLE | Artifact path or URL (image default /app/flow/flow.mjs) |
PORT | Port (default 8080) |
WALKEROS_FLOW_ID, WALKEROS_PROJECT_ID | App flow and project; with a runner token they enable secrets and the heartbeat |
WALKEROS_DEPLOYMENT_ID | Deployment ID sent with every heartbeat; the app accepts a heartbeat only with the deployment the runner token belongs to, else every beat logs Heartbeat rejected (404) at WARN |
WALKEROS_FLOW_RELEASE | Release the loaded flow must carry, or the boot fails |
WALKEROS_FLOW_NAME | Flow name it must carry; checked only with WALKEROS_FLOW_RELEASE |
WALKEROS_DEPLOY_TOKEN / WALKEROS_TOKEN | Runner token (wos_run_...), deploy token first; a person's automation token is refused. Env only, never a config file |
WALKEROS_APP_URL | App base URL (default https://app.walkeros.io) |
WALKEROS_HEARTBEAT_INTERVAL | Seconds (default 60, minimum 10) |
WALKEROS_CACHE_DIR | Error cache dir (then CACHE_DIR, then XDG ~/.cache/walkeros) |
WALKEROS_OBSERVE_LEVEL | off, standard or trace |
Secrets are fetched ONCE at boot when connected (401/403 is fatal). There is no
config polling and no hot-swap: a new flow version is a rebuild and a redeploy.
/health is always 200; /ready is 200 once the collector is constructed and,
with WALKEROS_FLOW_RELEASE set, the loaded flow matches it; 503 otherwise. A
mismatch exits with
Artifact mismatch: expected <name>@<release>, loaded <name>@<release>.
Constraints
- Filesystem: archive, URL and stdin inputs write to
/app/flow/, so a read-only root filesystem works only with a local.mjsartifact in place. PointWALKEROS_CACHE_DIRat a writable path (the image's/app/cache). - Architecture: the image is
linux/amd64only. - Default store: a
statestep withoutstoreuses the in-process__cachestore, correct only on ONE instance. Do not scale such a flow out without a shared store.
Full page: website/docs/apps/runtime.mdx.
Build-time values and package specs
config.bundle.envdeclares the values web$env.NAMEreferences resolve to at bundle time (literal strings only; a reference inside a value is refused). A local build layers it over the shell env; a hosted or manifest build uses it alone.$flowsiblings resolve against the ENTRY flow'sbundle.env. Values end up in a public bundle: never secrets. Server flows ignore it; their$envis read at runtime.- Package specs must be registry version, range or tag, for direct pins,
overridesvalues and transitive dependencies. Git,file:,link:and tarball URLs fail withUNSUPPORTED_PACKAGE_SPEC. Use a localpathfor development.
Manifest builds
walkeros bundle --manifest <url|path> (a bare flag reads BUILD_MANIFEST_URL)
builds from a JSON manifest: version: 1, toolchain (exact CLI version, else
TOOLCHAIN_MISMATCH), flowConfig, flowName?, buildEnv?, artifacts[]
(outputName, putUrl, contentType?, headers?, plus target for a bundle
or target: "wrap" + platform + skeleton for a wrap), and resultPutUrl.
The CLI PUTs each artifact, then a result
{ ok, toolchain, artifacts: [{ target, outputName, bytes, sha256 }], error?: { code, message, outputName? } }.
Unknown top-level keys are ignored; artifact entries are strict; local
filesystem inputs are refused with LOCAL_PATH_NOT_ALLOWED. Only --json,
-v, -s combine with it. Schemas: BuildManifestSchema, BuildResultSchema,
BUILD_ERROR_CODES from @walkeros/cli.
Troubleshooting
Bundle Fails
- Check the config:
walkeros validate flow.json(add--flow <name>to narrow to one flow) - Check package names: Ensure packages exist on npm
- Clear cache:
walkeros cache clear
Events Not Processing
- Validate event:
walkeros validate event.json --type event - Check mapping: Event must match entity/action in mapping
- Use simulate first:
walkeros push flow.json -e event.json --simulate destination.demo -v
Destination Not Found in Simulation
Destination "NAME" not found in collector. Available: ... means the flow has
no destination with that key; pick one from the list.
Destination Silent (0 Events Received)
Read the line under mapping::
pending: waits for consent (require): the destination hasrequire: ["consent"]and never started. Pass starting consent:--consent '{"functional":true}'(MCP:state.consent). Do not removerequireto test.skipped: consent (requires marketing; granted functional): itsconsentsetting denied the event. Grant the key in--consentor in the event'sconsent.- Check mapping: The event name must match a mapping rule (entity/action keys). Unmapped events pass through unmodified.
- Check policy: Policy shapes the payload, not the rule choice at a destination; verify policy isn't redacting fields the payload needs.
Web Simulation Transport
Web flows run in JSDOM. fetch, navigator.sendBeacon and XMLHttpRequest are
polyfilled as tracked no-ops, so the flow's own requests are recorded, not sent;
scripts, iframes and stylesheets the page loads are still fetched. Captured
network calls are included in PushResult.networkCalls when present.
Local Packages Not Found
Use absolute or relative paths in flow.<name>.config.bundle.packages:
{
"config": {
"platform": "web",
"bundle": {
"packages": {
"my-destination": {
"path": "./local/my-destination"
}
}
}
}
}
An App Answers 426
The walkerOS app answers 426 Upgrade Required to a CLI below its minimum
version. walkeros api prints the answer body, which carries the upgrade
message and minVersion, and exits 1. Upgrade the CLI and retry:
npm install -g @walkeros/cli@latest
See Upgrading for the answer body and the upgrade checklist.
Where CLI Lives
| Location | Purpose |
|---|---|
packages/cli/ | CLI source code |
packages/cli/src/commands/ | Command implementations |
packages/cli/examples/ | Example flow configs |
website/docs/apps/cli.mdx | Full CLI documentation |
Networking
Outbound requests to the walkerOS app (WALKEROS_APP_URL, default
https://app.walkeros.io) carry X-WalkerOS-Client: cli and
X-WalkerOS-Client-Version headers so the host can attribute usage and enforce
its minimum version. No PII.
Telemetry
The CLI supports anonymous usage telemetry (installation id, command name, outcome, duration, environment) to help improve the tool. Telemetry is off by default and requires explicit opt-in. No persistent identifier is written before consent.
- Opt in:
walkeros telemetry enable. - Opt out:
walkeros telemetry disable. - Current state:
walkeros telemetry status. - Forced off (override): set
DO_NOT_TRACK=1orWALKEROS_TELEMETRY_DISABLED=1. - Debug: set
WALKEROS_TELEMETRY_DEBUG=1to print the payload to stderr (requires opt-in; no network traffic). - Docs: see apps/telemetry for the full event list, privacy details, and legal basis.
- Contract (source of truth):
packages/cli/src/telemetry/flow.json.
Related Skills
- understanding-mapping - Mapping configuration
- understanding-flow - Data flow architecture
- create-destination - Creating destinations
- create-source - Creating sources
- debugging - Troubleshooting event flow
Detailed References:
- commands-reference.md - bundle, push, runneros, api, auth, validate and cache with their options
- flow-configuration.md - Complete Flow.Json reference
- server-deployment.md - Server flow deployment guide
Using the walkerOS Logger
Overview
The logger is walkerOS's standard logging system, available in all sources and
destinations via env.logger or logger parameter. It provides scoped,
level-aware logging that replaces console.log.
Core principle: Don't log what the collector already logs, and never log a
failure yourself. A step reports a failure by throwing a StepError; the
collector writes the one ERROR line per cause. Log only meaningful operations
like external API calls and transformations, at debug.
Logger Access
In Sources
export const sourceFetch: Source.Init<Types> = async (context) => {
const { env } = context; // env.logger is available here
// Logger is scoped automatically by collector: [type:sourceId]
env.logger.info('Server listening on port 3000');
// ... build and return the source instance
};
In Destinations
export const destinationDataManager: DestinationInterface = {
async init({ config, env, logger }) {
// logger parameter is scoped automatically: [datamanager]
logger.debug('Auth client created');
},
async push(event, { config, data, env, logger }) {
// logger parameter is scoped: [datamanager]
logger.debug('API response', { status: 200 });
},
};
Note: You don't need to create or configure the logger; it's provided automatically with proper scoping.
Logger Methods
interface Logger.Instance {
error(message: string | Error, context?: unknown | Error): void;
warn(message: string | Error, context?: unknown | Error): void;
info(message: string | Error, context?: unknown | Error): void;
debug(message: string | Error, context?: unknown | Error): void;
throw(message: string | Error, context?: unknown): never;
json(data: unknown): void;
scope(name: string): Logger.Instance;
}
Log Levels
- ERROR (0): the user's intent is not met (an event was not delivered, a setting has no effect)
- WARN (1): walkerOS acted and the intent is still met
- INFO (2): lifecycle and self-healing (server startup)
- DEBUG (3): the per-event trace and deliberate decisions (API calls, transformations, a skipped event)
Default: ERROR only (must configure to see WARN/INFO/DEBUG)
Context Parameter
All methods accept optional structured context:
logger.debug('Sending to API', {
endpoint: '/events',
method: 'POST',
eventCount: 5,
});
// Output: DEBUG [datamanager] Sending to API { endpoint: '/events', method: 'POST', eventCount: 5 }
When to Log (and When NOT to)
❌ DON'T Log These (Collector Handles)
- Init status: "Initializing...", "Init started", "Init complete"
- Push status: "Processing event...", "Event received"
- Generic status: "Settings validated", "Config loaded"
- Duplicate scoping: Don't add source/dest name (already in scope)
Why: Collector can log these automatically since it calls init/push and has scoped logger.
✅ DO Log These (Meaningful Operations)
- External API calls: Before/after with request/response details
- Auth operations: Token refresh, client creation/failures
- Transformations: Complex mappings or data processing
- Notes that are not failures: a deliberate decision, an advisory about
configuration (
warn: walkerOS acted, the intent is still met) - Never failures: a missing setting, a failed request or an unsendable event
is a thrown
StepError, not a log line
Usage Patterns
Pattern 1: Validation Errors (Throw a StepError)
import { StepError } from '@walkeros/core';
// ✅ GOOD - A missing setting is a failure: throw, never return or log
async init({ config }) {
const { apiKey, projectId } = config.settings || {};
if (!apiKey)
throw new StepError('Config settings apiKey missing', { code: 'config' });
if (!projectId)
throw new StepError('Config settings projectId missing', {
code: 'config',
});
}
Why a thrown StepError:
- The collector turns the throw into a failed push (or a failed start), counts it, and writes one ERROR line per cause and minute with the code
- A step that also logged the failure would write a second line for the same cause
codeandretryabletell a server source whether to ask the producer to resend (config,missing-inputand a 4xx are not retryable)logger.throwonly throws. It writes no log line, so it is not a way to report a failure
Never return early from init or push instead of throwing: a step that
returns normally counts as started, and a push that returns normally counts as
delivered. return false from init fails the push with the code not-ready,
which does not name the missing setting.
Pattern 2: External API Calls
import { StepError } from '@walkeros/core';
// ✅ GOOD - Trace the call at debug, throw on a failure
async push(event, { config, logger }) {
const endpoint = 'https://api.vendor.com/events';
logger.debug('Calling API', { endpoint, method: 'POST', eventId: event.id });
const response = await fetch(endpoint, {
method: 'POST',
body: JSON.stringify(event),
});
logger.debug('API response', { status: response.status, ok: response.ok });
if (!response.ok)
// The status sets retryable (429 and 5xx retry, other 4xx do not). The
// response body goes on `body`, never into the message, and never into a
// log line.
throw new StepError(`HTTP ${response.status}`, {
code: 'transport',
status: response.status,
body: await response.text(),
});
}
Prefer sendWeb / sendServer: they already throw a StepError with the
status and body.
Pattern 3: Auth Operations
import { StepError } from '@walkeros/core';
// ✅ GOOD - Trace the client creation, throw on failure
async init({ config, logger }) {
try {
const authClient = await createAuthClient(config.settings);
logger.debug('Auth client created');
return {
env: { authClient },
};
} catch (error) {
throw new StepError('Authentication failed', {
code: 'config',
cause: error,
});
}
}
The SDK's own message stays on cause, so a vendor error that carries personal
data never reaches a log line.
Pattern 4: Server Startup (Sources)
// ✅ GOOD - Log server listening (high-level info); routeLines lists one line
// per method and path the source serves
if (settings.port !== undefined) {
server = app.listen(settings.port, () => {
env.logger.info(
`Express source listening on port ${settings.port}\n` + routeLines,
);
});
}
Anti-Patterns (What NOT to Do)
❌ BAD: Verbose Init Logging
async init({ logger }) {
logger.debug('Data Manager init started'); // Redundant
logger.info('Data Manager initializing...'); // Redundant
logger.debug('Settings validated'); // Redundant
const authClient = await createAuthClient();
logger.debug('Auth client created'); // OK
logger.info('Data Manager ready'); // Redundant
}
Problem: Collector knows when init is called. Only log meaningful operations (auth client creation).
❌ BAD: Redundant Push Logging
async push(event, { logger }) {
logger.debug('Processing event', {
// Redundant
name: event.name,
id: event.id,
});
// Do work...
logger.info('Event processed'); // Redundant
}
Problem: Collector knows when push is called and can log automatically.
❌ BAD: Using console.log
// ❌ NEVER use console.log in sources/destinations
console.log('Processing event:', event.name);
// ✅ Use logger instead
logger.debug('API call', { endpoint });
Migration Checklist
When updating a source/destination to use the logger:
- Remove all
console.log,console.warn,console.errorstatements - Remove verbose init/push status logging (let collector handle)
- Throw a
StepErrorwith acodefor every failure (apiKey missing, failed request, unsendable event); never log a failure and return - Add
logger.debugbefore external API calls (with endpoint, method) - Add
logger.debugafter external API calls (with response status) - Add
logger.debugfor auth operations (client creation, token refresh) - Use structured context (objects) instead of string concatenation
- Verify tests still pass with logger mocked
Testing with Logger
Use createMockLogger from @walkeros/core to check debug traces. To prove
that a failure is reported, start a real collector (startFlow), mock only the
network or the vendor global, and read the push result; a mocked collector
cannot see loss:
import { startFlow } from '@walkeros/collector';
test('a missing apiKey fails the push', async () => {
const { elb } = await startFlow({
destinations: { vendor: { code: destination, config: { settings: {} } } },
});
const result = await elb({ name: 'page view' });
expect(result.ok).toBe(false);
expect(result.failed?.vendor.reason?.code).toBe('config');
});
Log Level Configuration
Default log level is ERROR. To see INFO/DEBUG logs:
import { startFlow } from '@walkeros/collector';
const { elb } = await startFlow({
logger: {
level: 'DEBUG', // Show all logs
},
destinations: {
/* ... */
},
});
Levels:
'ERROR': Only errors (default)'WARN': Errors + warnings'INFO': Errors + warnings + info'DEBUG': Everything
The levels follow the contract under Log Levels. A skipped event
writes a DEBUG line; a consent wait writes no line, and the push result lists
the destination under queued with the reason consent. Neither is a failure.
Format: on a server flow, "collector": { "logger": { "format": "json" } }
prints one JSON object per line (severity, message, time, scope,
context), which platforms such as Cloud Run read as structured logs. A web
flow rejects format. The flow's own handler and the runtime's handler both see
every line, so forwarding to a monitoring service needs no parsing.
Related Skills
- walkeros-understanding-destinations - Destination interface
- walkeros-understanding-sources - Source interface
Key Files:
- packages/core/src/logger.ts - Logger implementation
- packages/core/src/types/logger.ts - Logger types
- packages/core/src/mockLogger.ts - Testing utilities
Choosing a level on public endpoints
A server source endpoint is exposed to the open internet. Level choice decides whether ambient scanner noise turns into pages.
| Situation | Level | Also |
|---|---|---|
| Unparseable body, oversized payload | debug | count on collector.status.sources.<id>.rejected |
| Unknown path | no line | answered 404 |
| Event the collector rejects as invalid | error with the reason, one line per source and minute, no stack | counted as rejected; answered 400 only when the source waits for delivery |
Settled ok: false arriving after the HTTP ack | debug | the collector's failure line reports the cause at error; the ack already went out |
| Handler fault, rejected push promise | error | answered 500 |
Rules that follow from the table:
- Never
errorfor anything a stranger can trigger, apart from the collector's invalid-event line, which the source does not repeat. On Cloud Run an error-level stack trace on stderr opens an Error Reporting issue, so scanner traffic becomes an alert; the invalid-event line carries no stack. - Do not duplicate the collector's error logs. The collector already logs thrown pipeline failures at error. A source that logs them again doubles every real fault.
- A rejected push and a settled
ok: falseare different things. If a helper collapses both into one return shape, log the thrown case inside that helper at error, because the call site can no longer tell them apart.
Reference pattern: packages/server/sources/express/src/index.ts, the
error-boundary middleware for body-parser failures and the 400/500 split in the
synchronous POST branch.
Best Practice Examples:
- packages/server/destinations/datamanager/src/push.ts - API logging patterns
- packages/server/sources/express/src/index.ts - Server startup logging
Using Step Examples
Overview
Step examples are structured { in, out } pairs (with optional mapping for
destinations) that define the expected input/output behavior of each step in a
flow. They serve as:
- Test fixtures for automated
it.eachtesting - Simulation input for
walkeros push --simulate(pass an example'sinas--event; the CLI does not compare the result againstout) - MCP context for AI-assisted development
- Documentation showing real-world usage
Every source, transformer, and destination can ship step examples alongside its code.
The Three Type Zones
Each step in a flow sits at a boundary between arbitrary external formats and the walkerOS event model:
Source Transformer Destination
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ in: arbitrary │ │ in: walkerOS │ │ in: walkerOS │
│ (HTTP req, DOM) │ │ Event │ │ Event │
│ │ │ │ │ │ │ │ │
│ ▼ │ │ ▼ │ │ ▼ │
│ out: walkerOS │ │ out: walkerOS │ │ out: arbitrary │
│ Event │ │ Event | false │ │ (gtag, API) │
└─────────────────┘ └─────────────────┘ └─────────────────┘
arbitrary ──────────── walkerOS.Event ──────────── arbitrary
- Source converts platform-specific input into a walkerOS event
- Transformer receives and returns walkerOS events (or
falseto filter) - Destination converts a walkerOS event into vendor-specific output
The in/out/trigger Format
Every step example is an object with in (input), out (array of observable
effects), and optional trigger (how to invoke):
type StepEffect = readonly [callable: string, ...args: unknown[]];
type StepOut = readonly StepEffect[];
interface StepExample {
in?: unknown; // Platform-specific input
out?: StepOut; // Array of effect tuples - see shape rules below
trigger?: { type?: string; options?: unknown }; // How to invoke
mapping?: unknown; // Destination mapping rule
command?: 'config' | 'consent' | 'user' | 'run'; // Route to walker command
// also: title, description, public (Metadata), outcome, respond (failures)
}
out is always an array of effect tuples, even for a single effect. Each
tuple is [callable, ...args]. The first element names the call, the remaining
elements are its arguments. runExamples compares the tuples exactly with the
calls it records, so the callable is the recorded path.
Callable conventions
| Component type | Callable | Notes |
|---|---|---|
| Destination with a mock env | 'window.gtag', 'window.fbq', 'window.dataLayer.push', 'window.ttq.track' | The package's simulation path without call: ('call:window.gtag' records as 'window.gtag'). Dotted paths render literally. |
| Destination (HTTP) | 'fetch', 'sendServer' | The request call the destination makes. |
| Source | 'elb' | The walker public push API. |
| Transformer | 'return' | Reserved keyword. Renders as return <value> (no parens). |
'return' is reserved. Don't use it for anything else.
Empty out: [] means the step produced no observable effect (filtered input,
transformer passthrough, validator rejection). Reserved only for cases where the
destination/source deliberately emits nothing.
Source Step Example - Server (Express)
in is an HTTP request shape, out is a tuple of the elb() call the source
makes:
export const checkoutPost: Flow.StepExample = {
trigger: { type: 'POST' },
in: {
method: 'POST',
path: '/collect',
body: { name: 'order complete', data: { id: 'ORD-123', value: 149.97 } },
},
out: [
['elb', { name: 'order complete', data: { id: 'ORD-123', value: 149.97 } }],
],
};
Source Step Example - Commands (CMP, session)
A source that issues walker commands records them in its own call shape: the
first argument verbatim, callable 'elb'. Commands come first, in call order,
then the events. A CMP source:
out: [['elb', 'walker consent', { functional: true, marketing: true }]],
The session source (env.command('user' | 'session', ...), then its event):
out: [
['elb', 'user', { session: 's3ss10n-id', device: 'd3v1c3-id' }],
['elb', 'session', { id: 's3ss10n-id', isStart: true /* ... */ }],
['elb', { name: 'session start', data: { id: 's3ss10n-id' /* ... */ } }],
],
A source simulation (walkeros push --simulate source.X, MCP flow_simulate)
returns exactly these calls: call elb("walker consent",{...}). The collector's
own commands (starting consent from --consent, globals, custom, shutdown) and
the wiring commands on, hook, destination are never part of a source's
out. A consent-gated source needs --consent (MCP state.consent) to emit
anything.
Source Step Example - Browser
in is an HTML string, out is a tuple of the elb() call:
export const clickEvent: Flow.StepExample = {
trigger: { type: 'click', options: 'button' },
in: '<button data-elb="cta" data-elb-cta="label:Sign Up" data-elbaction="click:click">Sign Up</button>',
out: [
[
'elb',
{
name: 'cta click',
data: { label: 'Sign Up' },
trigger: 'click',
entity: 'cta',
action: 'click',
// plus context, globals, nested and source: an out lists every field
// the step emits
},
],
],
};
Transformer Step Example
in is a walkerOS event, out is a ['return', value] tuple with what the
transformer returns ({ event }, or false to drop). An empty array means the
transformer passed the event through unchanged:
export const step = {
orderPasses: {
in: { name: 'order complete', data: { id: 'ORD-123' } },
out: [
[
'return',
{ event: { name: 'order complete', data: { id: 'ORD-123' } } },
],
],
},
debugFiltered: {
in: { name: 'debug test', data: { message: 'noise' } },
out: [['return', false]], // Transformer rejects
},
passthrough: {
in: { name: 'page view' },
out: [], // No modification
},
};
These outs are what the transformer itself returns, the shape a test that calls
push directly compares (see Transformer Tests below). runExamples instead
records what reaches the next step, the event as the collector completed it. A
dropped event is out: [] there, and an event passed through unchanged is
['return', { event }] with every collector field.
Destination Step Example
in is a walkerOS event, mapping is the mapping rule that transforms it,
out is an array of call tuples - one per observable effect the destination
produces. Multi-call events (e.g., GA4 + Ads + GTM for a single walker event)
flatten into a single array in execution order:
import type { Flow } from '@walkeros/core';
import { getEvent } from '@walkeros/core';
// getEvent pins the timestamp (1700000000000) and derives the id from the event
// name; pass `id` when the expected output must name a readable one.
export const purchase: Flow.StepExample = {
in: getEvent('order complete', { id: '1700000000-gr0up-1' }),
mapping: {
name: 'Purchase',
data: { map: { value: 'data.value', currency: { value: 'EUR' } } },
},
out: [
[
'window.fbq',
'track',
'Purchase',
{ value: 462, currency: 'EUR' },
{ eventID: '1700000000-gr0up-1' },
],
],
};
Multi-tool example (one walker event produces GA4, Ads, GTM in order):
out: [
['window.gtag', 'event', 'purchase', { transaction_id: 'o1', value: 462 }],
['window.gtag', 'event', 'conversion', { send_to: 'AW-123', value: 462 }],
['window.dataLayer.push', { event: 'purchase', ecommerce: { ... } }],
],
Each export is a self-contained Flow.StepExample - no intermediate variables,
no all aggregation. The mapping field ties the mapping rule to the example
so tests can register it dynamically:
{ [event.entity]: { [event.action]: example.mapping } }.
Consumers iterate all examples via Object.entries(examples.step) -
export * as step exposes every named export directly.
Command Step Example
Some destinations need to respond to walker commands (walker consent,
walker user, walker run), not events. Set the command field to route in
through elb('walker <command>', in) instead of pushing it as an event:
export const consentGranted: Flow.StepExample = {
command: 'consent',
in: { marketing: true, analytics: true },
out: [
[
'window.gtag',
'consent',
'update',
{
ad_storage: 'granted',
ad_user_data: 'granted',
ad_personalization: 'granted',
analytics_storage: 'granted',
},
],
],
};
Supported commands: config, consent, user, run.
-
When
commandis absent (default),inis pushed as a regular event viaelb(in). -
When
commandis set,mappingis not applied - commands don't flow through event mapping. -
The
outholds the calls the destination makes for the command, as tuples like any otherout. The gtag example above runs withconfig.consentset; without it a['window.gtag', 'consent', 'default', {...}]call comes first, as in the gtag package'sconsentModeV2example. -
Simulate a command example with
--command <name>(MCP:command): the destination simulation runscollector.command(name, in)instead of a push and records the calls it makes. Add--consentwhen the destination needs starting consent; it applies first, then the command:walkeros push flow.json -e '{"functional":true,"marketing":true}' \ --simulate destination.ga4 --command consent
Failure Step Example (outcome, respond)
A step's failure modes are examples too. A failure fixture is public: false
and declares what the push result reports through outcome; respond states
what the vendor's boundary answers when the failure comes from a response:
export const rejectedByVendor: Flow.StepExample = {
public: false,
description: 'The vendor answers 200 and reports the event as failed.',
in: getEvent('page view'),
respond: { status: 200, body: { events: [{ status: 'failed' }] } },
outcome: { status: 'failed', code: 'transport', retryable: false },
};
export const missingInput: Flow.StepExample = {
public: false,
description:
'An event with nothing to send fails instead of counting as delivered.',
in: getEvent('page view'),
out: [],
outcome: { status: 'failed', code: 'missing-input' },
};
outcomeis{ status: 'failed' | 'queued', code, retryable? }. Absent means the step must not fail. The codes are the collector's failure codes (config,missing-input,transport,timeout,vendor-not-loaded, ...).- Every example with an
outcomeneeds adescription: the package's docs page lists it under "When it fails" (title,description,statusandcode, never itsin), public or not, and the docs build fails on one without adescription. respondis{ status?, body?, error?: 'network' | 'timeout' }.- A step with no vendor call for an example has
out: []. Asilentrule is an example withmapping: { silent: true }andout: [], and a deliberate skip (ignore: true) isout: []with nooutcome. runExamples(kind, step, examples, options?)andconformance(kind, step, options)from@walkeros/collector/devrun the examples on a real collector and compare the push result withoutcome. Conformance profiles: destinationshttp,sdk-env,web-sdk,vendor-global,socket;transformerandsourcefor the other kinds. The browser profiles (web-sdk,vendor-global) needpagefrom@walkeros/web-core/devin the options, and a web source may pass it;runExamplestakes the same option.
Public examples, claims and literal outs
A destination has exactly one public example per standard event it maps: the
most complete, fully implemented case. Its in starts from getEvent(name)
(page view, product view, product add, order complete). Edge cases,
other tools and failures are public: false; they still run as tests.
claims.runExamples('destination', step, examples, { claims })takes the standard events the destination maps. It fails when a claimed event has no public example, when that example does not start fromgetEvent(name), or when a public event example is not claimed. Init examples (config asin) are not events and are ignored byclaims.- Mappings run through the real destination config. The runner looks the
example up by event name and applies
example.mapping ?? config.mapping. A public example whose setup needs other settings runs in its ownrunExamplescall with thosesettings. - Every example runs as published.
runExamplesandconformancerun every function in an example (inin,mapping,trigger.options,respond.body) the waywalkerOS.jsonpublishes it: as source text, revived outside the module it was written in. There is no option to turn this off. A condition that calls an imported helper passes as written and fails here (example <name>: published code fails: ... is not defined), which is how it would fail in a user's flow: a published function may use only its parameters and standard globals. The text revived is the one the test run compiled, not the one in the builtdist/walkerOS.json, so a change only the package build makes is not covered. - An
outis a literal. Write it from the vendor's documented request (or a captured one), cite the source in the example, and only then run the code. Never compute anoutwith production code (a helper fromsrc/, the destination's own encoder): the example would then agree with the code by construction. A mismatch between research and code is a finding; fix the product or the example and say why. - No personal data or credentials in an example: placeholders
(
G-XXXXXXXXXX),example.comaddresses and RFC 5737 IPs (192.0.2.1).
import type { StandardEventName } from '@walkeros/core';
import { runExamples } from '@walkeros/collector/dev';
const CLAIMS: StandardEventName[] = [
'page view',
'product view',
'product add',
'order complete',
];
it('maps the claimed standard events', async () => {
await runExamples('destination', destinationX, events, {
settings,
env,
claims: CLAIMS,
});
});
Minimal example (destinations)
A destination package ships examples.minimal, a Flow.MinimalExample in
src/examples/minimal.ts (the path is MINIMAL_EXAMPLE_FILE on
@walkeros/core/dev), re-exported from src/examples/index.ts. It holds only
the settings the settings schema requires, plus any that init refuses to start
without. composeQuickstart (on @walkeros/core/dev) turns it into the install
line, the integrated code and the flow.json that the package's docs page
renders under ## Installation, and the conformance kit starts the destination
with exactly those settings and requires one public success example to be
delivered.
// src/examples/minimal.ts
import type { Flow } from '@walkeros/core';
export const minimal: Flow.MinimalExample = {
settings: { accessToken: 's3cr3t', pixelId: '1234567890' },
secrets: { accessToken: 'META_ACCESS_TOKEN' },
};
// src/examples/index.ts
export * as env from './env';
export * as step from './step';
export { minimal } from './minimal';
settings: required keys only, placeholder values. An optional key stays in the examples that need it; the kit rejects a top-level key the schema does not require, unlessinitrefuses to start without it.secrets(server only): a credential setting mapped to its secret name,<VENDOR>_<SETTING>inA-Z,0-9and_. Generated code readsprocess.env.<NAME>(integrated) and$secret.<NAME>(flow) and never shows the placeholder.secretson a web destination throws.config: only{ loadScript: true }, for a web destination that loads its vendor script only when asked to. The kit fails a script-loaded web destination whose minimal example lacks it.- Several exports: one minimal example per export, in the export's folder
(
src/firehose/examples/minimal.ts) and itsexportExamplesentry; the test passesexportNameandexportExamples, the page<Quickstart data={data} export="destinationFirehose" />.
Every destination conformance test passes one of two options:
await conformance('destination', destinationMeta, {
profile: 'http',
examples: examples.step,
schema: schemas.settings,
settings: examples.minimal.settings,
minimal: examples.minimal,
root: path.join(__dirname, '..', '..'),
});
// A package that cannot have one passes the reason instead of `minimal`, and
// its docs page keeps a hand-written setup:
// withoutMinimal: 'the install line cannot express the driver choice',
A test with neither fails, and so does one that disagrees with the file:
withoutMinimal while src/examples/minimal.ts exists, minimal without it,
both, or a blank reason. The doc-lint (npm run validate:docs) fails a page
whose package has the file but whose ## Installation shows a hand-written
npm install, startFlow( or bundled tab, or no <Quickstart>.
Writing Examples
Best Practices
- Use camelCase names -- descriptive of the scenario (
checkoutPost,debugFiltered,pageViewBasic) - Cover happy path and edge cases -- include at least one positive and one negative case for transformers
- Use realistic data -- real-looking event names, IDs, and values
- Keep examples minimal -- in
in, only include fields relevant to the step's logic; omit empty/default-value fields (context: {},nested: [],user: {}, etc.). Anoutis compared exactly, so it lists every field the step emits, empty ones included - Export from
examples/step.ts-- follow the existingdev.tsstructure - Title and describe public examples -- add
title(2-5 words) anddescription(one short sentence, 10-25 words) for every example that should appear in docs. Mark test-only fixtures withpublic: false.
File Structure
// src/examples/step.ts - only Flow.StepExample exports, nothing else
import type { Flow } from '@walkeros/core';
import { getEvent } from '@walkeros/core';
export const purchase: Flow.StepExample = {
in: getEvent('order complete', { id: '1700000000-gr0up-1' }),
mapping: { name: 'Purchase', data: { map: { value: 'data.value' } } },
out: [
[
'window.fbq',
'track',
'Purchase',
{ value: 462 },
{ eventID: '1700000000-gr0up-1' },
],
],
};
// src/examples/index.ts
export * as env from './env';
export * as step from './step';
The file exports only Flow.StepExample objects. No intermediate variables,
no all, no config. Consumers iterate via Object.entries(examples.step).
Rule: examples live only on the ./dev subpath
Examples (and schemas) must be exported only from src/dev.ts (the ./dev
package subpath), never from a production entry (src/index.ts / the package
main). Production bundles tree-shake on the import path: if examples is
reachable from the main entry, the bundler inlines every example (and its
getEvent fixtures) into the shipped JS, bloating consumer bundles with
test-only data. Importing them from ./dev keeps them available for simulation
and testing while leaving production output clean.
A no-restricted-syntax ESLint rule enforces this, so a export ... examples
from a production entry fails lint.
Multi-export packages: exportExamples and exportSchemas
A package whose package.json walkerOS.exports lists more than one export
adds a map of EVERY export, the default included, to src/dev.ts:
export const exportExamples = {
destinationBigQuery: bigqueryExamples,
destinationPubSub: pubsubExamples,
};
The step's export name comes from resolveExportName (exported from
@walkeros/cli): the step's import, else bundle.packages[pkg].imports[0],
else the default. The bundler runs only the step's import or the default
export, so set import on the step for examples and code to agree. When the
module has the map, map[exportName] is used and a missing name resolves to
nothing (never a fallback to examples); without a map, examples is used.
selectDevExamples(devModule, exportName) implements this. The CLI simulate,
MCP flow_examples and package_get, core cdn.ts and the tsup
walkerOS.json (exportExamples key) all follow it. Existing named exports
(pubsubExamples, sqsExamples, ...) stay for back compat.
The same package also exports exportSchemas, every export to that export's
schemas in the shape of a single-export schemas (settings required,
mapping, setup where they exist). Reuse the very objects behind
schemas.settings for the default entry, so a toBe test pins it:
export const exportSchemas = {
destinationBigQuery: {
settings: bigquerySchemas.settings,
mapping: bigquerySchemas.mapping,
},
destinationPubSub: {
settings: pubsubSchemas.settings,
mapping: pubsubSchemas.mapping,
setup: pubsubSchemas.setup,
},
};
walkeros validate and MCP package_get pick the entry for the step's export
with the same rule. The tsup build (buildDev) fails when either map does not
list exactly the walkerOS.exports names, when an exportSchemas entry has no
settings, and when any source, transformer, destination or store package
publishes no schemas.settings (a package without settings publishes
z.object({})). Prefixed keys in schemas (pubsubSettings, ...) stay.
Mock env and simulation paths
src/examples/env.ts exports push (the mock env simulate injects) and
simulation (the call paths to record). Simulate refuses a package destination
whose export has no push mock ("No mock env for ..."), so it never calls the
real vendor.
-
A
simulationpath names the call that CARRIES the vendor request (sendServer,fetch,call:PubSub.topic.publishMessage,call:JSONWriter.appendRows,call:AWS.SNSClient.send,call:fs.createWriteStream.write), never only a constructor. -
initis not the request. Do not record aninitcall (it carries the API key and sends nothing); a path may pass THROUGH init's result (call:Mixpanel.init.track). Amplitude and Mixpanel record their tracking calls, notinit. -
Grammar (
parseCallPath, shared by the recorder): an optionalcall:prefix, then.-separated segments; an empty segment makes the path unresolvable. The one recorder isobserveEnv(env, paths)in@walkeros/core(collectorwrapEnvand web-coregetEnvdelegate to it). It returns a Proxy view; nothing on the env objects or prototypes is mutated. -
Mid-path resolution: on an object, a segment is looked up as a property. After a function, a segment is the function's own property or a parent class's static; anything else (including
call,bind,name,length) navigates the call result (or the constructed instance, fornew). Frozen objects and frozen call results are recorded through a view; only a static on a frozen function stays unresolved. A mid-path Promise (async factory) is followed with.then. Only the leaf call is recorded:fnis the path withoutcall:,argsthe leaf call's arguments.instanceof, statics and#privatefields keep working, and a chain returningthisrecords once. -
Pin every list in the package tests with the async
expectSimulationResolves(examples.env)from@walkeros/core/dev: it walks each path through thepushmock (constructing, calling, awaiting) and fails on a missing segment or a non-function leaf.import { expectSimulationResolves } from '@walkeros/core/dev'; it('declares simulation paths that resolve', () => expectSimulationResolves(examples.env)); -
Not recordable: anonymous values such as SQLite prepared-statement rows.
-
Sources: a source package that declares
simulationruns on its mock client in a source simulation, and those calls are recorded. SQS recordscall:AWS.SQSClient.send(one receive per 100 ms while running); Pub/Sub pull recordscall:PubSub.subscription(messages arrive as events). Their module-mock tests importmoduleMockEnv, the same env without the client.
Metadata: title, description, public
Every Flow.StepExample accepts three optional metadata fields that control how
it surfaces in docs and MCP output:
title?: string- overrides the default camelCase-to-spaced heading in website docs. Keep it short (2-5 words), human-readable.description?: string- one short sentence (10-25 words) rendered above each example in docs and surfaced in MCPflow_examplesoutput. Explains what the example demonstrates, not how.public?: boolean- defaults totrue. Whenfalse, the example is excluded from the Examples section of the website docs and from default MCPflow_examplesoutput. It still runs in tests and remains callable via CLI/MCPflow_simulate. An example withoutcomeis listed under "When it fails" on the package page either way, so it needs adescription.
export const purchase: Flow.StepExample = {
title: 'Purchase',
description:
'A completed order mapped to the Meta Pixel Purchase standard event.',
in: getEvent('order complete', { id: '1700000000-gr0up-1' }),
mapping: {
name: 'Purchase',
data: { map: { value: 'data.value', currency: { value: 'EUR' } } },
},
out: [
[
'window.fbq',
'track',
'Purchase',
{ value: 462, currency: 'EUR' },
{ eventID: '1700000000-gr0up-1' },
],
],
};
// Internal fixture - runs in tests, hidden from docs and default MCP output.
export const debugFiltered: Flow.StepExample = {
public: false,
in: { name: 'debug test', data: { message: 'noise' } },
out: [['return', false]],
};
When to mark public: false
Mark public: false when:
- The example is a transformer rejection (
out: [['return', false]]) that only proves filtering works. - It's an error-path or malformed-input case.
- It's a redundant variant kept only for regression coverage.
- It's an internal fixture used by cross-package tests.
Keep public (omit the field) when:
- It's the happy path.
- It teaches a distinct mapping or usage pattern.
- It's the only example for some entity/action the package handles.
Source Trigger Metadata
Source step examples can include a trigger field for simulation:
{
in: '<button data-elb="cta" data-elb-cta="label:Sign Up" data-elbaction="click">Sign Up</button>',
trigger: { type: 'click', options: 'button' }, // a click needs its selector
out: [['elb', { name: 'cta click', data: { label: 'Sign Up' } /* ... */ }]],
}
When simulating via CLI or MCP, the step example maps to SourceInput:
in->content(the actual source input)trigger->trigger(which mechanism to fire)
Destination and transformer examples don't use trigger.
Simulating with Step Examples
Use --simulate <kind>.<name> to target a specific step, then provide the input
with --event. For a source step, that is SourceInput
({ content, trigger? }):
# Simulate a source step with trigger metadata (a click needs its selector)
walkeros push flow.json --simulate source.browser --event '{"content":"<button data-elb=\"cta\" data-elbaction=\"click\">Sign up</button>","trigger":{"type":"click","options":"button"}}'
The CLI does not compare the result against out. For each simulated step it
prints the matched mapping key and every recorded call (--json: under
simulations), and exits 1 only when the push fails with an error; a mapping
that produces the wrong output still exits 0. Assert on out in your own tests
(see Testing with Examples below). Add --ingest to give a transformer,
collector or destination step a request context, --consent for the collector
consent it starts from, --command for a command example, and --page-url for
the page of a web source (default: the trigger's options.url, else
http://localhost). See using-cli.
The MCP flow_examples tool returns trigger metadata alongside in/out,
and mapping for destination examples, giving full visibility into how input
events are transformed to vendor-specific output.
Validating Examples
Cross-step example validation is included automatically when validating a flow. It does not run any step:
# Validate flow config including step example compatibility
walkeros validate flow.json
For steps connected through next or a destination's before chain, flow
validation checks that:
- At least one upstream
outis structurally compatible with a downstreamin: same type; two objects share at least half of the keys of the object with fewer keys; any two arrays are compatible regardless of content. No compatible pair is an error. This is a shape check, not an equality check. - A connection is only checked when both steps declare
examples; a step without anexampleskey produces no warning. - When both steps declare
examplesbut one side has no usable value, the connection produces a warning. An upstreamoutcounts when it passes on at least one event (next bullet); a downstreamincounts unless its example setscommand. - An
outpasses on the events of its effects:elbgivesargs[0]when it is an object;returngivesa.eventfor{ event },afor a bare event, each item for an array (fan-out);false,message.ack,message.nack,response,respondand vendor calls give nothing. - Contracts bind only where a
@walkeros/transformer-validatestep links them, as at runtime. Each validate step's own examples run through that step's resolvedsettings.contract,mode,formatandoutput, and the verdict must agree with itsout(valid: passed on; invalid:["return", false]instrict, passed on with theisValidflag false inpass). A disagreement isCONTRACT_VIOLATION(warning, error with--strict). No other example is checked against the config-levelcontractblock, and a flow without a validate step gets no contract checks.
Testing with Examples
The primary use of step examples is automated testing with it.each.
Source Tests (createTrigger)
Sources export createTrigger from their examples. It follows the unified
Trigger.CreateFn interface: receives Collector.InitConfig, lazily starts the
flow, and returns a trigger function that simulates real-world invocations.
Each package implements createTrigger differently:
- Browser: Injects HTML into DOM, starts flow, dispatches native events
- Express: Boots real HTTP server, sends
fetch()requests
runExamples feeds each example's in through the package's createTrigger
and compares the elb calls the source makes with out:
import { runExamples } from '@walkeros/collector/dev';
import { sourceExpress } from '@walkeros/server-source-express';
import { examples } from '@walkeros/server-source-express/dev';
it('runs the step examples', async () => {
await runExamples('source', sourceExpress, examples.step, {
createTrigger: examples.createTrigger,
settings: { port: 0 },
});
});
The run passes, but today the option is typed for unknown content, so tsc
rejects a package's typed createTrigger here (TS2769). The express package's
own src/__tests__/stepExamples.test.ts shows a typed loop that narrows in
with a type guard instead.
Browser source note: The browser source's elbLayer processes events via a detached promise chain (fire-and-forget). A test that collects events with its own spy destination after an interactive trigger (click, submit) polls until they arrive:
while (!events.length) await Promise.resolve();
CMP Source Tests (Consent Assertion)
CMP sources push walker consent commands, not regular events. Assert on
collector consent state instead of using a spy destination:
const instance = await examples.createTrigger({
consent: {},
sources: {
usercentrics: {
code: sourceUsercentrics,
config: { settings: {} },
},
},
});
await instance.trigger(
example.trigger?.type,
example.trigger?.options,
)(example.in);
// Yield for the detached elb('walker consent') chain
await new Promise((resolve) => setTimeout(resolve, 20));
// out is [['elb', 'walker consent', consent]]
expect(instance.flow?.collector.consent).toEqual(example.out?.[0]?.[2] ?? {});
Server Function Handler Tests (collector.sources)
Server function handlers (fetch, AWS Lambda, GCP CloudFunction) don't own
servers. Their createTrigger accesses the source instance from
collector.sources after startFlow and calls source.push() with
platform-native types:
function findSource(collector) {
for (const source of Object.values(collector.sources || {})) {
if (source.type === 'fetch') return source;
}
}
// In trigger:
const source = findSource(flow.collector);
const response = await source.push(request);
Destination Functional Tests
For destinations, runExamples('destination', ...) (shown under Public examples
above) is the test: it starts a real collector, applies each example's mapping
through the destination config, records the calls named by
examples.env.simulation, and compares them with out. Pass the package's
env ({ push, simulation }), the settings, and for a web destination page
from @walkeros/web-core/dev (run in a jsdom test environment).
Transformer Tests
A test that calls the transformer's push directly wraps its return in the
out shape (runExamples('transformer', ...) records what reaches the next
step instead, see Transformer Step Example):
import type { WalkerOS } from '@walkeros/core';
import { createMockContext } from '@walkeros/core';
import { transformerMyTransformer } from '../transformer';
import { examples } from '../dev';
describe('transformer', () => {
it.each(Object.entries(examples.step))('%s', async (name, example) => {
const instance = await transformerMyTransformer(
createMockContext({ id: 'myTransformer' }),
);
const result = await instance.push(
example.in as WalkerOS.DeepPartialEvent,
createMockContext({ id: 'myTransformer' }),
);
const actual =
result === false
? [['return', false]]
: result
? [['return', result]]
: [];
expect(actual).toEqual(example.out);
});
});
Checklist
When adding step examples to a package or flow:
- Create
src/examples/step.tswithFlow.StepExampleexports - Export from
src/examples/index.ts - Use camelCase, descriptive example names
- Include at least one happy-path example
- For transformers, include a
false(filtered) case - Add
it.eachtest using step examples - Exactly one public example per standard event the destination maps, from
getEvent(name), withclaimsin arunExamplescall; everyouta literal written from the vendor's documentation - Failure examples (
public: false,outcome,respond, adescription) for every event the step cannot send, asilentexample for a destination, and a conformance test (conformancefrom@walkeros/collector/dev) - Destination:
src/examples/minimal.tswith only the settings the schema requires orinitrefuses to start without (credentials assecretson a server destination,config.loadScriptfor a script-loaded web destination), exported asexamples.minimaland passed toconformanceasminimal; orwithoutMinimal: '<reason>' - Verify examples compile:
npm run build - Run tests:
npm run verify:touched -- <pkg>(L1) - Add
titleanddescriptionto every public example. - Mark internal/test-only examples with
public: false. -
examples/env.tsexportspushand asimulationlist naming the request call, pinned withexpectSimulationResolves(examples.env). - Multi-export package:
exportExamplesandexportSchemasindev.tslist every export (the build fails otherwise). - In a flow file, add examples sparingly: they are mainly internal and for
demos (for example a client flow's source with
public: falseexamples showing its default behaviour).
Related Skills
- testing-strategy - Testing patterns and env mocking
- understanding-flow - Architecture and data flow
- using-cli - CLI simulate and validate commands
- create-destination - Destination creation workflow
- create-source - Source creation workflow
- create-transformer - Transformer creation workflow
Using store-level cache
Overview
Flow.Store.cache wraps any store with a read-through, write-through cache
tier. Reads check the cache first, fall through to the backing on miss, and
populate every tier on the unwind. Writes go to the backing first, then to the
cache best-effort.
The wrapping is transparent: a transformer wired to $store.crm does not know
whether reads hit a memory cache, a GCS tier, or the underlying Sheets API.
Core principle: the cache is advisory. Backing is the source of truth. Failed cache operations degrade performance, never correctness.
TTL is owned by the cache layer, not the store. The store persists
structured values; the cache wrapper manages expiry via the per-rule ttl. An
expired entry is treated as a miss and re-fetched from the backing. Caching
works over any structured backing.
A store's byte-native file mode (config.file: true) is for serving raw
assets byte-exact, not for caching. The cache wraps structured stores, so
flow_validate warns when a store sets both file: true and cache. Drop one:
serve bytes with file: true and no cache, or cache a structured store with no
file.
When to use this skill
- A transformer reads the same key many times across events (sessions, user lookups, product catalog enrichment)
- A store has rate limits or slow HTTP round-trips (Sheets, custom API, S3
metadata);
@walkeros/server-store-apireads a third-party API, and its misses are not cached - You need to deduplicate concurrent reads on a cold key (thundering herd)
- You want to compose multi-tier caches (memory → GCS → Sheets)
Minimal recipe: cache a Sheets store
The talk-demo use case. A sessions lookup runs on every event but most events
in a session share the same key.
{
"stores": {
"sessions": {
"package": "@walkeros/server-store-sheets",
"config": {
"credentials": "$secret.SHEETS_CREDENTIALS",
"settings": {
"id": "1AbC...",
"sheet": "Sessions"
}
},
"cache": { "rules": [{ "ttl": 300 }] }
}
},
"transformers": {
"sessionLookup": {
"code": {
"type": "session-lookup",
"push": "$code:async (event, context) => { if (!event.user?.session) return; const session = await context.env.store.get(event.user.session); if (session) return { event: { ...event, data: { ...event.data, session } } }; }"
},
"env": { "store": "$store.sessions" }
}
}
}
walkerOS ships no generic lookup transformer, so the consumer here is an inline
code transformer. Any transformer or destination that reads its env.store
works the same way.
The first lookup hits Sheets and populates the built-in in-memory tier with a 300-second TTL. The next 300 seconds of identical reads hit memory and skip the Sheets API.
Without the cache: 60 events in 60 seconds = 60 Sheets reads, each counted against the Sheets API read quota. With the cache: 60 events in 60 seconds = 1 Sheets read.
Recipe: memoize an object storage store
Same shape, longer TTL because every read is a network round-trip to the bucket:
{
"stores": {
"users": {
"package": "@walkeros/server-store-gcs",
"config": {
"settings": { "bucket": "my-user-records", "prefix": "users/" }
},
"cache": { "rules": [{ "ttl": 3600 }] }
}
}
}
One-hour TTL is reasonable when user records change rarely. Use flow_validate
to verify the config; the hit rate shows at runtime in the store's counters
(see Observability).
Recipe: TTL by key prefix
Different keys can have different TTLs. Rules evaluate top-down, first match
wins. The match context is { key, value? }, not event data:
"cache": {
"rules": [
{ "match": { "key": "key", "operator": "prefix", "value": "session:" }, "ttl": 300 },
{ "match": { "key": "key", "operator": "prefix", "value": "user:" }, "ttl": 3600 },
{ "ttl": 60 }
]
}
session:*keys cache for 5 minutesuser:*keys cache for 1 hour- Everything else caches for 1 minute
A rule without match always matches. Place it last as a fallback.
Recipe: multi-tier composition (memory → GCS → Sheets)
When the working set exceeds the memory tier's capacity, or cached values should
survive restarts and be shared across instances, add an object storage layer
between memory and the cold backing. The consumer still wires to $store.crm;
the tiers resolve automatically. Use fs, S3, GCS or the built-in memory tier as
a cache tier (a cache.store target). Nothing rejects the Sheets store there,
but every cache read and write then costs a Sheets request.
{
"stores": {
"files": {
"package": "@walkeros/server-store-gcs",
"config": {
"settings": { "bucket": "my-cache-bucket", "prefix": "cache/" }
},
"cache": { "rules": [{ "ttl": 300 }] }
},
"crm": {
"package": "@walkeros/server-store-sheets",
"config": { "settings": { "id": "1AbC...", "sheet": "Customers" } },
"cache": {
"store": "files",
"rules": [{ "ttl": 86400 }]
}
}
},
"transformers": {
"crmLookup": {
"code": {
"type": "crm-lookup",
"push": "$code:async (event, context) => { if (!event.user?.id) return; const crm = await context.env.store.get(event.user.id); if (crm) return { event: { ...event, data: { ...event.data, crm } } }; }"
},
"env": { "store": "$store.crm" }
}
}
}
Lookup chain on crm.get(K):
crm's tier is thefilesstore. Thefileswrapper checks its own tier (memory__cache) first, then GCS. HIT, return. A GCS HIT also populates memory.- On all MISS, call the underlying Sheets store. Each traversed tier populates
on the unwind:
crmwrites tofiles, which writes GCS and then memory.
TTL ordering: shortest at the top (the files memory tier, 300s), longest at
the cold end (the crm entries in GCS, 86400s). Each wrapper checks the expiry
of its own entries, also when a lower tier serves them, so a value is served
stale for at most the longest TTL in the chain (86400s here).
Async-safe by design. Whether your cache store's get is synchronous (the
built-in __cache, an in-memory store) or asynchronous
(@walkeros/server-store-fs, S3, GCS, the cache wrapper itself), the collector
reads through with an await internally. You can mix sync and async stores
freely in a multi-tier chain without any extra configuration; the same HIT/MISS
semantics apply.
Recipe: prevent thundering herd on a cold key
Single-flight deduplication is on by default. 50 concurrent
store.get('session:abc') calls on a cold cache produce one backing call,
not 50. All callers receive the same promise.
This is what makes store-level cache useful on a slow backing under high
concurrency. No configuration needed; just set cache on the store.
Verify it works with the inflight_dedups counter (see Observability below).
What cache rules cannot do (compared to event cache)
Store rules are a stricter subset:
- No
keyfield. The cache key comes from the caller (store.get(K)); there is no event path to compose. - No
updatefield.updatechanges the response a source cache sends; a store read has no response to change. - No
stopfield. Stores always fall through on miss; halting the pipeline is an event-cache concept. - Empty
namespace: ""is rejected by the schema (re-introduces the collision footgun across stores sharing__cache).
The schema rejects key, update and stop on a store cache. Use the
event-level Cache when you need them:
key and stop on sources, transformers or destinations, update on a source
cache only.
Default tier: built-in __cache
Omitting cache.store falls back to the collector's built-in __cache. It is
an in-memory LRU map with:
maxEntries: 10000(not configurable)- LRU access ordering on reads
- Batched eviction down to 80% on overflow
- Active TTL sweep every 60 seconds
Each wrapped store gets an automatic namespace prefix (the store id) so multiple
stores sharing __cache do not collide. Override with cache.namespace: "myns"
if you want explicit control.
At the INFO level the collector logs one line per wrapped store at startup:
INFO [store-cache:sessions] store "sessions" caches with namespace "sessions:" via __cache
Write-through error policy
wrapped.set(K, V) runs two steps:
- Backing first. Await
backing.set(K, V). If this throws, the wrapper throws. The cache is not touched. - Cache best-effort. If the backing succeeded, attempt to populate the cache. If this throws, log a warning and return success.
wrapped.delete(K) follows the same shape. A failed cache delete leaves a
poisoned entry that serves stale data until TTL. Both warnings are WARN lines,
which the default log level (ERROR) does not show.
Backing is the source of truth. Code that wraps set / delete should assume
the cache may be lagging.
Coherence model
Read this before relying on the cache for anything correctness-sensitive:
- Read-your-writes (in-process): yes. After
wrapped.set(K, V), a subsequentwrapped.get(K)in the same process returnsV. - Cross-process consistency: eventual, bounded by the longest TTL in the chain. No invalidation channel.
- Tier-skipping repopulation: a MISS in tier N that HITs in tier N+1 repopulates tier N. Subsequent reads hit tier N.
Pick TTLs accordingly. Short TTLs (1-60s) for mostly-static lookups behind a fast backing; long TTLs (minutes-hours) for cold, expensive lookups where staleness is tolerable.
Observability
Each wrapped store exposes counters:
| Counter | Use this to detect |
|---|---|
hits | Cache is actually working |
misses | Working-set size, cold start |
populates | New keys being added to cache |
writes | Set call volume |
deletes | Delete call volume |
inflight_dedups | Concurrent reads on a cold key |
The eviction counters (evictions_entries for the maxEntries cap,
evictions_ttl for expired entries) belong to the built-in tier:
collector.stores.__cache.counters. Each read through a wrapped store also
emits a flow record to the collector's observers (meta.op: "cache",
meta.status hit or miss).
For interactive debugging at runtime:
const { collector } = await startFlow({
/* ... */
});
const sessions = collector.stores.sessions;
if ('counters' in sessions) console.log(sessions.counters);
// { hits: 412, misses: 18, populates: 18, writes: 0, deletes: 0,
// inflight_dedups: 7 }
Healthy cache: hits / (hits + misses) rises over time. inflight_dedups
proves the herd prevention worked.
Known limitations
- No negative caching. A
get(K)that returnsundefinedfrom the backing is not populated. Every subsequent call for a missing key re-hits the backing until the value exists. Workaround: write a sentinel value on the first miss and treat it as "not present" in transformer logic. - No cross-process invalidation. Writes from one process do not invalidate caches in other processes. TTL is the only mechanism.
- Cycles are rejected at startup.
A.cache.store = BandB.cache.store = AmakestartFlowreject withCycle in cache.store chain: a -> b -> a. - A cached store is not destroyed at shutdown. The cache wrapper does not
forward
destroy, so a store declared withcachenever has itsdestroycalled. - Renaming a store is a breaking change to anything caching through it
(
cache.store: "X"references break). Migrate explicitly.
Fetch & stash without $code:
When a step only needs to read a value out of a store or write one into it, you
do not need to wire $store into the step's env and hand-write a $code:
push. The declarative state block on a source, transformer, or destination
does both directions through the mapping engine. The example uses the sessions
store declared in the first recipe:
"transformers": {
"stashGclid": {
"state": { "mode": "set", "store": "sessions", "key": "event.user.session", "value": "event.data.gclid" }
},
"restoreGclid": {
"state": { "mode": "get", "store": "sessions", "key": "event.user.session", "value": "event.data.gclid" }
}
}
key and value resolve against { event, ingest }, so every path names its
side: event.user.session reads the event, ingest.site reads the pipeline
context a server source lifted from the request. A path without one of these
prefixes never resolves: walkeros validate rejects it and the runtime warns.
key names the store slot and mode sets the direction:
setresolvesvalue(a path, constant,fn, ormap) and writes that payload to the store underkey.getreadskeyfrom the store and writes the fetched value to thevaluepath:event.xonto the event,ingest.xinto the ingest, where later steps and route matchers see it but no destination does. For aget,valuemust be a string path (or aValueConfigwithkey), not a constant or operator.
A store keyed by request context, such as a per-site registry, reads the key from ingest:
"registry": {
"state": { "mode": "get", "store": "registry", "key": "ingest.site", "value": "ingest.tenant" }
}
A transformer runs its get before its own next route, so the route can match
on ingest.tenant. A source runs its state before its next route too, so a
source-level get can feed that route.
To fill several fields from one lookup, add mapping. Its paths are relative to
the value read, and the result merges into the target:
"loadUser": {
"state": {
"mode": "get", "store": "customers", "key": "event.user.id", "value": "event.user",
"mapping": { "map": { "ltv": "ltv", "segment": { "key": "segment", "value": "unknown" } } }
}
}
An object result merges into an object target, anything else replaces it; on a
set the stored entry is read, merged and written back (not atomic). The
mapping runs on a miss too, so only declared fallbacks are written; fn and
condition then receive undefined, while validate receives the resolved
value (a declared fallback, or {} after a map).
Omit store to use the built-in __cache tier; state keys there are prefixed
with state: so they never collide with cache entries. State is fail-open:
a store error is logged and the event passes through unchanged. A store not
declared in flow.stores, or one with file: true, is a validation error
(walkeros validate and deploy preflight). Use state for simple fetch/stash;
reach for $code: only when the logic is genuinely non-declarative.
Full reference: Website: State.
Related Skills
- walkeros-understanding-stores -
Store interface,
$store.wiring, lifecycle - walkeros-understanding-flow - Architecture, stores as passive infrastructure
- walkeros-using-cli - Bundling and simulating flows with stores
Documentation:
- Website: Store-level cache - Full
reference for
Flow.Store.cache - Website: Stores - Stores overview
- Website: Collector Cache - Event-level cache integrated into the collector (separate concept)
Source files:
- packages/core/src/types/cache.ts -
Cache,StoreCacheRule,EventCacheRuletypes - packages/collector/src/store-cache-wrapper.ts - Wrapper implementation
- packages/collector/src/store.ts - Two-phase store init
Using @walkeros/transformer-ga4
Overview
@walkeros/transformer-ga4 decodes Google Analytics 4 Measurement Protocol v2
hits (/g/collect, /mp/collect) into walkerOS events. It sits in a server
source's before chain, reads the raw HTTP request via ctx.ingest, and
returns one walkerOS event per GA4 event in the hit (one hit can carry many
events).
Boundaries: server-side decoding in front of a server source, GA4 v2 only, G-
tids only by default. Per-field patching via extend/remove is supported.
When to use this skill
- You are wiring
transformer-ga4into a server flow for the first time - You want to override one of the 33 default GA4 mappings
- You need to drop a GA4 event (auto-fired noise like
user_engagement) - You want to add a custom event key for a
gtag('event', '<name>', ...)call - You need to widen
tidPatternto accept Ads (AW-) or DC (DC-) traffic - Events are not arriving and you suspect the decoder is silently dropping them
Quickstart wiring
Pick the source by runtime before copying the example below. gtag.js can
send several events in one POST body (one per line, as text/plain), so the
source must keep a non-JSON body as ingest.body. These server sources do:
-
@walkeros/server-source-express(shown below), the sourcerunnerosand thewalkeros/flowimage serve. Atext/plainbody that is valid JSON is parsed; any othertext/plainbody reaches the decoder as the raw string. -
@walkeros/server-source-fetch: a(Request) => Responsehandler with noport, for a runtime that calls it (Cloudflare Workers, Deno, Bun, Node.js 18+ with a fetch adapter), e.g. withcollectorfromstartFlow:import { Source } from '@walkeros/core'; import type { SourceFetch } from '@walkeros/server-source-fetch'; export default { fetch: Source.getSource<SourceFetch.Types>(collector, 'http').push, };runnerosand thewalkeros/flowimage cannot serve it (they only mount a source's NodehttpHandler). -
sourceCloudFunctionfrom@walkeros/server-source-gcpon Google Cloud Functions. -
sourceLambdafrom@walkeros/server-source-awson AWS Lambda. It answers a POST without a body with 400, so body-less single-event hits are lost there.
The other sources take the same ingest and before config as the express
example. The fetch source also takes paths; none of them takes port.
{
"version": 4,
"flows": {
"default": {
"config": { "platform": "server" },
"sources": {
"http": {
"package": "@walkeros/server-source-express",
"config": {
"settings": {
"port": 8080,
"paths": ["/g/collect"]
},
"ingest": {
"map": {
"url": { "key": "url" },
"path": { "key": "path" },
"method": { "key": "method" },
"body": { "key": "body" }
}
}
},
"before": "ga4"
}
},
"transformers": {
"ga4": { "package": "@walkeros/transformer-ga4" }
},
"destinations": {
"log": { "package": "@walkeros/destination-demo" }
}
}
}
}
Wiring rules:
- The express source listens on
/collectby default and matches paths exactly. gtag.js sends to/g/collect, so setsettings.pathsto["/g/collect"](or the full path, if the collector URL carries a path prefix). Without it every GA4 hit gets a 404. - On the website, point gtag at the collector origin:
gtag('config', 'G-XXXXXXXXXX', { server_container_url: 'https://collect.example.com' }). gtag.js appends/g/collect. With the walkerOS gtag destination, setsettings.ga4.server_container_url(orsettings.ga4.transport_url). Hits then go to the collector instead of Google. The/g/collectsuffix, the body-less single-event POST, and one-line-per-event batches are gtag.js behaviour, not a walkerOS contract: confirm them in the browser Network tab. before: "ga4"runs every request on the source through the decoder, and a request without GA4 parameters is dropped. If the source also receives walkerOS events (for example on/collect), gate the chain:"before": { "match": { "key": "ingest.path", "operator": "eq", "value": "/g/collect" }, "next": "ga4" }.- Batched hits (several events in one POST body, one per line) reach the decoder
through
server-source-expressas the rawtext/plainstring, alongside GET and body-less POST hits. A body sent asapplication/jsonmust be valid JSON, or express answers 400 before the decoder runs. transformer-ga4belongs insource.before, notdestination.before. Decoding is a pre-collector concern: GA4 hits are not yet walkerOS events.- The source must populate
ctx.ingest.url(required, string) andctx.ingest.body(optional, string). Raw text body only: pre-parsed JSON will not decode. The express, fetch, GCP and AWS server sources keep a non-JSONtext/plainbody (gtag's batched POST) as the raw string iningest.body; only a body that parses as JSON arrives parsed. - One HTTP request can fan out to N walkerOS events.
Common mistake: config.ingest must be the map operator with direct field
paths on the request scope (no req. prefix), e.g.
{ "map": { "url": { "key": "url" } } }. The bare { "url": "req.url" } form
has no operator: walkeros validate and the bundle reject it
(MAPPING_VALUE_NO_OPERATOR). If it still runs, the collector logs an ERROR,
ctx.ingest stays empty, and the decoder fails every hit with missing-input.
This wiring contract (before-chain placement plus the config.ingest keys the
source must populate) is also surfaced as a package hint, so package_get on
@walkeros/transformer-ga4 returns it without this skill.
Configuration recipes
Patch one field in a default mapping (extend + remove)
Use extend to add or override individual fields of a shipped default rule
without replacing it in full. Use remove to strip fields from the output.
The purchase default ships id, currency, value, tax, shipping, and
coupon. To add affiliation from a GA4 event parameter and drop currency:
"transformers": {
"ga4": {
"package": "@walkeros/transformer-ga4",
"config": {
"settings": {
"mapping": {
"purchase": {
"extend": {
"data": { "map": { "affiliation": "params.ep.affiliation" } }
},
"remove": ["currency"]
}
}
}
}
}
}
extend.data.map is deep-merged onto the default data.map, leaving id,
value, tax, shipping, and coupon intact. remove: ["currency"] strips
that field from the final payload. A null value in extend clears an
inherited field, for example "extend": { "data": { "map": { "tax": null } } }
drops tax from the default data.map.
Full replace: if you need to rewrite a rule from scratch (no merge), omit
extend and remove and supply the complete rule directly. A rule with neither
keyword keeps the existing replace behavior.
"settings": {
"mapping": {
"purchase": {
"name": "order complete",
"data": {
"map": {
"id": "params.ep.transaction_id",
"value": "params.epn.value",
"currency": "params.ep.currency",
"coupon": "params.ep.promo_code"
}
}
}
}
}
Drop an event
Set ignore: true on any key. This is how user_engagement, session_start,
and first_visit are silenced by default:
"settings": {
"mapping": {
"click": { "ignore": true }
}
}
Add a custom event mapping
For events fired via gtag('event', '<your_name>', ...):
"settings": {
"mapping": {
"newsletter_subscribe": {
"name": "newsletter signup",
"data": {
"map": { "source": "params.ep.source" }
}
}
}
}
Unknown event names fall through to the '*' key, which by default emits
ga4 track with data.event_name set to the original en. Override '*' to
change the global fallback rule.
Set a custom tidPattern
By default only G- tids are accepted. Widen via a string regex (compiled at
init):
"settings": {
"tidPattern": "^(G|AW|DC)-"
}
Change the batch cap
settings.maxEvents (default 100) caps the events decoded from one request. A
POST body with more non-empty lines is dropped whole, before decoding:
"settings": {
"maxEvents": 250
}
Custom destination after decode
The transformer returns walkerOS events with entity action names. Anything
downstream of the collector treats them as native events and needs no GA4
awareness:
"destinations": {
"bq": {
"package": "@walkeros/server-destination-gcp",
"import": "destinationBigQuery",
"config": {
"settings": {
"projectId": "my-eu-project",
"datasetId": "events",
"location": "EU"
}
}
}
}
Mapping path syntax
Mappings reference fields on the decoded GA4Hit shape via dotted paths:
| Path | Meaning |
|---|---|
params.ep.X | Event parameter (string), e.g. params.ep.transaction_id |
params.epn.X | Event parameter (numeric), e.g. params.epn.value |
params.up.X | User property (string) |
params.upn.X | User property (numeric) |
items[0].X | First item in the items array |
items | Full items array (mapped to nested via item rules) |
Use params.ep.X not ep.X. The decoder materializes the prefixed keys under
the params namespace.
Hit-level fields on the decoded event
Set from the hit, each only when present, value as sent.
| Hit param | Event field |
|---|---|
dl | source.url (page URL) |
dr | source.referrer |
uid, cid, sid | user.id, user.device, user.session |
ul | user.language (browser language) |
sr | user.screenSize (screen resolution) |
p, _p, _s | source.platform, source.pageLoadId, source.hitSequence |
globalsis left empty:globals.languageis the site's language, not the visitor's. Mappings or destinations that readglobals.languageorglobals.screenfrom decoded events must switch touser.languageanduser.screenSize.timestampis the receive time (Date.now()), not the session start (sid). Event ids do not depend on it.source.urlfeeds server destinations that send a page URL (Meta, Snapchat, Pinterest, Bing, TikTok, Criteo, Piwik PRO; Piwik PRO fails events without it withmissing-input) and the fingerprint transformer's defaultsiteinput. Fingerprint hashes of GA4-decoded events therefore change once on upgrade.
Troubleshooting
No events arriving
- A 404 on
/g/collectmeans the source does not listen there. Setsettings.pathsto include/g/collect(the express and fetch sources default to/collect). - Check that
source.config.ingestis themapoperator and populatesurl(required, string) andbody(optional, string), e.g.{ "map": { "url": { "key": "url" }, "body": { "key": "body" } } }. A bare{ "url": "req.url" }object failswalkeros validate(MAPPING_VALUE_NO_OPERATOR); at runtimectx.ingeststays empty and every hit fails withmissing-input, one ERROR line per cause and minute. - Confirm
before: "ga4"is set on the source, not on a destination. - Check the
tidPattern: by default^G-blocksAW-/DC-traffic silently. - Confirm the request really is Measurement Protocol v2. v1 hits will not decode.
- A POST body with more non-empty lines than
settings.maxEvents(default100) is dropped whole. The client still gets a success response; the only trace is one ERROR line with the line count and the cap. Raise the cap if your own server-to-server batches are larger.
Wrong field in the output event
- Check the path syntax:
params.ep.X, notep.X. - Verify the GA4 hit actually carries the parameter. Use a debug destination to
log the decoded
paramsobject. - If you used a full-replace rule (no
extend), yourdata.mapis the entire rule. Useextendto inherit the defaults and add only the fields you need.
Batched POST hits return 400
@walkeros/server-source-express only rejects a POST body that is sent as
application/json and does not parse. gtag.js sends text/plain, which express
keeps as the raw string. If a proxy in front of the collector rewrites the
content type to application/json, restore text/plain.
gcs consent not applied
v1 decodes basic gcs (G1XX → marketing / analytics booleans) only.
Functional/preferences flags and the newer gcd parameter are not decoded.
Override the consent path manually if you need richer mapping.
Limitations
- GA4 v2 only. v1 Measurement Protocol is out of scope.
G-tids only by default. OverridetidPatternfor Ads/DC.- Basic
gcsonly. Nogcd, no functional/preferences flags. - Raw text body required. Pre-parsed JSON bodies will not decode. Batched
gtag POSTs (
text/plain, one URL-encoded hit per line) reachingest.bodyraw on the express source. - At most
maxEventsevents per request. Default100; a larger batch is dropped whole before decoding, because each event becomes its own push and a source's batch limit does not cover a raw text body. - Server-side only. Web ingest via interception sources is not supported.
Related Skills
- walkeros-understanding-transformers - Transformer interface, return values, and before/next chaining this skill relies on
- walkeros-understanding-mapping -
The
data/mappath syntax used in the GA4 mapping recipes - walkeros-mapping-configuration - Mapping recipes for GA4 and other vendors
- walkeros-using-cli - Validate and simulate a flow that wires the GA4 transformer
See the website docs for the authoritative reference and the README for the in-repo summary.
Writing Documentation
When to Use This Skill
- Creating a new package README
- Writing website documentation (MDX)
- Creating or updating skills
- Reviewing documentation for quality
- Documenting Phase 7 of create-destination or create-source
Prerequisites
- understanding-flow - Architecture context
- understanding-events - Event naming rules
Documentation Types
Where Content Belongs
| Type | Purpose | Audience |
|---|---|---|
| Package README | Installation, basic usage, API reference | Package users |
| Website docs | Guides, integration examples, detailed config | Integrators |
| Skills | Process knowledge, workflows | AI assistants, contributors |
Divio Documentation Types
Keep these separate - don't mix tutorials with reference:
| Type | Purpose | User State |
|---|---|---|
| Tutorial | Learning | Studying, beginner |
| How-To Guide | Problem-solving | Working, knows what they need |
| Reference | Information lookup | Working, needs facts |
| Explanation | Understanding | Studying, needs context |
Example Validation (CRITICAL)
The Problem
AI-generated examples can be:
- Syntactically correct but use non-existent APIs
- Plausible-looking but don't match actual exports
- Outdated, referencing deprecated patterns
Source of Truth Hierarchy
TIER 1: apps/quickstart/
✓ Tested ✓ Compiled ✓ CI-validated
→ USE FOR: All code examples
TIER 2: packages/core/src/eventGenerator.ts
✓ Canonical events ✓ Real data structures
→ USE FOR: Event examples
TIER 3: packages/*/src/index.ts exports
✓ Actual public API
→ USE FOR: Verifying API names exist
TIER 4: Package READMEs & Website docs
⚠ May contain errors
→ VERIFY against Tier 1-3 before trusting
Pinned onboarding examples
Getting-started and onboarding code examples are sourced from Tier-1
apps/quickstart/src/*.ts (CI-validated) first, then the docs source them. The
current canonical examples:
first-event.ts, the zero-to-first-event console example used by the quickstart and integrated-mode pages and the README.ga4-ecommerce.ts, theproduct addto GA4add_to_cartmapping used by the GA4 reference page and the GA4 ecommerce tutorial.
These are pinned against drift by the doc-lint at
apps/scripts/validate-docs.ts (run npx tsx apps/scripts/validate-docs.ts).
It extracts canonical tokens from the Tier-1 source and asserts the docs
snippets contain them, tolerating intentional doc simplifications (dropping the
seen capture, arrow push instead of a method). first-event.ts is pinned on
the quickstart, integrated-mode, and README pages, plus the GA4 ecommerce page;
react.mdx is pinned shape-only (it pushes no page view).
The doc-lint also runs these structural and coherence checks:
-
elb event form (docs, skills, READMEs): every event is pushed as a partial event,
elb({ name: 'entity action', data }). A quoted"entity action"string as the firstelbargument is an error;walkercommands keep their string name (elb('walker consent', ...)). -
Bundled-snippet platform guard: every JSON flow with
bundlecarriesconfig.platform(getting-started, README, flow reference). -
Docker env guard: quickstart Docker snippets use
BUNDLE, notFLOW. -
Forbidden CLI verbs: no
walkeros serveor standalonewalkeros simulatein getting-started, guides, or the gtag page. -
Same-page simulate-key resolution (getting-started): every
push --simulate <type>.<key>must resolve to a step key declared in a flow.json on the SAME page; a simulate command on a page with no flow.json fails. This is what catches an integrated page showing a bundled push loop. -
Flow.json schema validation (all docs): every flow snippet, fragments included, is checked with core's validators (
validateFlowConfig, thenvalidateFlowStructure, from@walkeros/core/dev). A fragment that starts at a key gets braces; a flow part (sources,destinations, ...) is checked inside a v4 flow, with the platform from the fragment or the page path; a root part (contract,variables) is checked as a v4 file; on a page whose frontmatterpackage:is a step package, a step or step config part is checked as that step. Package settings are not checked against the package's schema, and nothing is fetched.version: 3migration "before" snippets and JSON that is no flow part are skipped and counted in the summary line. -
Boundary register (getting-started + MCP page): no prices, tier names, or upgrade language. No
:::cloud(all docs): there is no walkerOS Cloud, and the callout type no longer exists.
The rule: when you edit a pinned example, edit the Tier-1 .ts source and
re-run the doc-lint. If the docs snippet must change too, update both so the
lint passes.
Examples must run
A package page shows only setup and examples that a test ran, rendered from the
package's walkerOS.json; never a hand-written copy of them:
- Install and setup are generated (destination pages).
## Installationrenders<Quickstart data={data} />(withexport="<name>"for a package with several exports): the install line, the integrated code and theflow.json, composed from the package'sexamples.minimal, the settings its conformance test delivers with. For a package withsrc/examples/minimal.ts, the doc-lint fails annpm install,startFlow(or bundled tab under## Installation, and a section that does not render<Quickstart>. A destination whose conformance test passeswithoutMinimal(gtag, sqlite) keeps a hand-written setup, as do source, transformer and store pages. - Frontmatter names the data. The page's
package:frontmatter and thewalkerOS.jsonit imports name the same package; the doc-lint fails a mismatch. - Failures come from examples.
<Configuration>lists every step example with anoutcomeunder## When it fails(title,description, status and code; never itsin). Add a failure mode as a failure example with adescriptionin the package instead of a hand-written list on the page; the docs build fails on a failure example without adescription.
Hand-written examples (source, transformer, store and guide pages, READMEs) follow the same bar:
- A flow runs. Every complete JSON flow example (
"version": 4) passeswalkeros validate --strictand runs withwalkeros push --simulate;validatealone does not prove a config does what the page says. - Values satisfy the schema. Placeholders match the setting's pattern (a numeric pixel id where the schema asks for digits), and timestamps are milliseconds wherever the code reads milliseconds.
- Output carries no real version. Example output uses a stand-in such as
release: { web: '3' }; a real version number goes stale with the next release. - Renamed headings keep their anchor. A heading's slug is a public URL. When
its text changes, pin the old slug (
## New title {#old-slug}) and update inbound links.
The package side (minimal, withoutMinimal, failure examples) is in
using-step-examples.
Getting-started editorial rules
Three rules govern the getting-started surface (decided 2026-07-06, encoded in the doc-lint):
See your event, per mode. Integrated-mode pages (quickstart, react, nextjs,
modes/integrated, ga4-ecommerce) verify via their own console output; they must
never show walkeros push --simulate. Only bundled-mode pages may show it, and
only targeting a <type>.<key> that a flow.json on the SAME page defines.
Open/paid boundary (Self-Host Sufficiency Rule). Every getting-started task
must reach a working result with MIT packages, the CLI, and reader-controlled
infrastructure only. There is no walkerOS Cloud: never write "Cloud" as a
walkerOS product, and never add a "What Cloud adds" pointer. Where docs describe
commands or tools that talk to the hosted app (auth, projects, flows,
deploy), call it "the walkerOS app". Never prices, tier names, or upgrade
language anywhere in open docs. Self-host is always listed first.
Examples recipes. Pages under getting-started/examples/ are lean proof
recipes with a fixed shape: exactly one flow.json, one run loop, one
expected-output block, plus a short intro and next steps. The run loop uses
walkeros push … --simulate <type>.<key> where <key> is a step key declared
in the page's own flow.json; recipes are bundled flows, so the per-mode rule
applies and the same-page simulate gate enforces resolution. The expected-output
block shows what actually arrives (the captured destination call or the logged
event), derived from the package's step examples or the real CLI output; never
invent config, mappings, or output. Every snippet is derived from validated
material (package examples, tested sources, existing validated docs snippets).
The shape is enforced by checkRecipeProof in apps/scripts/validate-docs.ts.
Validation Checklist
Before publishing ANY code example:
- API exists? Check
packages/*/src/index.tsexports - Pattern validated? Compare against
apps/quickstart/ - Events canonical? Use patterns from
eventGenerator.ts - Example compiles? TypeScript check passes
- Imports correct? Package names match actual packages
- Env vars, CLI flags, container modes exist? Each one is grepped in
source before it is documented (e.g.
grep -rn "process.env.PORT" packages/runner/src,grep -n -- "--port" packages/runner/src/bin.ts). A name that only appears in other docs is not evidence
Red Flags
| Red Flag | What It Indicates |
|---|---|
| API name not in package exports | Hallucinated or outdated API |
| Import path doesn't match package.json | Wrong package reference |
| Event name with underscore | Wrong format (should be space) |
| No imports shown | Context missing, harder to validate |
| Env var or flag with no source grep | Likely fiction copied from old docs |
DRY Patterns
Configuration snippet (package pages)
When to use: Every package page for destinations, sources, transformers, and
stores. The <Configuration> snippet owns the ## Configuration,
## Settings, ## Mapping, ## Examples and ## When it fails H2 sections.
One callsite per page, after ## Installation.
import data from '@walkeros/web-destination-clarity/walkerOS.json';
import Configuration from '@site/src/components/snippets/_configuration.mdx';
import Quickstart from '@site/src/components/snippets/_quickstart.mdx';
## Installation
<Quickstart data={data} />
<Configuration type="destination" data={data} />
A page without a minimal example (sources, transformers, stores, gtag, sqlite)
shows a <CodeSnippet> install line under ## Installation instead of
<Quickstart>.
type must be one of destination, source, transformer, store. data is
the full walkerOS.json import. The snippet:
- emits
## Configurationwith a short pointer to the shared group-level configuration reference, - emits
## Settingsusing<PropertyTable schema={data.schemas.settings} />when the schema is present, otherwise a "no package-specific settings" note, - emits
## Mappingusing<PropertyTable schema={data.schemas.mapping} />when present, otherwise a pointer to the standard rule fields, - emits
## Examplesonly ifdata.examples.stepholds a public example (publicnotfalse) and renders each public one through<StepExample>, - emits
## When it failswhen a step example has anoutcome, one line per failure example from itsdescription.
Never hand-roll ## Configuration, ## Settings, ## Mapping,
## Examples or ## When it fails on a package page. Place package-specific
content (vendor quirks, advanced usage, platform-specific setup) BELOW the
<Configuration> block with its own sentence-case H2 heading. Fold
## Features bullet lists into the intro paragraph as prose: no ## Features
heading on package pages.
Shared configuration reference (group index pages)
On group index pages (docs/destinations/index.mdx, sources/index.mdx,
transformers/index.mdx, stores/index.mdx), add a ## Configuration section
that documents the wrapper fields around settings: the fields shared across
all packages of that type (consent, mapping, env, id, …).
Drive the table from the core Zod schemas, filtering dev-only fields at the
representation layer. configJsonSchema holds the config under definitions
(its root has no properties), so unwrap it before omitting fields:
import { schemas } from '@walkeros/core/dev';
import { omitSchemaProperties } from '@site/src/utils/schema';
export const destinationConfig = (({ definitions }) =>
omitSchemaProperties({ ...definitions.DestinationConfig, definitions }, [
'settings',
'init',
'mock',
]))(schemas.DestinationSchemas.configJsonSchema);
## Configuration
<PropertyTable schema={destinationConfig} />
Available schemas:
schemas.DestinationSchemas.configJsonSchemaschemas.SourceSchemas.configJsonSchemaschemas.TransformerSchemas.configJsonSchemaschemas.StoreSchemas.configJsonSchema
Use omitSchemaProperties to hide settings (documented per-package) and
dev-only fields such as init and mock from the user-facing table (the
definition names are DestinationConfig, SourceConfig, TransformerConfig
and StoreConfig).
Code display: CodeView vs CodeBox
@walkeros/explorer exports two components for code display:
<CodeBox>renders Monaco, the full editor. Use when the user can edit the code (playgrounds, interactive demos). Ships the IDE.<CodeView>renders Shiki, read-only. Use for static code in docs, marketing pages, and anywhere editing is not a requirement. Same<Box>frame as<CodeBox>(traffic lights, header, copy button, tabs) so visuals match. Zero Monaco runtime cost.
Tokens match because Shiki uses the same TextMate grammars as VS Code / Monaco,
with the matching dark-plus / light-plus themes.
<CodeView> and <CodeSnippet> render their code server-side, so it appears in
the static HTML and the markdown/llms.txt export. Code is shown exactly as
authored (no automatic reformatting). npm run validate:snippets (part of
validate:sources, run in CI) fails on a docs <CodeSnippet> that Prettier
would format differently, and npm run validate:snippets -- --write fixes it in
place. Excerpts Prettier cannot parse (settings: {...}, { ... }
placeholders, a lone opening tag) are only checked for indentation: indent them
by hand, two spaces per level. JSX belongs in language="tsx", not
typescript.
Rule: default to <CodeView>. Only reach for <CodeBox> when the code must
be editable.
PropertyTable component
<PropertyTable schema={...} /> from @walkeros/explorer is the primitive. It
renders a pure table, with no heading and no captions. Never wrap it in a custom
heading on package pages; use the <Configuration> snippet instead.
When NOT to use:
- Pages without package configuration
- Reference tables (Logger API, CLI commands)
- Conceptual explanations
Schema Exports (dev.ts)
Every destination/source should export schemas:
// src/dev.ts
export * as schemas from './schemas';
export * as examples from './examples';
Don't Duplicate
- Link to source files instead of copying type definitions
- Reference
apps/quickstart/examples instead of writing from scratch - Use PropertyTable instead of hardcoded markdown tables
Writing Hints (src/hints.ts)
Hints are the "experienced colleague" layer in walkerOS.json: they tell AI
agents when, why, and what to watch out for beyond what schemas and
examples convey. Surfaced via MCP package_get. Not human-facing docs.
Audience: AI agents configuring packages on behalf of users.
Core Principle: Expand Awareness, Don't Narrow It
Hints should open up the space of what's possible, not prescribe a single path. An LLM reading hints should think "I have more options than I realized", not "I must follow these steps exactly."
Writing Rules
| Rule | Do | Don't |
|---|---|---|
| Describe capabilities | "Supports SA key, ADC, and custom client" | "Use a SA key file when outside GCP" |
| Reference schemas/examples | "See settings.projectId in the schema" | Repeat what the schema description says |
| Explain why behind defaults | "Defaults to EU location; override via location" | "Set location to US" |
| Flag non-obvious interactions | "When snakeCase: true, all data keys transform before send" | Describe obvious behavior |
| Symptoms → causes | "Empty table? Check projectId and dataset existence" | Step-by-step fix instructions |
Key Naming
- kebab-case, group related hints with prefixes:
auth-*,storage-*,query-*,troubleshoot-* - Keep keys descriptive enough to scan:
auth-methodsnota1
When to Add Hints
Most packages don't need hints: schemas and examples cover the common case. Add hints when:
- Multiple auth or config strategies exist and it's non-obvious when to use which
- Non-obvious default behaviors need explaining
- Features interact in ways the schema can't express
- Prerequisites outside walkerOS are required
- Common troubleshooting patterns exist
Accuracy Check
Before publishing hints, verify each claim is factually correct. Don't describe features that aren't implemented. Don't assume behavior; confirm it.
Quality Check
- Each hint expands awareness (describes capabilities, not prescriptions)
- Code snippets reference schema fields, not duplicate them
- No hint restates what a schema
.describe()already says - Troubleshooting hints use symptom → cause format
- Hints are atomic: one concept per hint
- Every claim is factually correct against current implementation
Export Pattern
// src/hints.ts
import type { Hint } from '@walkeros/core';
export const hints: Hint.Hints = {
'auth-methods': {
text: 'Supports three auth methods: ...',
code: [{ lang: 'json', code: '{ "settings": { ... } }' }],
},
};
// src/dev.ts
export * as schemas from './schemas';
export * as examples from './examples';
export { hints } from './hints';
Note: hints is a direct export (not * as), because it's already a
Record<string, Hint>.
Quality Checklist
Structure
- Follows appropriate Divio type (Tutorial/How-To/Reference/Explanation)
- Code example within first 100 words
- First example under 20 lines
- Uses
<details>for advanced content - Has "Next Steps" or "Related" section
Content
- All event names use
"entity action"format with space - Events are pushed as partial events:
elb({ name, data }) - Flow config shown as primary usage pattern
- Examples are complete and copy-pasteable
- Includes imports in code examples
AI Readability
- Clear semantic headers (H2, H3, H4 hierarchy - no skipped levels)
- Tables for structured data
- Links to source of truth TypeScript files
- Static fallback content alongside dynamic components
Consistency
- Uses standard table formats
- Follows package/skill/website templates
- Terminology matches: walkerOS, collector, destination, source
- Headings use sentence case (e.g., "Next steps" not "Next Steps")
-
walkerOS.jsonconvention followed (walkerOS field in package.json, buildDev in tsup)
Templates
Package README Template (thin pointer)
READMEs are thin pointers to the website docs, which are the single source
of truth. A README carries only: logo, title, one-line description, install, ONE
current version: 4 Quick Start, and a prominent link to the package's docs
page. No ## Features, no ## Configuration Reference, no inline API
reference, no ## Examples block. All reference content lives on the website so
it can never drift in two places.
The docs link MUST be an absolute https://www.walkeros.io/docs/... URL, never
a relative path: packages publish to npm individually and relative links break
on npmjs.com. These absolute docs links are validated by
npm run validate:links (it maps the URL to the website/docs/ source file and
asserts it exists).
<p align="left">
<a href="https://www.walkeros.io">
<img alt="walkerOS" title="walkerOS" src="https://www.walkeros.io/img/walkerOS_logo.svg" width="256px"/>
</a>
</p>
# @walkeros/[package-name]
[one-sentence description, adapted from the docs page intro]
[Documentation](https://www.walkeros.io/docs/[path]) •
[NPM Package](https://www.npmjs.com/package/@walkeros/[package-name]) •
[Source Code](https://github.com/elbwalker/walkerOS/tree/main/packages/[path])
## Installation
```bash
npm install @walkeros/[package-name]
```
## Quick start
```json
{
"version": 4,
"flows": {
"default": {
"config": { "platform": "[web|server]" },
"[sources|destinations]": {
"[name]": {
"package": "@walkeros/[package-name]",
"config": { "settings": { "[requiredSetting]": "[value]" } }
}
}
}
}
}
```
## Documentation
Full configuration, mapping, and examples live in the docs:
**https://www.walkeros.io/docs/[path]**
## Contribute
Feel free to contribute by submitting an
[issue](https://github.com/elbwalker/walkerOS/issues), starting a
[discussion](https://github.com/elbwalker/walkerOS/discussions), or getting in
[contact](https://calendly.com/elb-alexander/30min).
## License
MIT
Verify the Quick Start against apps/quickstart/ and the package's real export
name before publishing, and run it through walkeros validate --strict. It
shows the settings the package requires (a destination's examples.minimal):
"config": {} for a package with required settings fails --strict.
Platform-agnostic packages (core, collector) show their most representative
current usage instead of a config.platform flow.
walkerOS.json
The walkerOS field in package.json describes the package's role, and the
build (buildDev) writes it with the schemas, examples and hints into
dist/walkerOS.json:
{
"walkerOS": {
"type": "destination",
"platform": ["web"],
"docs": "https://www.walkeros.io/docs/destinations/web/[name]"
}
}
type is the step kind, platform a platform or a list of them, docs the
docs page URL; a package with several exports adds exports, each export name
mapped to a short description. The README does not repeat it.
Website Doc Template (MDX)
Website docs fold features into the intro paragraph (no ## Features heading).
Thin-pointer READMEs also have no ## Features heading; key capabilities go
into the one-line description. This template is website-only.
---
title: [Title]
description: [SEO description]
sidebar_position: [N]
package: '@walkeros/[package]'
---
import data from '@walkeros/[package]/walkerOS.json';
import Configuration from '@site/src/components/snippets/_configuration.mdx';
import Quickstart from '@site/src/components/snippets/_quickstart.mdx';
# [Title]
<PackageLink package="@walkeros/[package]" />
[1-sentence description that folds in key features as prose; no separate `##
Features` heading on website docs.]
## Installation
<Quickstart data={data} />
{/* Without a minimal example: <CodeSnippet code={`npm install @walkeros/[package]`} language="bash" /> */}
<Configuration type="destination" data={data} />
## Next steps
- [Related guide 1](/docs/...)
Priority Matrix
Issue Classification
| Priority | Criteria | Action |
|---|---|---|
| P0 Critical | Incorrect examples, wrong APIs, security issues | Fix immediately |
| P1 High | Missing PropertyTable, outdated domains, missing sections | Fix soon |
| P2 Medium | Inconsistent terminology, skipped headings | Plan to fix |
| P3 Low | Style issues, minor wording | Backlog |
Non-Negotiables
Punctuation
Never use em dashes (U+2014) or en dashes (U+2013). Use a comma, period, or
rephrase the sentence instead: write "free and without sampling caps" rather
than joining "free" and "without sampling caps" with an em dash. Check a file
with grep -nP "[\x{2013}\x{2014}]" <file>.
Event Naming
Event naming ("entity action", space not underscore): see
walkeros-understanding-events.
Package References
CORRECT: `@walkeros/collector` (with backticks)
WRONG: @walkeros/collector (no backticks)
Domain References
CORRECT: `www.walkeros.io` or relative paths
WRONG: the company's old domain (the doc-lint warns on any mention of it)
Process
For New Package Documentation
- Verify examples exist in
apps/quickstart/or create them first - Write README using template above
- Write website doc using MDX template
- Run quality checklist
- Verify all code examples against Tier 1-3 sources
For Documentation Updates
- Identify issue priority using matrix above
- Check current state against source of truth
- Make minimal changes - don't over-engineer
- Verify examples still compile
- Run quality checklist
Schema naming (.meta({id, title}))
Every exported schema in packages/core/src/schemas/*.ts (and in the *Schema
exports of destination / source / transformer / store packages) must carry
.meta({ id, title, description }) so the generated JSON Schema links back to
the canonical TypeScript name. Without meta, the website's PropertyTable falls
back to __schema0, object, or any.
Convention:
- id: PascalCase, namespace-prefixed so it is globally unique across every
schema in
packages/core/src/schemas. Examples:WalkerOSConsent,DestinationConfig,CollectorPushContext,LoggerConfig. - title: dotted form matching the VS Code TS hover:
WalkerOS.Consent,Destination.Config,Collector.PushContext,Logger.Config. - description: one short sentence. May repeat an existing
.describe()where both are present.
Reference implementation (copy this shape for new schemas):
export const ConsentSchema = z.record(z.string(), z.boolean()).meta({
id: 'WalkerOSConsent',
title: 'WalkerOS.Consent',
description: 'Consent state mapping (group name to granted state).',
});
Coverage is enforced by
packages/core/src/schemas/__tests__/meta-coverage.test.ts. When adding a new
schema, run that test: it fails with the exact schema name if meta is missing.
Destination / source / transformer / store packages that ship their own
SettingsSchema should also call .meta({ id, title }) on it, as the CMP
sources do (id: 'CookiebotSettings').
Related Skills
- walkeros-understanding-flow - Data flow architecture
- walkeros-understanding-events - Event structure and naming
MCP servers 2
Sponsors
Details
- Marketplace
- elbwalker/walkerOS
- Category
- Community