Reproducible Agent Environment: 29 skills for coding standards enforcement, quality gates, repo scaffolding, repo auditing, golden/parity test harnesses, semi-formal code reasoning, ~80% ASD-STE100 technical specifications, GitHub activity review, JAX memory/retracing diagnosis, JAX JIT/fusion hygiene, efficient rendering, ISR MyST scientific paper writing & pre-2026 voice alignment, and domain-specific development (biomechanics, DataJoint, gait analysis, camera models)

version
1.11.0
GitHub stars
5

Install

/plugin install rae@reproducible_agent_environment

Run it inside Claude Code after adding the marketplace with /plugin marketplace add peabody124/reproducible_agent_environment.

Contents

Discovered from source

Skills 27

Overview

Compare the current repository's configuration against the canonical RAE template (templates/pyproject.toml) and report discrepancies. For major structural differences (e.g., no src/ layout in an established repo), ask the user before proposing changes.

This skill does NOT automatically fix everything. It reports what it finds, fixes trivial gaps (missing ruff subsections, missing dev deps), and asks about anything that would be disruptive to change.

Steps

1. Read the Canonical Template

Read templates/pyproject.toml from the RAE plugin directory to get the current source of truth. This template defines the expected configuration for:

  • Build system (hatchling)
  • Python version constraint (>=3.11)
  • Dev dependencies (pytest, pytest-cov, ruff)
  • Ruff config (line-length, lint rules, isort, format)
  • Pytest config (testpaths, pythonpath, addopts, coverage)
  • Coverage config (fail_under = 80)

2. Read the Current Repo

Read the current repository's:

  • pyproject.toml (required — if missing, this is a scaffold-repo situation, not an audit)
  • CLAUDE.md (optional but recommended)
  • Directory structure (src/ vs flat layout)

3. Compare and Categorize

Compare each section and categorize discrepancies:

Auto-fixable (apply without asking):

  • Missing [tool.ruff.lint.isort] section
  • Missing [tool.ruff.format] section
  • Missing [tool.ruff] src key
  • Missing ruff lint rules that are in the template
  • Missing [tool.pytest.ini_options] section
  • Missing [tool.coverage.run] or [tool.coverage.report] sections

Ask the user first:

  • No src/ layout (established repos may have good reasons)
  • Dev dependencies missing from [project.optional-dependencies] when they're installed some other way (e.g., devcontainer, system-level)
  • line-length differs from 120
  • Extra ignore rules not in the template (may be project-specific, e.g., F722 for jaxtyping)
  • Missing CLAUDE.md (offer to create one)
  • Build system differs from hatchling

Informational only (report but don't change):

  • No .github/ CI workflows
  • No .pre-commit-config.yaml (RAE does not require this)
  • Coverage threshold differs from 80 (may be intentionally lower for legacy repos)

4. Report Findings

Present a summary table:

## Audit Results: {repo_name}

### Aligned with RAE
- [x] ruff line-length = 120
- [x] Python >= 3.11
- ...

### Gaps Found
| Gap | Category | Action |
|-----|----------|--------|
| Missing [tool.ruff.format] | Auto-fix | Will add |
| Missing pytest config | Auto-fix | Will add |
| No src/ layout | Ask user | Uses body_models/ directly |
| Extra ignore F722 | Informational | Needed for jaxtyping |

### Recommended additions to CLAUDE.md
- Document F722 exception and why it's needed
- Ensure the **Fail-Fast Field Access & Anti-Defensive-Programming Rule** (no `getattr(obj, "field", None)` or `d.get("field", None)` on known schemas; cite the `getattr(person, "mhr", None)` silent 20-DOF finger-fit bug) is documented in `CLAUDE.md` / `AGENTS.md`
- ...

5. Apply Fixes

  • Apply all auto-fixable changes to pyproject.toml
  • For each "ask user" item, present the discrepancy and ask whether to fix or document as intentional
  • Intentional deviations should be noted in the repo's CLAUDE.md under a ## RAE Deviations section so future audits understand why

6. Verify

After applying fixes:

ruff format --check .
ruff check .

Report any issues from the verification step.

Important Notes

  • The canonical source of truth is always templates/pyproject.toml in the RAE plugin
  • This skill does NOT enforce src/ layout changes on established repos — that's too disruptive
  • Extra ignore rules are often legitimate (F722 for jaxtyping, etc.) — ask, don't remove
  • The goal is convergence over time, not instant compliance

Bead-Driven Development

Overview

This skill orchestrates existing skills with beads integration. It provides prompt refinements rather than reimplementing functionality:

  • writing-plans → create plan + beads for major tasks
  • executing-plans → use subagents + track via beads
  • investigation → create blocking beads on failure, update plan

Announce at start: "I'm using the bead-driven-development skill to orchestrate planning and execution with persistent tracking."

Prerequisites

This skill requires beads and superpowers plugins. Install them first:

Install beads

# Install beads CLI
curl -fsSL https://raw.githubusercontent.com/steveyegge/beads/main/scripts/install.sh | bash

# Install uv (Python package manager)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Add beads marketplace and install plugin
/plugin marketplace add steveyegge/beads
/plugin install beads

Install superpowers

/plugin marketplace add obra/superpowers-marketplace
/plugin install superpowers@superpowers-marketplace

Initialize beads in your repo

bd init

Required skills from superpowers:

  • writing-plans
  • executing-plans
  • investigation (or from ~/.claude/skills/)

Unified Workspace Convention

All phases share one workspace: scratch/{YYYY-MM-DD}-plan-{topic}/

scratch/2026-01-18-plan-auth-system/
├── README.md          ← Plan (Phase 1)
├── scripts/           ← Temp scripts (Phase 2)
└── debug/             ← Investigations (Phase 3)
    └── bd-xxx/

Beads provide the commit history. Plan doesn't need to be committed separately.

Phase 1: Planning with Beads

Invoke writing-plans skill with these additions:

"Create plan in unified workspace following investigation pattern: scratch/{YYYY-MM-DD}-plan-{topic}/README.md Example: scratch/2026-01-18-plan-auth-system/README.md

After finalizing the plan, create a bead for each major task:

bd create 'Task 1: Component name' -t task
bd create 'Task 2: Next component' -t task
bd dep add <task-2-id> <task-1-id> --type blocks

Include bead IDs in the plan file for tracking.

Important: Add a section in the plan noting:

  • Temp scripts go in: scratch/{date}-plan-{topic}/scripts/
  • Debug investigations go in: scratch/{date}-plan-{topic}/debug/
  • Beads provide commit history (plan doesn't need to be committed)

Note: TodoWrite will still be used for fine-grained in-session tracking."

Phase 2: Execution with Beads

Invoke executing-plans skill with these additions:

"Before starting each task:

  1. Run bd ready to find next unblocked task
  2. Mark bead in_progress: bd update <id> --status in_progress

Use TodoWrite as normal for fine-grained step tracking within the task.

Workspace: Put any temporary scripts, test outputs, or debugging artifacts in the plan's scratch directory: scratch/{date}-plan-{topic}/scripts/

After each successful task:

  1. Commit with bead ID: git commit -m 'feat: description (bd-xxx)'
  2. Add commit SHA to bead: bd comment <id> 'Commit: <sha>'
  3. Dispatch code-reviewer subagent (per subagent-driven-development two-stage review)
  4. If review passes: bd close <id> --reason 'Implemented and reviewed'
  5. Sync TodoWrite → bead status when marking task complete

Run bd sync every 2-3 completed tasks to persist to git.

On failure → transition to Phase 3 (Failure Recovery)."

Phase 3: Failure Recovery with Beads

When any task fails, invoke investigation skill with these additions:

Step 1: Create blocking debug bead

bd create 'Debug: <issue description>' -t bug
bd dep add <failed-task-id> <debug-bead-id> --type blocks

Step 2: Investigation in plan's scratch directory Create investigation subfolder: scratch/{date}-plan-{topic}/debug/{bead-id}/README.md

Or if separate investigation needed: scratch/{date}-debug-{bead-id}/README.md

Document findings, hypotheses, and tests in the investigation folder.

Step 3: Resolution Copy key findings summary to debug bead:

bd comment <debug-id> 'Resolution: <summary of fix>'

If plan needs updates: edit scratch/{date}-plan-{topic}/README.md and note changes.

If new tasks discovered:

bd create 'New task from investigation' -t task --discovered-from <debug-id>

Step 4: Resume Close debug bead:

bd close <debug-id> --reason 'Resolved: <summary>'

Original task becomes unblocked. Signal: "Plan updated, resume from Task N with bd ready"

Hybrid Tracking Model

ToolPurposeScope
BeadsMajor milestones/tasksCross-session persistence
TodoWriteFine-grained stepsIn-session tracking

Sync rule: When TodoWrite marks task complete → close corresponding bead.

Quick Reference

CommandPurpose
bd readyFind next unblocked task
bd update <id> --status in_progressMark task started
bd close <id> --reason '...'Mark task complete
bd comment <id> 'message'Add context to bead
bd dep add <id> <blocker> --type blocksAdd dependency
bd syncPersist to git (every 2-3 tasks)
bd show <id>Get task context

Workflow Summary

User: "execute this plan with beads"
           ↓
[bead-driven-development loads]
           ↓
Phase 1: Invoke writing-plans + beads prompt
  → Creates unified workspace
  → Plan in README.md
  → Creates beads for major tasks
  → Adds blocks dependencies
           ↓
Phase 2: Invoke executing-plans + beads prompt
  → Uses bd ready to pick next task
  → Updates bead status + TodoWrite
  → Commits with bead ID: (bd-xxx)
  → Two-stage code review
  → Close bead after review
  → bd sync every 2-3 tasks
           ↓
(On failure) Phase 3: Invoke investigation + beads prompt
  → Creates blocking debug bead
  → Investigates in workspace: debug/{bead-id}/
  → Copies findings to bead notes
  → Updates README.md when resolved
  → Resumes with bd ready

See Also

  • references/prompt-templates.md - Exact copy-paste prompts for each skill
  • references/install-beads.md - Installation instructions when beads is not installed

Camera Model Reference

Overview

Our camera system is implemented in JAX and follows OpenCV conventions. There are two layers:

  1. Core library (multi_camera.analysis.camera) - Low-level functions operating in millimeters
  2. Camera adapter (multiview_biomechanical_pose.geometry.camera_adapter) - High-level functions accepting/returning meters

Source files:

  • MultiCameraTracking/multi_camera/analysis/camera.py - Core camera functions
  • MultiviewBiomechanicalPose/multiview_biomechanical_pose/geometry/camera_adapter.py - Meter-safe wrappers

Camera Parameters Dict

All camera functions expect a single dict with parameters for ALL cameras:

camera_params = {
    "mtx":  array,  # (num_cameras, 4)  - [fx, fy, cx, cy] stored at 1/1000 scale
    "rvec": array,  # (num_cameras, 3)  - Rodrigues rotation vectors (dimensionless)
    "tvec": array,  # (num_cameras, 3)  - Translation vectors in METERS (stored at 1/1000 scale)
    "dist": array,  # (num_cameras, 5+) - Distortion coefficients [k1, k2, p1, p2, k3, ...]
}

The mtx and tvec fields are stored at 1/1000 scale (effectively meters). The library functions get_intrinsic() and get_extrinsic() multiply by 1000 internally to produce pixel-scale intrinsics and mm-scale extrinsics.

TFRecord format uses plural names (camera_intrinsics, camera_rvecs, camera_tvecs, camera_distortion) which the dataset pipeline converts to singular names automatically.


Unit Conventions (CRITICAL)

ContextUnitsNotes
3D points throughout the systemmetersWorld coords, camera coords, joint positions
camera_params["tvec"] storagemeters (1/1000 scale)Scaled up by 1000 inside get_extrinsic()
camera_params["mtx"] storage1/1000 of pixelsScaled up by 1000 inside get_intrinsic()
get_extrinsic() returns4x4 matrix in mmTranslation component is in millimeters
get_intrinsic() returns3x3 K matrix in pixelsfx, fy, cx, cy in pixel units
project_distortion() input3D points in mmYou MUST multiply meters by 1000
project_distortion() output2D points in pixels
triangulate_point() input2D points in pixelsWith optional confidence channel
triangulate_point() output3D points in mmDivide by 1000 to get meters
camera_adapter functionsmeters in/outHandle mm conversion internally
2D keypointspixelsOriginal image resolution

The Conversion Rule

When calling core library functions directly, always convert:

# Meters -> mm before projection
points_2d = cam_lib.project_distortion(camera_params, cam_idx, points_3d_m * 1000.0)

# mm -> meters after triangulation
points_3d_m = triangulate_point(camera_params, points_2d) / 1000.0

When using camera_adapter functions, no conversion is needed - they handle it internally.


OpenCV Extrinsic Convention

Our camera model follows the OpenCV world-to-camera convention:

P_camera = E @ P_world
  • get_extrinsic() returns a 4x4 SE(3) matrix that transforms points FROM world coordinates TO camera coordinates
  • This is NOT the camera position in world space
  • The camera position in world space would be -R^T @ t (or E_inv[:3, 3])
  • Rotation is stored as Rodrigues vectors, converted via SO3.exp(rvec)
  • Uses jaxlie SE3/SO3 for all rigid body transformations
# get_extrinsic returns world-to-camera transform (units: mm)
E = get_extrinsic(camera_params, i)  # 4x4 matrix

# To get camera position in world (in mm):
camera_pos_mm = jnp.linalg.inv(E)[:3, 3]

# To get camera position in meters:
camera_pos_m = camera_pos_mm / 1000.0

COLMAP Conversion Warning

COLMAP stores camera-to-world transforms. When converting from COLMAP:

R_w2c = R_c2w.T
t_w2c = -R_w2c @ t_colmap

Core Library Functions (multi_camera.analysis.camera)

All core functions operate in millimeters. Import as:

from multi_camera.analysis import camera as cam_lib
from multi_camera.analysis.camera import get_intrinsic, get_extrinsic

get_intrinsic(camera_params, i) -> 3x3 K matrix

Returns the intrinsic matrix with focal lengths and principal point in pixel units:

K = get_intrinsic(camera_params, 0)
# [[fx,  0, cx],
#  [ 0, fy, cy],
#  [ 0,  0,  1]]

get_extrinsic(camera_params, i) -> 4x4 SE(3) matrix

Returns the world-to-camera transformation matrix in millimeters:

E = get_extrinsic(camera_params, 0)
# 4x4 homogeneous transform [R|t; 0 0 0 1]
# t is in MILLIMETERS
# This is NOT the camera position - it's a world-to-camera transform

get_projection(camera_params, i) -> 3x4 projection matrix

Returns K @ E[:3] - the full 3x4 projection matrix:

P = get_projection(camera_params, 0)  # 3x4

project_distortion(camera_params, i, points) -> 2D pixels

Projects 3D points to 2D image coordinates with radial distortion:

# CRITICAL: points must be in MILLIMETERS
points_2d = cam_lib.project_distortion(camera_params, cam_idx, points_3d_m * 1000.0)

Currently uses simplified radial model (k1 only). Handles behind-camera points by shrinking them toward image center.

undistort_points(points, K, dist, num_iters=5) -> undistorted 2D

Iteratively removes lens distortion from 2D image points. Supports full OpenCV model (k1-k6, p1-p2, s1-s4, tau_x, tau_y):

undistorted = cam_lib.undistort_points(
    points_2d[None, ...],   # (1, N, 2) - needs batch dim
    K[None, ...],           # (1, 3, 3)
    dist_coeffs[None, :],   # (1, n_coeffs)
)[0]                        # remove batch dim

distort_3d(camera_params, i, points) -> distorted 3D in camera coords

Computes equivalent 3D camera-frame points that would project to the same 2D coordinates under an ideal pinhole camera. Used for rendering 3D models (e.g., SMPL) on distorted images:

# points in mm, returns camera-frame coords in mm
vertices_distorted = cam_lib.distort_3d(camera_params, i, vertices_mm)

Uses full radial+tangential model (k1, k2, k3, p1, p2).

triangulate_point(camera_params, points2d, return_confidence=False) -> 3D mm

DLT triangulation from multiple 2D observations. Handles undistortion internally:

# points2d: (num_cameras, T, J, 3) with [u, v, confidence]
# Returns: (T, J, 3) or (T, J, 4) with confidence, in MILLIMETERS
points_3d_mm = cam_lib.triangulate_point(camera_params, points2d, return_confidence=True)
points_3d_m = points_3d_mm[..., :3] / 1000.0

robust_triangulate_points(camera_params, points2d, sigma=150, threshold=0.5) -> 3D mm

Robust triangulation using geometric median of pairwise triangulations (Roy et al. 2022). Automatically weights cameras by agreement:

points_3d, camera_weights = cam_lib.robust_triangulate_points(
    camera_params, points2d, return_weights=True
)
# points_3d: (T, J, 4) in mm
# camera_weights: (num_cameras, T, J)

reprojection_error(camera_params, points2d, points3d) -> residual in pixels

Computes difference between observed and projected 2D points:

# points3d must be in mm (same space as projection)
residual = cam_lib.reprojection_error(camera_params, points2d, points3d_mm)

Camera Adapter Functions (Meter-Safe API)

These functions accept and return meters. Prefer these over direct core library calls when possible.

from multiview_biomechanical_pose.geometry.camera_adapter import (
    transform_points_world_to_camera,
    transform_points_camera_to_world,
    transform_points_between_cameras,
    project_camera_coordinates,
    unproject_from_image_with_distortion,
    unproject_and_transform_to_world,
    transform_pose_se3_world_to_camera,
    transform_pose_se3_camera_to_world,
    compute_relative_extrinsics,
)

Point Transformations

# World -> Camera (meters in, meters out)
pts_cam = transform_points_world_to_camera(pts_world_m, camera_params, cam_idx)

# Camera -> World (meters in, meters out)
pts_world = transform_points_camera_to_world(pts_cam_m, camera_params, cam_idx)

# Camera A -> Camera B (meters in, meters out)
pts_target = transform_points_between_cameras(pts_source_m, camera_params, source_idx, target_idx)

Projection from Camera Coordinates

# Projects points already in camera frame to 2D (meters in, pixels out)
pts_2d = project_camera_coordinates(pts_cam_m, camera_params, cam_idx)

Uses identity extrinsics internally to avoid double-transformation.

Unprojection

# 2D pixels + depth -> 3D camera coords (pixels+meters in, meters out)
pts_cam = unproject_from_image_with_distortion(pts_2d, depth_m, camera_params, cam_idx)

# 2D pixels + depth -> 3D world coords (pixels+meters in, meters out)
pt_world = unproject_and_transform_to_world(image_point, depth_m, camera_params, cam_idx)

SE(3) Pose Transformations

# Transform full pose (position + quaternion) between frames
pos_cam, quat_cam = transform_pose_se3_world_to_camera(pos_world_m, quat_wxyz, camera_params, cam_idx)
pos_world, quat_world = transform_pose_se3_camera_to_world(pos_cam_m, quat_wxyz, camera_params, cam_idx)

Quaternions use wxyz format (jaxlie convention).


Common Workflows

Project 3D Skeleton to Image

from multi_camera.analysis import camera as cam_lib

# joints_3d: (T, J, 3) in meters
# Project to camera i
joints_2d = cam_lib.project_distortion(camera_params, i, joints_3d * 1000.0)
# joints_2d: (T, J, 2) in pixels

Triangulate from Multi-View 2D Keypoints

# keypoints_2d: (num_cameras, T, J, 3) with [u, v, confidence]
points_3d = cam_lib.triangulate_point(camera_params, keypoints_2d, return_confidence=True)
# points_3d: (T, J, 4) in mm - convert to meters:
points_3d_m = points_3d[..., :3] / 1000.0
confidence = points_3d[..., 3]

Render SMPL Mesh on Distorted Image

from multi_camera.analysis.camera import get_intrinsic, distort_3d

# vertices: (N, 3) in meters
K = np.array(get_intrinsic(camera_params, i))
vertices_distorted_mm = np.array(distort_3d(camera_params, i, vertices * 1000.0))
vertices_distorted_m = vertices_distorted_mm / 1000.0

# Render with ideal pinhole (K, identity R/T) since distortion is baked into 3D

Compute Reprojection Error

from multi_camera.analysis.fit_quality import reprojection_quality

# keypoints3d: (T, J, 3) in mm
# keypoints2d: (N_cams, T, J, 3) with [u, v, confidence]
metrics = reprojection_quality(keypoints3d_mm, camera_params, keypoints2d)

Convert Between Aniposelib and Our Format

# Aniposelib -> our format (mm -> meter storage)
from aniposelib.cameras import CameraGroup
cgroup = CameraGroup.from_names(cam_names)
cam_dicts = cgroup.get_dicts()
camera_params = {
    "mtx": np.array([[c["matrix"][0][0], c["matrix"][1][1],
                       c["matrix"][0][2], c["matrix"][1][2]]
                      for c in cam_dicts]) / 1000.0,
    "dist": np.array([c["distortions"] for c in cam_dicts]),
    "rvec": np.array([c["rotation"] for c in cam_dicts]),
    "tvec": np.array([c["translation"] for c in cam_dicts]) / 1000.0,
}

# Our format -> Aniposelib (meter storage -> mm)
from aniposelib.cameras import Camera, CameraGroup
cameras = []
for i, name in enumerate(cam_names):
    K = np.array([[params["mtx"][i,0]*1000, 0, params["mtx"][i,2]*1000],
                  [0, params["mtx"][i,1]*1000, params["mtx"][i,3]*1000],
                  [0, 0, 1]])
    cam = Camera(name=name, matrix=K, dist=params["dist"][i],
                 rvec=params["rvec"][i], tvec=params["tvec"][i]*1000)
    cameras.append(cam)
cgroup = CameraGroup(cameras)

Distortion Model

Our cameras use the OpenCV radial-tangential distortion model:

  • Projection (project_distortion): Simplified model using k1 only (tangential terms commented out)
  • Unprojection (undistort_points): Full model with k1-k6, p1-p2, s1-s4, tau
  • 3D distortion (distort_3d): Full model with k1, k2, k3, p1, p2

Distortion coefficients are stored in OpenCV order: [k1, k2, p1, p2, k3, ...]

For EgoHumans data: images are pre-undistorted and distortion is set to [0, 0, 0, 0, 0].


Anti-Patterns (NEVER DO THESE)

1. Never manually access camera parameter arrays

# WRONG - bypasses scaling and formatting
fx = camera_params["mtx"][0, 0]
R = cv2.Rodrigues(camera_params["rvec"][0])[0]
t = camera_params["tvec"][0]
# CORRECT - use library functions
K = get_intrinsic(camera_params, 0)        # Properly scaled 3x3
E = get_extrinsic(camera_params, 0)        # Properly scaled 4x4 SE(3)

2. Never create single-camera parameter dicts

# WRONG - breaks the API contract
cam0_params = {
    "mtx": camera_params["mtx"][0:1],
    "rvec": camera_params["rvec"][0:1],
    "tvec": camera_params["tvec"][0:1],
    "dist": camera_params["dist"][0:1],
}
# CORRECT - pass full dict with camera index
result = cam_lib.project_distortion(camera_params, 0, points)

3. Never forget the mm conversion for core library functions

# WRONG - passing meters to mm-expecting function
projected = cam_lib.project_distortion(camera_params, i, points_3d_m)
# CORRECT
projected = cam_lib.project_distortion(camera_params, i, points_3d_m * 1000.0)

4. Never reimplement geometric transforms

# WRONG - manual projection ignoring distortion
x_2d = fx * (x_3d / z_3d) + cx
y_2d = fy * (y_3d / z_3d) + cy
# CORRECT - use library with distortion
from multiview_biomechanical_pose.geometry.camera_adapter import project_camera_coordinates
pts_2d = project_camera_coordinates(pts_cam_m, camera_params, cam_idx)

5. Never manually unproject without distortion correction

# WRONG - ignores lens distortion
x_3d = (x_2d - cx) * depth / fx
# CORRECT
from multiview_biomechanical_pose.geometry.camera_adapter import unproject_from_image_with_distortion
pts_3d = unproject_from_image_with_distortion(pts_2d, depth_m, camera_params, cam_idx)

6. Never confuse extrinsic matrix with camera position

# WRONG - treating tvec as camera position in world
camera_position = camera_params["tvec"][i]
# CORRECT - extrinsic is world-to-camera, invert to get camera position
E = get_extrinsic(camera_params, i)
camera_pos_mm = jnp.linalg.inv(E)[:3, 3]
camera_pos_m = camera_pos_mm / 1000.0

Quick Reference: Import Paths

# Core camera library
from multi_camera.analysis import camera as cam_lib
from multi_camera.analysis.camera import (
    get_intrinsic,          # camera_params, i -> 3x3 K (pixels)
    get_extrinsic,          # camera_params, i -> 4x4 E (mm, world-to-camera)
    get_projection,         # camera_params, i -> 3x4 P (mm)
    project_distortion,     # camera_params, i, pts_mm -> 2D pixels
    undistort_points,       # pts_px, K, dist -> undistorted_px
    distort_3d,             # camera_params, i, pts_mm -> distorted 3D mm
    triangulate_point,      # camera_params, pts_2d -> 3D mm
    robust_triangulate_points,  # camera_params, pts_2d -> 3D mm + weights
    reprojection_error,     # camera_params, pts_2d, pts_3d_mm -> residual px
)

# High-level meter-safe adapter
from multiview_biomechanical_pose.geometry.camera_adapter import (
    transform_points_world_to_camera,       # pts_m -> pts_m (camera frame)
    transform_points_camera_to_world,       # pts_m -> pts_m (world frame)
    transform_points_between_cameras,       # pts_m -> pts_m (target camera)
    project_camera_coordinates,             # pts_cam_m -> 2D pixels
    unproject_from_image_with_distortion,   # 2D px + depth_m -> 3D cam_m
    unproject_and_transform_to_world,       # 2D px + depth_m -> 3D world_m
    transform_pose_se3_world_to_camera,     # pos_m, quat -> pos_m, quat
    transform_pose_se3_camera_to_world,     # pos_m, quat -> pos_m, quat
    compute_relative_extrinsics,            # -> 4x4 relative transform (mm)
)

# Fit quality metrics
from multi_camera.analysis.fit_quality import reprojection_quality

Common Mistakes

MistakeFix
Passing meters to project_distortionMultiply by 1000 first
Treating get_extrinsic() translation as camera positionIt's world-to-camera; invert for camera position
Accessing camera_params["mtx"] raw valuesUse get_intrinsic() which applies 1000x scaling
Creating per-camera param dictsPass full dict + camera index
Ignoring distortion in projection/unprojectionAlways use distortion-aware functions
Assuming triangulate_point returns metersIt returns mm - divide by 1000
Forgetting batch dims for undistort_pointsNeeds (batch, N, 2) input shape
Using COLMAP extrinsics directlyCOLMAP is camera-to-world; ours is world-to-camera

Overview

When you discover a better pattern, configuration, or guideline during work, this skill guides you through proposing it upstream to the Reproducible Agent Environment repository.

Parameters

  • improvement_type (optional): "guideline", "skill", "config", or "template"
  • description (required): Brief description of the improvement

Steps

1. Evaluate Scope

Determine if the improvement is universal or project-specific.

Constraints:

  • You MUST verify the improvement works in the current project
  • You MUST check if a similar pattern already exists upstream
  • You SHOULD consider if the improvement applies broadly or only to specific domains

Decision:

  • IF universal → Proceed to step 2
  • IF project-specific → Add to local overrides with comment explaining why, then STOP

2. Document the Improvement

Prepare a clear description of the change.

Constraints:

  • You MUST include before/after examples
  • You MUST explain the rationale (why is this better?)
  • You SHOULD reference any research or sources that informed the improvement
  • You MUST specify which file(s) would be modified

3. Create Upstream PR

Constraints:

  • You MUST create a branch with descriptive name (e.g., improve/ruff-config, guideline/async-patterns)
  • You MUST write a clear commit message explaining the improvement
  • You SHOULD include test cases or examples if applicable
  • You MUST open PR with the documented rationale

4. Update Local

After PR is merged:

  • Update the RAE plugin to pull the improvement
  • Verify it works correctly in your project

See references/propose-upstream.md for the detailed upstream contribution workflow.

Examples

User: "I found a better ruff configuration for catching common issues" Agent: Evaluates if it's universal (yes), documents the change with before/after examples, creates PR to update templates/pyproject.toml.

User: "/config-improvement guideline 'Add guidance on async/await patterns'" Agent: Checks if async guidance exists (no), drafts new section for python-standards.md, creates PR with examples.

Overview

Read and internalize the coding guidelines relevant to the current task. Guidelines are bundled as reference files in the enforce-guidelines skill at enforce-guidelines/references/.

Parameters

  • task_type (optional): "python", "git", "refactor", "debug", "review", or "all"

Steps

1. Determine Relevant Guidelines

Based on task type, identify which guidelines to review from enforce-guidelines/references/:

Task TypeGuidelines to Read
pythoncoding-standards.md, python-standards.md
gitgit-workflow.md, pre-commit-checklist.md
refactorcoding-standards.md, anti-patterns.md
debugcoding-standards.md (TDD section)
reviewanti-patterns.md, coding-standards.md
new-reporepo-structure.md, python-standards.md
allAll guidelines

2. Summarize Key Points

Extract the 3-5 most relevant rules for the current task.

  • You MUST highlight any MUST/MUST NOT constraints
  • You SHOULD note any task-specific gotchas

3. Apply During Work

Keep these guidelines in mind while executing the task. Reference them when making decisions.

  • You MUST cite the relevant guideline when it influences a decision
  • You SHOULD flag if you discover a case not covered by guidelines

Examples

User: "I'm about to refactor the auth module" Agent: Reads coding-standards.md and anti-patterns.md, summarizes DRY principles and slop patterns to avoid, proceeds with refactor while citing guidelines.

User: "/consult-guidelines python" Agent: Reviews coding-standards.md and python-standards.md, extracts key rules about ruff, type hints, and path management.

DataJoint Biomechanics Schema Reference

CRITICAL: NEVER Modify Database Entries

DO NOT update, delete, or alter any DataJoint database entries. This includes:

  • update1(), delete(), drop() on any table
  • Modifying settings lookup tables (KinematicReconstructionSettingsLookup, ProbabilisticReconstructionSettingsLookup, KineticReconstructionSettingsLookup, KeypointSet, etc.)
  • Altering any computed table entries

Database entries are shared state used by the entire lab. Changing a settings entry changes it for everyone and invalidates prior results computed with those settings.

Overview

The biomechanics pipeline uses DataJoint across four repositories with two complementary data collection approaches:

  1. Multi-Camera (MMC) - Lab-based with calibrated camera rigs → MultiCameraTracking + BodyModels/kinematic_dj.py
  2. Portable/Monocular (PBL) - Smartphone videos with optional IMU → PortableBiomechanicsSessions + BodyModels/monocular_dj.py

Both produce the same output format: qpos, qvel, joints, sites from MuJoCo body models.


DataJoint Query Basics

Operators

OperatorMeaningExample
&Restrict (filter)(Table & {'field': value})
*Join tables(Table1 * Table2)
-Set difference(Table1 - Table2) finds rows NOT in Table2
.proj()Select attributesTable.proj('field1', 'field2')

Fetch Patterns

# fetch1() - Exactly one row, returns tuple or dict
timestamps, qpos = (KinematicReconstruction.Trial & key).fetch1('timestamps', 'qpos')

# fetch() - Multiple rows, returns arrays
all_keys = (Session & filter).fetch('KEY')
keypoints = (TopDownPerson & key).fetch('keypoints')

# With ordering (critical for multi-camera consistency)
kp, names = (Table & key).fetch('keypoints', 'camera_name', order_by='camera_name')

Filtering by Project and Participant

video_project Field

video_project is a categorical field (varchar(50)) that identifies the study/cohort. It lives in MultiCameraRecording and Video tables.

Common projects:

ProjectDescription
GAIT_CONTROLSHealthy control gait data
PROSTHETIC_GAITProsthetic gait subjects
CM_GAITCervical myelopathy gait
CLINIC_GAITClinical gait assessment
PEDIATRIC_GAITPediatric subjects
ASB2024ASB conference dataset

Query Patterns by Project

# List all unique projects
projects = np.unique((MultiCameraRecording).fetch("video_project"))

# Find all participants in a project
participants = (Session & (Recording & (MultiCameraRecording & 'video_project="GAIT_CONTROLS"'))).fetch("participant_id")

# Filter trials by project
trials = KinematicReconstruction.Trial & (Recording & (MultiCameraRecording & f"video_project='{project}'"))

# Multiple projects with IN clause
filt = 'video_project in ("CLINIC_GAIT", "GAIT_CONTROLS", "PROSTHETIC_GAIT")'
sessions = Session & (Recording * MultiCameraRecording & filt)

Participant Naming Conventions

Project TypeFormatExamples
ASB studiesASB_###ASB_001, ASB_022
Numeric IDsintegers72, 504, 127
Special codesmappedTF01, TF02, TF47

Session → Video Path (via video_project)

Session (participant_id, session_date)
    → Recording → MultiCameraRecording (video_project)
        → SingleCameraVideo → Video (video_project, filename)

Key Output Fields Reference

FieldTableShapeDescription
qposKinematicReconstruction.Trial(T, 41)Joint angles in generalized coordinates (radians)
qvelKinematicReconstruction.Trial(T, 41)Joint velocities (rad/s)
jointsKinematicReconstruction.Trial(T, N_joints, 3)3D joint positions (mm)
sitesKinematicReconstruction.Trial(T, N_sites, 3)3D anatomical markers (mm)
keypointsTopDownPerson(T, N_kpts, 3)2D keypoints [x, y, conf]
keypoints3dPersonKeypointReconstruction(T, N_kpts, 4)3D keypoints [x, y, z, conf] (mm)
keypoints_3dLiftingPerson(T, N_kpts, 4)Lifted 3D from monocular [x, y, z, conf]
timestampsVideoInfolist[datetime]Absolute frame times
delta_timeVideoInfo(T,)Seconds from first frame
fpsVideoInfofloatFrames per second

Common Access Patterns

# Get kinematic reconstruction (ALWAYS specify method!)
qpos, sites = (KinematicReconstruction.Trial & key &
               {'kinematic_reconstruction_settings_num': 137}).fetch1('qpos', 'sites')

# Get 3D triangulated keypoints
kp3d = (PersonKeypointReconstruction & key).fetch1('keypoints3d')

For 2D keypoints synchronized with KR qpos, see the Temporal Synchronization section below. Do NOT use (TopDownPerson * VideoInfo & key).fetch1('timestamps', 'keypoints') — this produces misaligned frames.


Temporal Synchronization (CRITICAL)

KR qpos and 2D keypoints have independently managed timestamps. Naive matching silently corrupts frame alignment.

Correct approach: Use fetch_keypoints(only_detected=True) from BodyModels:

from body_models.datajoint.dataset import fetch_keypoints as bm_fetch_keypoints

# Returns keypoints in 1:1 frame correspondence with KR qpos
timestamps, keypoints_raw = bm_fetch_keypoints(trial_key, only_detected=True)
# keypoints_raw: (C, T, 87, 3) — (x, y, confidence)

# ALWAYS verify correspondence
assert qpos.shape[0] == keypoints_raw.shape[1]

Why direct TopDownPerson/VideoInfo fetch is broken: KR timestamps are zeroed relative to first detection; VideoInfo timestamps are zeroed relative to video start. When first detection occurs at frame N>0, matching on timestamp magnitude pairs the wrong frames (N-frame offset).

Camera parameter reordering: KinematicReconstruction.updated_calibration uses Calibration table order; keypoints use alphabetical SingleCameraVideo order. These differ — reorder explicitly before projection.

See rae:fetching-synchronized-data for the complete pattern with camera reordering and contiguous segment selection.


Two Data Collection Approaches

1. Multi-Camera (MMC) → KinematicReconstruction

Key space: (participant_id, session_date) bridged to (recording_timestamps, camera_config_hash)

Session → Recording → MultiCameraRecording → SingleCameraVideo → Video
                   ↘ SessionCalibration.Grouping → Calibration
                                     ↓
Video → TopDownPerson → PersonKeypointReconstruction (3D triangulation)
                                     ↓
                   KinematicReconstruction.Trial (qpos, qvel, joints, sites)

Bridge Tables:

  • Recording - Links Session → MultiCameraRecording
  • SingleCameraVideo - Links MultiCameraRecording → Video
  • SessionCalibration.Grouping - Links Session → Calibration (required for KinematicReconstruction)

Use when: Lab-based data, multiple synchronized cameras, highest accuracy needed.

2. Portable/Monocular (PBL) → MonocularReconstruction

Key space: (subject_id, session_start_time) via Firebase

Subject → Session → Session.FirebaseSessionInfo
                              ↓
                    FirebaseSession (Computed)
                              ↓
            ┌─────────────────┼─────────────────┐
       AppVideo           Attitude/Gyro      PhoneAttitude
     (→ Video)            (IMU sensors)      (phone quaternion)
            ↓
    TopDownPerson → LiftingPerson (3D lifting)
                              ↓
            MonocularReconstruction.Trial (qpos, qvel, joints, sites, rnc)

Additional output: rnc (T, 3) - camera rotation vector from phone attitude

Use when: Smartphone videos, remote/home monitoring, wearable IMU/EMG data.

Comparison

AspectMulti-CameraMonocular
InputMultiple synced camerasSingle smartphone camera
3D MethodTriangulation2D→3D lifting
Accuracy~10-15mm, ~2-5°~15-20mm, ~5-10°
qpos DOF4140
Extra dataupdated_calibrationrnc (phone rotation), IMU/EMG
Schemakinematic_dj.pymonocular_dj.py

Method Numbers by Pipeline

CRITICAL: Always specify the method number in queries to avoid duplicates.

PipelineTableMethod FieldDefault Method
Multi-CameraKinematicReconstructionkinematic_reconstruction_settings_num137
MonocularMonocularReconstructionmonocular_reconstruction_settings_num1

Multi-Camera: Method 137

# CORRECT - Always specify method for KinematicReconstruction
key = {'participant_id': '102', 'session_date': date(2023, 7, 21),
       'kinematic_reconstruction_settings_num': 137}  # CRITICAL!

# WRONG - Returns BOTH 130 and 137 for dual-processed participants
key = {'participant_id': '102', 'session_date': date(2023, 7, 21)}

Dual-processed participants (have both 130 and 137): ASB_022, ASB_007, ASB_008, ASB_020, ASB_041, ASB_071, ASB_081, ASB_083, ASB_085, ASB_095, ASB_125, ASB_130

Method 137 config: humanoid_torque_rl.xml, floor leveling enabled, 40k iterations

Monocular: Method 1

# For MonocularReconstruction queries
key = {'subject_id': 123, 'session_start_time': ...,
       'monocular_reconstruction_settings_num': 1}

Bridging Algorithm

Bottom-up MeTRAbs detection inserted into top-down tables:

MeTRAbs (580 keypoints) → BottomUpBridging → BottomUpBridgingPerson
    ↓ filter_skeleton("bml_movi_87") → indices 264-350
TopDownPerson (method="Bridging_bml_movi_87")
    ↓ re-fetch from BottomUpBridging (not TopDownPerson!)
LiftingPerson (method="Bridging_bml_movi_87")

Why "weird": LiftingPerson normally uses TopDownPerson, but for bridging it re-fetches directly from BottomUpBridging.


Quick Reference: Key Files

TaskFile
Video/VideoInfo/TopDownPersonPosePipeline/pose_pipeline/pipeline.py
Bridging algorithmPosePipeline/pose_pipeline/wrappers/bridging.py
MultiCameraRecording/SingleCameraVideoMultiCameraTracking/multi_camera/datajoint/multi_camera_dj.py
Session/Recording (MMC)MultiCameraTracking/multi_camera/datajoint/sessions.py
SessionCalibrationMultiCameraTracking/multi_camera/datajoint/session_calibrations.py
PersonKeypointReconstructionMultiCameraTracking/multi_camera/datajoint/multi_camera_dj.py
KinematicReconstructionBodyModels/body_models/datajoint/kinematic_dj.py
MonocularReconstructionBodyModels/body_models/datajoint/monocular_dj.py
FirebaseSession/AppVideo (PBL)PortableBiomechanicsSessions/portable_biomechanics_sessions/emgimu_session.py
PBL↔MMC linkagePortableBiomechanicsSessions/portable_biomechanics_sessions/mmc_linkage.py
DataJoint principlesPipelineOrchestrator/docs/datajoint_principles.md

Common Mistakes

  1. Missing method number - Queries without kinematic_reconstruction_settings_num=137 return duplicates
  2. Wrong key space - Using (video_project, filename) when you need (participant_id, session_date)
  3. Assuming LiftingPerson uses TopDownPerson - For bridging methods, it re-fetches from BottomUpBridging
  4. Missing SessionCalibration.Grouping - KinematicReconstruction requires this bridge
  5. Forgetting order_by - Multi-camera queries need order_by='camera_name' for consistent ordering
  6. Confusing MonocularReconstruction vs KinematicReconstruction - Monocular uses FirebaseSession.AppVideo, Kinematic uses SessionCalibration.Recordings
  7. Not filtering by video_project - Queries across all projects mix different populations; always filter to your target cohort
  8. Matching KR timestamps with VideoInfo timestamps - Use fetch_keypoints(only_detected=True) instead; see Temporal Synchronization section
  9. Assuming camera_params order matches keypoint order - Calibration order != SingleCameraVideo alphabetical order; reorder explicitly
  10. Zeroing timestamps manually (ts - ts[0]) - Destroys physical time correspondence between data sources

Review the diff against main and remove all AI-generated slop introduced in the outstanding code changes, most recent commit, or branch based on user instructions.

Refer to the anti-patterns guide in enforce-guidelines/references/anti-patterns.md for the full list of slop patterns to catch.

Common slop to remove:

  • Extra comments that a human wouldn't add or that are inconsistent with the rest of the file
  • Defensive getattr(obj, "field", None) or d.get("field", None) on known/expected schemas: Replace with direct fail-fast field/key access (obj.field or d["field"]). Never allow getattr(..., None) to silently evaluate a misspelled attribute to None (as in the getattr(person, "mhr", None) bug that silently disabled the 20-DOF finger fit on every frame).
  • Extra defensive checks or try/catch blocks that are abnormal for that area of the codebase (especially if called by trusted / validated codepaths)
  • Casts to Any to get around type issues
  • Placeholder patterns (TODOs without context, "Phase 1" comments, commented-out code)
  • Over-engineering (abstractions for single-use code, unnecessary factories)
  • Any other style that is inconsistent with the file

Report at the end with only a 1-3 sentence summary of what you changed.

Going further: prevent over-engineering at write-time

deslop is a corrective pass — it scrubs slop out of changes already made. To stop over-engineering before it is written, the third-party ponytail skills (https://github.com/DietrichGebert/ponytail, MIT) are complementary:

  • ponytail — a write-time mode that enforces minimal/YAGNI code (stdlib and native features before custom code, no unrequested abstractions).
  • ponytail-review / ponytail-audit — report-only over-engineering passes on a diff or the whole repo. Narrower than deslop (complexity/LOC only, no type-evasion or DB-mutation checks) and they list findings rather than applying them.

To adopt: either copy the pertinent skills/<name>/SKILL.md folders from that repo into your skills directory (no hooks needed — the skills work standalone), or install the full plugin from the repo for the activation hooks and /ponytail mode switching as well.

Efficient Rendering

Performance and correctness patterns for rendering meshes, point clouds, and overlays — offline batches and differentiable in-the-loop rendering alike. The two recurring failure modes are (1) silently falling back to a CPU software GL rasterizer (10–50× slower) and (2) renders that don't line up with the real camera because a convention was dropped. Most of this skill is about avoiding those two.

This skill is about how to render fast and correctly. For the camera math itself (projection, triangulation, unit conventions), see the camera-model skill — this one assumes you already have intrinsics K, an extrinsic world→camera transform, and distortion coefficients.


The golden rules

  1. Pin a real GPU for EGL before importing any GL library, and verify you didn't land on llvmpipe. This is the single biggest win.
  2. Create the renderer/context once per resolution and reuse it across every frame at that size. Context creation is ~60 ms (Mesa/EGL path); per-frame work should be single-digit ms.
  3. Pick one distortion strategy and apply it consistently — either undistort the image and render with a pinhole, or keep the distorted image and bake distortion into the 3D vertices. Never half of each.
  4. Keep one coordinate convention end to end (OpenCV: +X right, +Y down, +Z forward). Convert to GL exactly once, at the camera node.
  5. Composite only the dirty region (mesh bounding box), not the whole frame.
  6. Precompute everything pose-independent once (skinning weights, normals topology, projection matrices, undistort remaps, camera frusta/bounds).
  7. For JAX rasterizers, chunk pixels and jax.checkpoint so peak memory is chunk × primitives, not H·W × primitives.

1. Headless GPU rendering with EGL

Offscreen rendering (no X server) goes through EGL. The trap: if the NVIDIA EGL vendor library isn't selected, GL silently falls back to Mesa's llvmpipe CPU rasterizer and everything still "works" — just 10–50× slower.

Set the platform before importing OpenGL/pyrender/anything that pulls in GL:

import os
os.environ.setdefault("PYOPENGL_PLATFORM", "egl")  # MUST precede any GL import
# only now:
import pyrender

MuJoCo uses its own backend switch. When rendering MuJoCo geometry (e.g. mujoco.Renderer), MuJoCo reads MUJOCO_GL, not PYOPENGL_PLATFORM. Set MUJOCO_GL=egl (the unset/glfw default needs a display and dies headless; osmesa is the CPU path). Set both before any import so a process that mixes a MuJoCo renderer and a pyrender overlay routes both through hardware EGL:

os.environ.setdefault("MUJOCO_GL", "egl")
os.environ.setdefault("PYOPENGL_PLATFORM", "egl")

Setting the env var alone does not pin a GPU — see the next paragraph. The llvmpipe verification below is just as necessary for the MuJoCo path.

Pin a GPU — but do not assume the EGL device index equals the CUDA index. EGL device order does not match nvidia-smi / CUDA_VISIBLE_DEVICES order, and there is no reliable EGL→card mapping. So don't set EGL_DEVICE_ID to a CUDA index and trust it — that's the classic way to land on the wrong card or on llvmpipe. Instead enumerate EGL devices, build a throwaway context on each, read GL_RENDERER, and pick a hardware GPU (rank by free memory from nvidia-smi if you want the emptiest one):

from OpenGL import EGL
devices = EGL.eglQueryDevicesEXT(...)        # enumerate
for d in devices:
    # create a context on `d`, read GL_RENDERER, keep the first non-llvmpipe one
    ...
os.environ["EGL_DEVICE_ID"] = str(chosen_egl_index)  # only after probing/verifying
# If GLVND can't find the NVIDIA ICD, point it at the vendor JSON:
#   os.environ["__EGL_VENDOR_LIBRARY_FILENAMES"] = "/path/to/10_nvidia.json"

If you must honor a pre-set EGL_DEVICE_ID, still validate it resolves to real hardware (the GL_RENDERER assert below) rather than trusting the number.

NVIDIA Devcontainer Setup (nvidia/cuda:*-ubuntu24.04 gotchas):

  1. NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics: nvidia/cuda base images default to compute,utility. Unless "-e", "NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics" is in devcontainer.json runArgs (and containerEnv), nvidia-container-cli only mounts CUDA compute libraries and omits libEGL_nvidia.so.0, libGLX_nvidia.so.0, and libnvidia-rtcore.so.
  2. Missing 10_nvidia.json and nvidia_icd.json: nvidia/cuda Ubuntu 24.04 images do not include GLVND or Vulkan JSON manifests, and apt-get install libgl1 libegl1 libvulkan1 only registers Mesa (50_mesa.json and lvp_icd.json). Ensure /usr/share/glvnd/egl_vendor.d/10_nvidia.json ("library_path": "libEGL_nvidia.so.0") and /etc/vulkan/icd.d/nvidia_icd.json ("library_path": "libGLX_nvidia.so.0") exist (created in templates/devcontainer-gpu/devcontainer.json onCreateCommand), or set __EGL_VENDOR_LIBRARY_FILENAMES / VK_ICD_FILENAMES at runtime.
  3. Manual --device=/dev/nvidia* filtering and Vulkan/RTX: If devcontainer.json runArgs manually lists --device=/dev/nvidia0, --device=/dev/nvidiactl, --device=/dev/nvidia-uvm (e.g. to skip unhealthy GPUs), you must also include --device=/dev/nvidia-modeset and --device=/dev/dri. Without them, CUDA and EGL pbuffers work, but Vulkan (vkCreateDevice, Isaac Sim omni.rtx, SAPIEN, ManiSkill) segfaults or fails with No device could be created.

Always verify you got the GPU, not the CPU fallback. Do this once at startup and fail loudly:

r = pyrender.OffscreenRenderer(W, H)
import OpenGL.GL as gl
renderer_str = gl.glGetString(gl.GL_RENDERER).decode()
assert "llvmpipe" not in renderer_str.lower(), f"CPU GL fallback: {renderer_str}"

Typical impact at ~2K resolution / ~18k-vertex mesh: llvmpipe ≈ 800 ms/frame, NVIDIA EGL ≈ 15 ms/frame.

One process, one context. EGL contexts cannot be shared across processes. If you parallelize across processes, each worker needs its own context (and must not have CUDA/JAX forked into it — use multiprocessing.get_context("spawn"), never fork, once CUDA is initialized). For most workloads a single process with a reused context beats multiprocessing.


2. Reuse the GL context and scene

Allocate the OffscreenRenderer (plus Scene, camera node, lights) once per viewport size and cache it. Re-pose the geometry each frame; never reallocate the renderer.

_RENDER_CACHE: dict[tuple[int, int], dict] = {}

def get_context(h: int, w: int) -> dict:
    key = (int(h), int(w))
    if key not in _RENDER_CACHE:
        renderer = pyrender.OffscreenRenderer(viewport_width=w, viewport_height=h)
        scene = pyrender.Scene(bg_color=[0, 0, 0, 0])  # transparent for compositing
        _RENDER_CACHE[key] = {"renderer": renderer, "scene": scene}
    return _RENDER_CACHE[key]

import atexit
atexit.register(lambda: [c["renderer"].delete() for c in _RENDER_CACHE.values()])

The cached Scene, camera, and lights persist; only the mesh churns. Add the posed mesh, render, then remove its node in a finally so the next frame starts clean without rebuilding the scene:

node = ctx["scene"].add(mesh)
try:
    color, depth = ctx["renderer"].render(ctx["scene"], flags=pyrender.RenderFlags.RGBA)
finally:
    ctx["scene"].remove_node(node)

Keep video captures open too — a VideoCapture held across frames reads sequentially in O(1); reopening or seeking per frame is a large hidden cost.


3. Camera handling for rendering

Getting the render to land on the right pixels is mostly about three conversions: OpenCV→GL axis flip, intrinsics→projection, and a single decision about distortion.

3a. OpenCV → GL axis convention

OpenCV cameras look down +Z with +Y down; OpenGL cameras look down −Z with +Y up. The camera pose (camera→world) for a GL renderer is the inverse of your world→camera extrinsic, post-multiplied by a flip:

GL_FLIP = np.diag([1.0, -1.0, -1.0, 1.0])   # OpenCV -> OpenGL
cam_pose = np.linalg.inv(extrinsic_w2c) @ GL_FLIP   # what pyrender wants

Apply this flip in exactly one place. If your overlay is upside-down or mirrored, this is almost always the culprit — don't compensate elsewhere.

3b. Intrinsics → projection

Use an intrinsics camera directly rather than hand-rolling a projection matrix; it handles the principal-point offset correctly:

camera = pyrender.IntrinsicsCamera(fx=fx, fy=fy, cx=cx, cy=cy,
                                   znear=0.05, zfar=100.0)

Set znear/zfar to bracket your actual scene depth — too wide wastes z-buffer precision (z-fighting), too tight clips geometry. For a self-contained object, deriving the frustum from the vertex bounds once per sequence (not per frame) both fixes clipping and avoids recomputation.

Watch your units against the frustum. A zfar that's fine in meters will clip the whole subject to a blank frame if the mesh is in centimeters — keep znear/zfar/depth_tolerance in the same unit as your geometry.

For a canonical "look at the object" view (no real camera), an orthographic camera sized to the vertex extent is robust: set xmag = ymag = extent, stand the camera off at center.z + extent·3, and zfar = extent·10 so the standoff plus body depth always fits. A fixed small zfar here is the usual cause of a blank render.

When projecting points yourself, use the OpenCV pinhole with a safe-z guard and pixel centers (the 0.5 offset matters for alignment with sampled images):

z = jnp.maximum(verts_cam[:, 2], 1e-3)
u = fx * verts_cam[:, 0] / z + cx
v = fy * verts_cam[:, 1] / z + cy
# pixel grid for sampling: xs + 0.5, ys + 0.5

3c. Distortion — pick ONE of two strategies

Real lenses have radial/tangential distortion. There are two correct ways to make a render line up with a distorted photo; mixing them produces a subtle, maddening misalignment near the image edges.

Strategy A — undistort the image (preferred when you control the footage). Undistort the video frame once, then render and project with a plain pinhole. Use the original K as the new camera matrix so the projection model is unchanged (do not use getOptimalNewCameraMatrix, which shifts the image plane and breaks alignment with your K):

map1, map2 = cv2.initUndistortRectifyMap(K, dist, np.eye(3), K, (w, h), cv2.CV_32FC1)
# precompute once; reuse for every frame:
frame_undist = cv2.remap(frame, map1, map2, cv2.INTER_LINEAR)
# now render + project with plain pinhole K, no distortion anywhere

Likewise undistort 2D keypoints before unprojecting/triangulating. This keeps all 3D math pinhole-simple and is the right call for differentiable rendering.

Strategy B — bake distortion into the vertices (when you must keep the original image). Transform the 3D vertices in camera space into the "pre-distorted" positions that, under an ideal pinhole, land where the real lens would put them. Then render with a plain pinhole onto the untouched distorted image:

verts_distorted_cam = distort_3d(camera_params, cam_idx, verts_cam_mm)  # see camera-model
# render with pinhole K, identity extrinsic, over the original distorted frame

Use B when the deliverable must overlay the raw footage (e.g. you can't ship undistorted video) or distortion is strong. Use A everywhere else — it's simpler and differentiable. Whichever you choose, apply it to both the image and the geometry, never one without the other.


4. Mesh rendering

  • Skip geometry validation when you trust your topology: trimesh.Trimesh(v, f, process=False) avoids a surprisingly expensive cleanup pass per frame.
  • Batch all meshes into one scene before a single render() so the shared z-buffer resolves inter-mesh occlusion correctly in one pass — don't render bodies separately and composite, or near/far occlusion will be wrong.
  • Materials: MetallicRoughnessMaterial(metallicFactor=0, roughnessFactor≈0.7) with modest ambient (~0.35) and a camera-mounted directional light reads well for anatomy/body meshes. Render RGBA so you get an alpha mask for free.
  • Per-vertex color / heatmaps: render with the FLAT flag to skip lighting so the raw vertex colors survive (error maps, region labels, confidence).
  • Translucent over opaque: for a skin shell over bones, render bones OPAQUE and skin BLEND with SKIP_CULL_FACES so the back of the semi-transparent shell isn't culled.
  • Backface culling for a single closed opaque shell: drop faces whose normal points away from the camera (n · view_dir ≥ 0) — fewer triangles, identical result.
  • Layered translucency, exactly: rasterize each layer opaquely (its own z-buffer fixes self-occlusion) and composite back-to-front in numpy (painter's order). More predictable than GL depth-sorted blending.
  • Precompute skinning once. Nearest-vertex skin weights/indices depend only on the template geometry, not the pose — compute once, reuse every frame. Skinning is often the dominant per-frame cost otherwise.

5. Point clouds and points "inside" a mesh

Rendering markers/points that may be occluded by (or embedded within) a body mesh needs real depth reasoning, not just cv2.circle at projected locations.

Cheap occlusion via a coarse z-buffer. Project all points, bucket them into a coarse pixel grid, take the per-bucket nearest depth with segment_min, and flag a point visible only if it's within a depth tolerance (~3 cm) of its bucket's front. Avoids a full per-pixel depth compare and is fully vectorized:

bu = jnp.clip(jnp.floor(u * scale).astype(jnp.int32), 0, Wb - 1)
bv = jnp.clip(jnp.floor(v * scale).astype(jnp.int32), 0, Hb - 1)
bucket = bv * Wb + bu
bucket_min = jax.ops.segment_min(jnp.where(candidate, z, 1e6), bucket, num_segments=Hb*Wb)
visible = z <= bucket_min[bucket] + depth_tolerance

Differentiable point/soft rendering (SoftRas-style). Per pixel, take a depth-aware softmax over nearby points/vertices: w ∝ exp(-‖pix − proj‖² / 2σ²) · exp(−z / depth_temp), normalized with logsumexp for stability. The alpha channel is the max spatial weight. This is what lets a renderer pass gradients back to 3D positions.

Keep peak memory bounded with chunking + remat. The naive weight matrix is (H·W, V) — multiple GiB. Tile pixels into chunks and recompute each chunk's weights in the backward pass instead of storing them:

@jax.checkpoint
def render_chunk(pix_chunk):           # (C, 2)
    d2 = jnp.sum((pix_chunk[:, None] - proj[None])**2, -1)      # (C, V)
    logit = jnp.where(vis[None], -d2/(2*sigma**2) - z[None]/depth_temp, -1e30)
    w = jnp.exp(logit - jax.scipy.special.logsumexp(logit, -1, keepdims=True))
    return w @ vertex_attr             # (C, attr)

img = jax.lax.map(render_chunk, pix.reshape(n_chunks, C, 2))

256² × 18k verts drops from ~4.5 GiB to a few hundred MiB. The @jax.checkpoint is load-bearing: without it the enclosing scan/map stores every chunk's intermediate weight matrix on the autodiff tape, which defeats the chunking. Pick the chunk size to divide H·W evenly and mark H, W, chunk as static JIT args (one compile per resolution).

Hard triangle rasterization, differentiable where it counts. When you need crisp coverage: compute barycentrics for every face per pixel, z-buffer to pick the nearest covering face (argmin over depth). Make the face id and coverage stop_gradient (a hard mask shouldn't leak gradient), but let the barycentric coordinates and depth flow gradients to the vertex positions. Same chunk-and-checkpoint trick for memory.

Vertex normals, vectorized: accumulate (area-weighted) face normals onto vertices with segment_sum over the flattened face index — no Python loop:

fn = jnp.cross(v1 - v0, v2 - v0)                      # (F,3), |fn| = 2·area
vn = jax.ops.segment_sum(jnp.repeat(fn, 3, 0), faces.reshape(-1), num_segments=V)
vn = vn / jnp.maximum(jnp.linalg.norm(vn, axis=-1, keepdims=True), 1e-8)

6. 2D overlays: keypoints, skeletons, trails

  • Project once in camera frame, rescale to the output resolution, then draw. Vectorize the projection across frames/cameras with jax.vmap rather than a Python loop.

  • Encode visibility in the marker, not just position: filled circle + white halo for visible, a dim hollow ring for occluded/low-confidence. Skip NaN / out-of-bounds points.

  • Comet-tail trails via accumulated alpha blends — width and alpha ramp from tail to head:

    for seg in range(t0 + 1, t + 1):
        frac = (seg - t0) / max(t - t0, 1)
        cv2.line(overlay, p0, p1, color, thickness=max(1, int(1 + (w_max-1)*frac)))
        cv2.addWeighted(overlay, 0.15 + 0.85*frac, img, 1 - (0.15 + 0.85*frac), 0, img)
    
  • Crisp small markers: upscale the frame 2–4×, draw markers at scaled coordinates, downscale the final video. Cheaper and sharper than antialiasing every primitive.

  • Smooth playback: interpolate tracks onto a denser time grid and repeat the nearest source frame, so markers glide while video steps.


7. Video overlay pipeline

  • Composite only the bounding box of the render's alpha mask. For sparse overlays this is most of the speedup:

    ys, xs = np.where(mask)
    y0, y1, x0, x1 = ys.min(), ys.max()+1, xs.min(), xs.max()+1
    a = (mask[y0:y1, x0:x1, None] * alpha)
    out[y0:y1, x0:x1] = (rgb[y0:y1, x0:x1] * a + frame[y0:y1, x0:x1] * (1 - a))
    

    Integer (uint16) intermediates with a // 255 divide avoid a full float32 round-trip of the whole frame.

  • Prefer piping raw frames to ffmpeg over cv2.VideoWriter for H.264: more reliable codecs, faster, and you control pixel format. Use yuv420p + -movflags +faststart for browser-playable output.

    proc = subprocess.Popen([ffmpeg, "-f","rawvideo","-pix_fmt","bgr24",
        "-s",f"{w}x{h}","-r",str(fps),"-i","-","-an","-c:v","libx264",
        "-crf","18","-pix_fmt","yuv420p","-movflags","+faststart", out_path],
        stdin=subprocess.PIPE)   # -an: no audio, -crf 18: visually lossless
    for f in frames_bgr:
        proc.stdin.write(np.ascontiguousarray(f).tobytes())
    

    If you must use cv2.VideoWriter (mp4v), a best-effort ffmpeg re-encode to H.264 afterward makes it streamable.


8. JAX-specific performance

  • One JIT boundary for FK + mesh extraction. Wrap forward kinematics and world-space vertex extraction in a single jax.jit so it all runs on device; crossing the host boundary between them costs ~100 ms of dispatch.

  • Vectorize vertex extraction — gather per-vertex transforms and einsum instead of a Python loop. Vectorizing alone (numpy) takes a body mesh from ~275 ms to ~2 ms; the same einsum jitted on-device is sub-millisecond:

    world = xpos[v2g] + jnp.einsum("vij,vj->vi", xmat[v2g].reshape(-1,3,3), local_vert)
    
  • vmap for multi-instance, padding to a fixed count so shapes stay stable and you compile once. Carry a validity mask rather than slicing.

  • Cache compiled functions by instance id when the static model object can't be a JIT argument, and stop_gradient static models inside the jitted fn.

  • Donate buffers in the training/render step (eqx.filter_jit(donate=...)) to free large inputs immediately.

  • Static-arg the resolution/chunk size; accept one compile per resolution.

See the jax-memory-and-retracing skill for diagnosing OOMs and unexpected recompiles.


Pre-flight checklist

  • PYOPENGL_PLATFORM=egl (and MUJOCO_GL=egl if rendering MuJoCo) set before any GL import.
  • Verified GL_RENDERER is the NVIDIA GPU, not llvmpipe.
  • Renderer/context allocated once and reused; cleaned up at exit.
  • Exactly one OpenCV→GL flip, at the camera node.
  • znear/zfar bracket the real scene depth (no z-fighting, no clipping).
  • One distortion strategy, applied to both image and geometry.
  • Pose-independent work (skinning, normals topology, remaps, frusta, projection matrices) precomputed once.
  • Compositing restricted to the mask bounding box.
  • JAX rasterizer chunks pixels + jax.checkpoint; statics are resolution/chunk.
  • Output encoded yuv420p + faststart if it needs to play in a browser.

Anti-patterns

Anti-patternFix
Importing GL before setting PYOPENGL_PLATFORMSet the env var first, at the very top
Relying on PYOPENGL_PLATFORM for a MuJoCo rendererMuJoCo reads MUJOCO_GL; set MUJOCO_GL=egl too
Trusting it "works" without checking GL_RENDERERAssert no llvmpipe at startup
New OffscreenRenderer per frameCache by viewport size, reuse
Distorting vertices and undistorting the imagePick one strategy, apply consistently
getOptimalNewCameraMatrix as the undistort targetKeep the original K as the new matrix
Compensating an upside-down render downstreamFix the single OpenCV→GL flip instead
Materializing (H·W, primitives) in a JAX rasterizerChunk pixels + jax.checkpoint
Recomputing skinning/normals/frusta every framePrecompute once per sequence
Compositing the whole frame for a small overlayRestrict to the mask bbox
cv2.VideoWriter mp4v for a deliverablePipe raw frames to ffmpeg H.264/yuv420p
fork for render workers after CUDA initmultiprocessing.get_context("spawn")

The Iron Law

IF A GUIDELINE APPLIES TO YOUR TASK, YOU DO NOT HAVE A CHOICE. YOU MUST FOLLOW IT.

This is not a suggestion. Guidelines are requirements. Violations require correction before work is considered complete.

When This Skill Activates

This skill activates automatically when:

  • Writing Python code → python-standards.md, coding-standards.md
  • Creating a new repository → repo-structure.md
  • Making commits → git-workflow.md
  • Reviewing or refactoring code → anti-patterns.md
  • Any code changes → coding-standards.md

You MUST check for applicable guidelines before starting any task. Even a 1% chance a guideline applies means you consult it.

Decision Flow

Before responding to ANY code-related request:

1. What type of task is this?
   └─→ Identify: python, git, new-repo, refactor, debug, review

2. Which guidelines apply?
   └─→ Map task type to required guidelines (see mapping below)

3. Read the applicable guidelines
   └─→ Actually read them, don't assume you know the content

4. Extract MUST/MUST NOT constraints
   └─→ These are non-negotiable requirements

5. Proceed with task, citing guidelines when they influence decisions

Where to Find Guidelines

Guidelines are bundled as reference files alongside this skill:

  • references/coding-standards.md
  • references/python-standards.md
  • references/repo-structure.md
  • references/git-workflow.md
  • references/anti-patterns.md
  • references/pre-commit-checklist.md

Guideline Mapping

Task TypeRequired Guidelines
Any Python codepython-standards.md, coding-standards.md
New repositoryrepo-structure.md, python-standards.md
Git operationsgit-workflow.md
Refactoringanti-patterns.md, coding-standards.md
Code reviewanti-patterns.md, python-standards.md
Bug fixescoding-standards.md (TDD section)
New featurescoding-standards.md, python-standards.md

Red Flags (Rationalization Detection)

These thoughts indicate you're trying to skip guidelines:

ThoughtReality
"This is just a quick fix"Quick fixes still follow guidelines
"I already know the standards"Read them anyway, they may have changed
"This is too simple to need guidelines"Simple code still needs type hints and formatting
"I'll fix it later"Fix it now, there is no later
"The user didn't ask for TDD"TDD is the default, not optional
"100 chars is close enough to 120"Line length is 120, configure ruff correctly

Enforcement Mechanism

Before Starting Work

  1. Identify task type from the mapping above
  2. Read applicable guidelines - actually open and read them
  3. Extract key constraints - list the MUST/MUST NOT rules
  4. Acknowledge constraints - state which rules apply to this task

During Work

  1. Cite guidelines when they influence a decision
  2. Check compliance before completing each step
  3. Run verification commands (ruff, pytest) as guidelines require

Before Completing Work

  1. Run all required checks: ruff format . && ruff check . && pytest
  2. Verify coverage: pytest runs with --cov by default, coverage must be ≥80%
  3. Verify against checklist from applicable guidelines
  4. Confirm no violations remain

Constraints

  • You MUST read guidelines before starting code tasks
  • You MUST cite the specific guideline when it influences a decision
  • You MUST NOT skip guidelines because the task seems simple
  • You MUST NOT proceed if guideline compliance is unclear
  • You MUST run ruff format and ruff check before completing Python work
  • You MUST run pytest if tests exist
  • You SHOULD flag if you discover a case not covered by guidelines

Verification Checklist

Before marking any code task complete:

  • Identified applicable guidelines
  • Read the guidelines (not just remembered them)
  • Listed MUST constraints that apply
  • Followed all MUST constraints
  • Avoided all MUST NOT patterns
  • Ran ruff format on changed files
  • Ran ruff check with no errors
  • Ran pytest with coverage ≥80%
  • No anti-patterns from anti-patterns.md

Cannot check all boxes? Work is not complete.

Examples

User: "Add a helper function to parse dates"

Agent (correct):

This is Python code, so I need to consult python-standards.md and coding-standards.md.

Key constraints:

  • MUST use type hints for function signatures
  • MUST run ruff format/check
  • SHOULD write test first (TDD)

Let me write a failing test first, then implement...

Agent (incorrect):

Sure, here's a quick function:

def parse_date(s):
    return datetime.strptime(s, "%Y-%m-%d")

The incorrect response skips guidelines consultation, misses type hints, and doesn't run verification.

Excalidraw Diagram Generation

Generate Excalidraw diagrams, render them to images, and visually verify the output.

Setup

excalirender (Bun-based standalone binary, no Node.js required):

curl -fsSL https://raw.githubusercontent.com/JonRC/excalirender/main/install.sh | sh
excalirender --version

Alternative — Docker (no install):

docker run --rm -v "$(pwd):/data" -w /data jonarc06/excalirender diagram.excalidraw -o output.png

Fallback — npx (requires Node.js 18+):

npx -y @excalidraw/excalidraw-brute-export-cli diagram.excalidraw output.png

Workflow

Step 1: Generate .excalidraw JSON

Write a valid .excalidraw file to scratch/ (or user-specified path). Follow the JSON Format Reference below exactly.

Step 2: Render to Image

excalirender scratch/diagram.excalidraw -o scratch/diagram.png

Useful flags:

  • -s 2 — 2x scale (sharper)
  • --dark — dark mode
  • --transparent — transparent background
  • -o output.svg — SVG output (use file extension)

Step 3: Visually Verify

Read the output image with the Read tool to inspect the rendered diagram. Check:

  • All elements are visible and properly positioned
  • Arrows connect to the correct shapes
  • Text is readable and not overlapping
  • Layout is balanced and clear
  • Embedded panels show actual pixels, not a blank box — a fileId mismatch renders a silent white rectangle (exit 0, correct size, no image). Inspect the panel, not just the file.

For a composite scientific figure, also run the design rubric and re-render until it passes:

  • A naive viewer can name the main message in ~5 seconds
  • There is ONE clear reading path (L→R linear / cycle / parallel-column compare)
  • ≤3 accent colors, used semantically
  • Box edges and baselines align to a grid with uniform gutters
  • All text is readable at the target export size
  • No chartjunk (3D, shadows, gradients, stray ticks/legends) that doesn't serve the message

Step 4: Iterate if Needed

Fix any issues in the .excalidraw JSON and re-render. Common fixes:

  • Adjust x/y positions to fix overlapping elements
  • Increase width/height of shapes containing long text
  • Fix arrow points coordinates for proper routing
  • Add missing boundElements entries on shapes that arrows connect to

Excalidraw JSON Format Reference

Top-Level Structure

{
  "type": "excalidraw",
  "version": 2,
  "source": "https://excalidraw.com",
  "elements": [],
  "appState": {
    "viewBackgroundColor": "#ffffff",
    "gridSize": null
  },
  "files": {}
}

Common Element Properties

Every element shares these base properties:

{
  "id": "rect1",
  "type": "rectangle",
  "x": 100,
  "y": 200,
  "width": 200,
  "height": 80,
  "angle": 0,
  "strokeColor": "#1e1e1e",
  "backgroundColor": "transparent",
  "fillStyle": "solid",
  "strokeWidth": 2,
  "strokeStyle": "solid",
  "roughness": 1,
  "opacity": 100,
  "seed": 1234567,
  "version": 1,
  "versionNonce": 987654,
  "isDeleted": false,
  "groupIds": [],
  "frameId": null,
  "boundElements": null,
  "updated": 1700000000000,
  "link": null,
  "locked": false,
  "roundness": { "type": 3 }
}

Property Reference:

PropertyTypeValues / Notes
idstringUnique identifier. Use short strings like "box1", "arr1"
typestring"rectangle", "ellipse", "diamond", "text", "arrow", "line", "freedraw", "image", "frame"
x, ynumberTop-left corner position in canvas pixels
width, heightnumberDimensions in pixels
anglenumberRotation in radians (0 = no rotation)
strokeColorstringHex color for border/stroke
backgroundColorstring"transparent" or hex color for fill
fillStylestring"solid", "hachure", "cross-hatch", "zigzag"
strokeWidthnumber1 (thin), 2 (normal), 4 (thick)
strokeStylestring"solid", "dashed", "dotted"
roughnessnumber0 (architect/clean), 1 (artist/default), 2 (cartoonist/rough)
opacitynumber0-100
seednumberRandom int for deterministic rough.js rendering
versionnumberStart at 1
versionNoncenumberRandom int
isDeletedbooleanAlways false for new elements
groupIdsarrayArray of group ID strings. Elements sharing a groupId are grouped
frameIdstring|nullID of containing frame element
boundElementsarray|nullArray of {"id": "...", "type": "arrow"} or {"id": "...", "type": "text"}
roundnessobject|null{"type": 3} for rounded corners, null for sharp
lockedbooleanfalse for editable

Element Types

Rectangle, Ellipse, Diamond

Use the common properties above. Set type to "rectangle", "ellipse", or "diamond".

To make a shape a container for text, add the text element's ID to boundElements:

{
  "id": "box1",
  "type": "rectangle",
  "x": 100, "y": 100, "width": 200, "height": 80,
  "boundElements": [{"id": "text1", "type": "text"}],
  "...common properties..."
}
Text

Additional properties beyond the common set:

{
  "id": "text1",
  "type": "text",
  "text": "Hello World",
  "fontSize": 20,
  "fontFamily": 1,
  "textAlign": "center",
  "verticalAlign": "middle",
  "containerId": "box1",
  "originalText": "Hello World",
  "autoResize": true,
  "lineHeight": 1.25
}
PropertyValues
fontSizeNumber in pixels. Common: 16, 20, 28, 36
fontFamily1 = Excalifont (hand-drawn), 2 = Nunito (sans-serif), 3 = Comic Shanns (code), 4 = Liberation Sans (system)
textAlign"left", "center", "right"
verticalAlign"top", "middle"
containerIdID of parent shape, or null for standalone text
originalTextSame as text (used for undo)
autoResizetrue to auto-fit container
lineHeight1.25 (default)

Text-in-Container binding: The text element has containerId: "box1" AND the shape has boundElements: [{"id": "text1", "type": "text"}]. Both sides must reference each other. When text is inside a container, set the text's x, y, width, height to match the container's inner area (Excalidraw recomputes these, but providing reasonable values avoids rendering artifacts).

Arrow

Additional properties beyond the common set:

{
  "id": "arr1",
  "type": "arrow",
  "x": 300, "y": 140,
  "width": 200, "height": 0,
  "points": [[0, 0], [200, 0]],
  "startBinding": {
    "elementId": "box1",
    "fixedPoint": [1, 0.5]
  },
  "endBinding": {
    "elementId": "box2",
    "fixedPoint": [0, 0.5]
  },
  "startArrowhead": null,
  "endArrowhead": "arrow",
  "elbowed": false
}
PropertyValues
pointsArray of [dx, dy] relative to element's x,y. First point MUST be [0, 0]
startBindingnull or {"elementId": "...", "fixedPoint": [fx, fy]}
endBindingSame as startBinding
fixedPoint[0-1, 0-1] normalized position on target shape. [0, 0.5] = left center, [1, 0.5] = right center, [0.5, 0] = top center, [0.5, 1] = bottom center
startArrowheadnull, "arrow", "bar", "dot", "triangle", "diamond"
endArrowheadSame options as startArrowhead
elbowedfalse for curved/straight, true for right-angle routing

Arrow-Shape binding: The arrow has startBinding.elementId: "box1" AND box1 has boundElements: [{"id": "arr1", "type": "arrow"}]. Both sides must reference each other.

Arrow positioning: Set the arrow's x, y to the start point. Set width and height to the bounding box of all points. The points array is relative to x, y, so the first point is always [0, 0].

Line

Same as arrow but with type: "line". No startBinding/endBinding/arrowheads.

{
  "type": "line",
  "points": [[0, 0], [100, 50], [200, 0]],
  "...common properties..."
}
Frame

Groups elements visually with a labeled border:

{
  "id": "frame1",
  "type": "frame",
  "name": "Backend Services",
  "x": 50, "y": 50, "width": 600, "height": 400,
  "...common properties..."
}

Child elements set "frameId": "frame1" to belong to the frame.

Image (embedded raster panel)

Embeds a rendered PNG/JPG/SVG (a matplotlib chart, a MuJoCo/mesh render, a photo) as a panel. An embedded image is two coupled parts, and the one load-bearing rule is:

element.fileId === the files-map key === that entry's .id. The element holds no pixels; the bytes live in the files entry.

If they don't match (or the files entry is missing), excalirender draws a silent white box — exit 0, correct size, no pixels. So Step 3 must inspect the panel, not just that the file rendered.

Image element (extras beyond the common base):

PropertyValues
fileIdHex string; the key into files. Use sha1(bytes).hexdigest() (40 chars)
status"saved" for generated files
scale[1, 1] default; [-1, 1] flips horizontally
cropnull, or {x, y, width, height, naturalWidth, naturalHeight} for a sub-region
width/heightdisplay box — set to the image's natural aspect ratio or it stretches (excalirender fills the box, it does not preserve aspect)

Files entry (top-level files map, keyed by fileId):

PropertyValues
idSame as the key and the element's fileId
dataURLFull "data:image/png;base64,..." string including the prefix
mimeType"image/png", "image/jpeg", or "image/svg+xml"
createdEpoch ms (e.g. 1700000000000)

Hand-authoring megabytes of base64 is impractical — generate the pair with this stdlib helper. Keep PNGs in scratch/ and reference them by path; never paste base64 into chat.

import base64, hashlib

def embed_image(path, x, y, width, height, mime="image/png"):
    """Return (image_element, files_patch) for a rendered panel. fileId is the
    sha1 of the bytes, so re-embedding an identical panel dedupes."""
    data = open(path, "rb").read()
    file_id = hashlib.sha1(data).hexdigest()
    data_url = f"data:{mime};base64," + base64.b64encode(data).decode()
    element = {
        "id": file_id[:8], "type": "image", "x": x, "y": y,
        "width": width, "height": height, "angle": 0, "strokeColor": "#1e1e1e",
        "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2,
        "strokeStyle": "solid", "roughness": 0, "opacity": 100, "seed": 42,
        "version": 1, "versionNonce": 1, "isDeleted": False, "groupIds": [],
        "frameId": None, "boundElements": None, "updated": 1700000000000,
        "link": None, "locked": False, "fileId": file_id, "status": "saved",
        "scale": [1, 1], "crop": None,
    }
    files_patch = {file_id: {"mimeType": mime, "id": file_id,
                             "dataURL": data_url, "created": 1700000000000}}
    return element, files_patch

Downsample large renders to display resolution before embedding — base64 inflates ~33% and lands inline in the JSON (and again in SVG export), so a few multi-MB panels make an unreadable, undiffable file. Keep .excalidraw files with embedded panels out of git.

Color Palette

Use the Excalidraw default palette (Open Color shades):

Stroke colors (shade 7-9, saturated):

ColorHex
Black#1e1e1e
Red#e03131
Pink#c2255c
Grape#9c36b5
Violet#6741d9
Indigo#3b5bdb
Blue#1971c2
Cyan#0c8599
Teal#099268
Green#2f9e44
Yellow#f08c00
Orange#e8590c

Background fills (shade 1-3, pastel):

ColorHex
Red#ffc9c9
Blue#a5d8ff
Green#b2f2bb
Yellow#ffec99
Violet#d0bfff
Cyan#99e9f2
Orange#ffd8a8
Gray#e9ecef

Grouping

To group elements, assign the same group ID string to their groupIds array:

{"id": "a", "groupIds": ["group1"], "..."},
{"id": "b", "groupIds": ["group1"], "..."}

Nested groups: "groupIds": ["innerGroup", "outerGroup"] (innermost first).

Diagram Patterns

Architecture Diagram (Boxes + Arrows)

Layout: horizontal flow, 250px spacing between box centers.

[Client]  -->  [API Gateway]  -->  [Service]  -->  [Database]
x=0           x=300               x=600           x=900

Each box: width=200, height=80. Arrows start at right edge of source (x+200, y+40), end at left edge of target (x, y+40).

Flowchart (Decision Diamonds)

Layout: vertical flow, 150px spacing between element centers.

     [Start]
        |
   <Decision?>
    /       \
 [Yes]     [No]

Diamonds: width=160, height=100. Use fixedPoint: [0.5, 1] for bottom exit, [0, 0.5] and [1, 0.5] for side exits.

Layered Architecture

Layout: vertical stack with frames.

+-- Presentation Layer --+
|  [Component] [Component] |
+------------------------+
+-- Business Logic ------+
|  [Service]  [Service]    |
+------------------------+
+-- Data Layer ----------+
|  [Repository] [Cache]   |
+------------------------+

Use frame elements for each layer, 50px vertical gap between frames.

Composite scientific figures (rendered panels + annotation)

This section applies only to composite/overview figures. A plain architecture diagram or flowchart ignores all of it.

A scientific overview figure is rendered raster panels (matplotlib/plotly charts, MuJoCo/mesh/point-cloud renders, photos) under a vector annotation layer (arrows, panel labels, callouts, title). Excalidraw is the programmatic layout + annotation layer; it does not render the data — panels are rendered elsewhere and embedded. If every panel is a matplotlib chart, just use matplotlib subfigures/GridSpec instead. Reach for excalidraw the moment panels are heterogeneous (charts + 3D renders + hand annotation).

Workflow

  1. State the message. One sentence: the figure's single take-home, and the medium (paper column / slide / poster). Every element serves it.
  2. Render each panel separately at final 1:1 size. Charts: share one rcParams style block across all panels (sans-serif, fixed linewidth, the figure's hex color cycler), savefig(dpi=300, transparent=True, bbox_inches="tight", pad_inches=0). 3D/mesh/MuJoCo: hand off to /efficient-rendering (RGBA + transparent bg + matte material, so it composites with no grey box); overlay-on-video result panels use /camera-model for intrinsics/extrinsics/units. Keep PNGs in scratch/.
  3. Embed each panel with embed_image(...). Image elements go first in elements[] (lowest z-order) so annotations sit on top.
  4. Lay out + annotate (the value-add): one frame per panel on a shared grid (snap x/y to a ~20px module, uniform gutters, equal-size siblings, shared baselines); bold lowercase a/b/c labels at a fixed corner offset (+8,+8); arrows along one reading direction (L→R / cycle / parallel-column compare); a title that asserts the finding.
  5. Render + verify: excalirender fig.excalidraw -o fig.png -s 2, Read the PNG, run the Step-3 rubric, iterate.
  6. Export: prefer SVG/PDF (annotations stay vector, only panels are raster — publication-safe). Full-figure PNG (-s 2+) for drafts/slides.

Defaults & palette

  • Publication look: roughness: 0, fillStyle: "solid", fontFamily: 2 (or 4) on all text, body labels ≥16px canvas. (roughness: 1 / Excalifont is for casual sketches only.)
  • DPI × export scale: embed at the pixel size you want at export scale s; set element width ≈ natural/s; export with matching -s. Otherwise source is crisp but output blurs.
  • Color: ≤3 accent hues; gray (#1e1e1e / #e9ecef) for everything de-emphasized; one hue per concept, reused everywhere that concept appears — feed the same hex list to the matplotlib cycler and to render colors. Never red+green to distinguish two classes.
  • Multiview: panels comparing views/methods must share the camera/unit convention from /camera-model — don't mix mm and m across panels.

Companion skills (suggest, never auto-install)

Excalidraw does layout/annotation, not data rendering or palette selection. When a figure needs more, suggest a vetted external skill and let the user install it:

NeedSkill (license)
Tufte-styled minimal-ink chart panelstufte-data-viz (caylent, MIT, read-only)
Colorblind-safe palette / "what colors?"color-expert (meodai, CC-BY-4.0)
Choose & justify a chart from raw dataSciPilot figure skill (MIT, local Python)
Crisp editorial schematic, no hand-drawn lookdiagram-design (MIT)
Text-diffable infra/architecture diagramMermaid / D2 skills

Vetting bar: permissive license (MIT/CC-BY), real adoption, prefer read-only/no-exec. Avoid AntV mcp-server-chart for unpublished data — by default it POSTs your chart spec to a hosted server.

Best Practices

  • IDs: Use descriptive short strings ("box_api", "arr_to_db", "text_title")
  • Spacing: 200-300px horizontal gap between connected nodes, 120-150px vertical gap
  • Arrow points: First point MUST be [0, 0]. A horizontal arrow of length 200: points: [[0, 0], [200, 0]]
  • Seeds: Set seed to any random integer (e.g., 42, 999, 1577544) for consistent rough.js rendering
  • Bidirectional bindings: ALWAYS set both the arrow's startBinding/endBinding AND the shape's boundElements array. Missing either side breaks the connection
  • Text sizing: For text inside containers, set the text element's width/height smaller than the container (leave ~20px padding each side)
  • Roughness 0: Use for clean technical diagrams. Use 1 for the classic hand-drawn look
  • File extension: Always use .excalidraw — this is a plain JSON file
  • Output location: Write diagrams to scratch/ by default, or user-specified path

Fetching Synchronized Keypoints + Reconstruction Data

Overview

KinematicReconstruction qpos and 2D keypoints come from different DataJoint tables with independently managed timestamps. Naive timestamp matching silently produces frame offsets that corrupt downstream tasks (guidance, evaluation, metrics). The correct approach uses fetch_keypoints(only_detected=True) from BodyModels, which guarantees 1:1 frame correspondence with KR qpos by construction.

When to Use

  • Fetching 2D keypoints that must align frame-by-frame with KR qpos
  • Building inference/guidance pipelines that pair observations with reconstruction
  • Debugging high reprojection error or misaligned 2D-3D data
  • Writing any code that joins TopDownPerson keypoints with KinematicReconstruction

When NOT to use: Fetching keypoints for standalone 2D analysis (no KR alignment needed), or querying KR qpos without 2D observations.


The Correct Pattern

from body_models.datajoint.dataset import fetch_keypoints as bm_fetch_keypoints
from body_models.datajoint.kinematic_dj import KinematicReconstruction

trial_key = {
    "participant_id": participant_id,
    "session_date": session_date,
    "recording_timestamps": recording_timestamps,
}
full_key = {**trial_key, "kinematic_reconstruction_settings_num": 137}

# 1. Fetch qpos from KR
qpos = (KinematicReconstruction.Trial & full_key).fetch1("qpos")  # (T, 41)

# 2. Fetch keypoints using the SAME function that built the KR
timestamps, keypoints_raw = bm_fetch_keypoints(trial_key, only_detected=True)
# timestamps: (T,) video-start-relative seconds
# keypoints_raw: (C, T, 87, 3) — (x, y, confidence) per camera/frame/marker

# 3. CRITICAL: verify 1:1 correspondence
assert qpos.shape[0] == keypoints_raw.shape[1], (
    f"Frame count mismatch: qpos={qpos.shape[0]} vs keypoints={keypoints_raw.shape[1]}"
)

# 4. Extract components
confidences = keypoints_raw[..., 2]    # (C, T, 87)
keypoints_2d = keypoints_raw[..., :2]  # (C, T, 87, 2)

Why this works: fetch_keypoints(only_detected=True) applies the exact same frame filtering as KeypointDataset used when building the KR. Both arrays share the same frame indices — qpos[i] corresponds to keypoints[:, i] by construction.


Camera Parameter Reordering

KinematicReconstruction.updated_calibration stores camera parameters in Calibration table order. Keypoints from fetch_keypoints are in alphabetical SingleCameraVideo order. These may differ.

from multi_camera.datajoint.calibrate_cameras import Calibration
from multi_camera.datajoint.multi_camera_dj import SingleCameraVideo

cal_camera_names = (Calibration & parent_key).fetch1("camera_names")
camera_rows = (SingleCameraVideo & trial_key).fetch(as_dict=True, order_by="camera_name")
kp_camera_names = [row["camera_name"] for row in camera_rows]

# Reindex camera_params to match keypoint camera order
cal_names_list = list(cal_camera_names)
reorder_idx = [cal_names_list.index(name) for name in kp_camera_names]
for pkey in ("mtx", "rvec", "tvec", "dist"):
    val = np.asarray(camera_params[pkey])
    if val.ndim > 1:
        camera_params[pkey] = val[reorder_idx]

Without this, camera i in keypoints_2d gets projected with the wrong camera's intrinsics/extrinsics.


Contiguous Segment Selection

After alignment, find the longest window with continuous detections and no time gaps:

detected = np.sum(confidences, axis=(0, 2))  # (T,) — sum across cameras and joints
dt = np.median(np.diff(timestamps))
time_gaps = np.concatenate([np.diff(timestamps), [dt]]) > (1.5 * dt)
valid = np.logical_and(detected > 0, ~time_gaps)

# find_longest_segment returns (start, end) exclusive
start, end = find_longest_segment(valid)
timestamps = timestamps[start:end]
qpos = qpos[start:end]
keypoints_2d = keypoints_2d[:, start:end]
confidences = confidences[:, start:end]

Anti-Patterns

PatternWhy It Fails
Fetching TopDownPerson keypoints + VideoInfo timestamps separately, then matching with isin_with_tolerance()KR timestamps zeroed relative to first detection; VideoInfo zeroed relative to video start. When first detection is at frame N>0, KR[0] incorrectly matches video[0] instead of video[N].
Zeroing timestamps with ts - ts[0] before matchingDestroys the physical time correspondence between independently-sourced arrays.
Using fetch_keypoints(only_detected=False) with KR qposReturns all video frames including those without detections. Frame count won't match KR qpos (which only covers detected frames).
Assuming camera_params order matches keypoint orderCalibration order != SingleCameraVideo alphabetical order. Must reorder explicitly.

Real-world impact: The timestamp-matching anti-pattern caused a 38-frame temporal offset on trial ASB_144 cam 8 (first detection at video frame 38), producing 900px reprojection error and unusable guidance.


Invariants

  1. Frame correspondence: qpos.shape[0] == keypoints_raw.shape[1] — always assert this
  2. Camera ordering: After reordering, camera_params index i matches keypoints_2d[i]
  3. Timestamps are video-start-relative: Not zeroed by KR; usable directly for time-gap detection
  4. only_detected=True is mandatory when aligning with KR qpos

Quick Reference

FunctionSourceReturns
fetch_keypoints(key, only_detected=True)body_models.datajoint.dataset(timestamps, keypoints) — (T,) and (C, T, 87, 3)
KinematicReconstruction.Trial.fetch1("qpos")body_models.datajoint.kinematic_dj(T, 41) MuJoCo state
KinematicReconstruction.fetch1("updated_calibration")sameCamera params dict (Calibration order)
Calibration.fetch1("camera_names")multi_camera.datajoint.calibrate_camerasList of camera names in calibration order

Complete reference implementation: BiomechanicalFlowModel/src/biomechanical_fm/datajoint/datajoint_fetcher.py:fetch_trial_data()

Related skills: rae:datajoint-biomechanics-schema for schema details, rae:camera-model for projection conventions.

GaitLab Dataset Reference

Overview

The gait_lab schema stores traditional clinical gait lab data from Vicon c3d files. This is a separate system from the markerless multicamera (MMC) pipeline. The same participant may have data in both systems, recorded during the same session (MAC_LAB participants).

Source: GaitLabDataset/gait_lab_dataset/

Schema name: gait_lab


Table Hierarchy

Subject (subject_hash)
    -> Session (subject_hash, session_name)
        -> Trial (subject_hash, session_name, trial_name)
        |   -> Mocap (point_names, points in mm, point_rate, timestamps)
        |   |   -> GaitEvents (event, side, time)
        |   |   -> GaitParameters (parameter, side, value)
        |   |   -> MocapAnalysis (property, value)
        |   |   -> MocapProjection (toffset, camera params, error, proj, frames)
        |   -> GaitTrialVideo -> pose_pipeline.Video
        |   -> Analog (analog_names, channels, timestamps, units)
        -> PTEval (eval_html) -> PTDemographics (age, diagnosis)
        -> Assessment (assessment)
        -> Height (computed, meters)
        -> Weight (computed, kg)
        -> SessionConditions -> SessionCondition + ConditionTrial
        -> MPParam (property, value, all_values)
    -> SubjectTrainTest (train boolean)

Table Definitions

Core Tables

Subject - De-identified subject identifier

subject_hash: varchar(100)  # Primary key

Session - Clinical session grouping

-> Subject
session_name: varchar(100)

Trial - Individual gait trial

-> Session
trial_name: varchar(100)
---
timestamp: timestamp          # When the trial was recorded
description: varchar(1000)    # Free-text (e.g., "Barefoot with cane", "Right AFO")
vid_present: boolean          # Whether video was recorded
mocap_present: boolean        # Whether c3d mocap data exists

Motion Capture Data

Mocap - 3D marker data from c3d files

-> Trial
---
point_names: longblob    # List of marker names (e.g., ["SACR", "LASI", "RASI", ...])
points: longblob         # Shape: [n_frames, n_markers, 5]
                         #   Channels: [x, y, z, residual, visibility]
                         #   Units: MILLIMETERS
                         #   Visibility: 0 = valid, non-zero = occluded
point_rate: float        # Sampling rate, typically 100 Hz
frames: longblob         # Frame indices
timestamps: longblob     # Computed as frames / point_rate (seconds)

Analog - Force plate and EMG data

-> Trial
---
analog_names: longblob   # Channel labels (e.g., forceplate channels, EMG channels)
channels: longblob       # Shape: [n_channels, n_samples] (transposed)
timestamps: longblob     # Synced to mocap: mocap_start + arange(n_samples) / analog_rate
units: longblob          # Physical units per channel (e.g., "N", "V")

Analog data is synchronized to mocap via a shared trigger pulse. Timestamps are computed relative to the mocap frame start time.

GaitEvents - Annotated gait events from Vicon processing

-> Mocap
event_num: smallint
---
event: varchar(100)          # e.g., "The instant the heel strikes the ground"
side: enum('Left', 'Right')
time: float                  # Event time in seconds relative to mocap timestamps

GaitParameters - Pre-computed clinical gait parameters

-> Mocap
parameter: varchar(50)       # e.g., stride length, cadence
side: enum('Left', 'Right')
---
value: float

Video Linkage

GaitTrialVideo - Links gait trials to the pose pipeline's Video table

-> Trial
-> Video                     # from pose_pipeline (video_project, filename)

Session Metadata

SessionConditions (Computed) - Parsed trial conditions from descriptions

-> Session
---
num_conditions: smallint     # Number of distinct gait conditions tested

Part table SessionCondition:

-> SessionConditions
support: varchar(50)         # "none", "cane", "walker", "gait_trainer", "handheld", "crutch"
left_foot: varchar(50)       # "barefoot", "afo", "smo", "kafos", "brace", "ucb", "insert"
right_foot: varchar(50)      # Same options as left_foot
shoe: varchar(50)            # Whether shoes were worn
---
num_mocap: smallint           # Trials with synchronized mocap
num_frontal: smallint         # Frontal-view-only video trials
num_sagital: smallint         # Sagittal-view-only video trials

Part table ConditionTrial:

-> GaitTrialVideo
-> SessionConditions.SessionCondition
---
view: smallint               # 0=with mocap sync, 1=frontal only, 2=sagittal only

Height (Computed) - Session-level height

-> Session
---
height: float                # Meters, from MocapAnalysis("HEIGHT") or MPParam("Height")

Weight (Computed) - Session-level weight

-> Session
---
weight: float                # kg, from MocapAnalysis("BODYMASS") or MPParam("Bodymass")

MocapProjection: Video-Mocap Synchronization

The MocapProjection table synchronizes 3D mocap markers with 2D video keypoints, solving for both a temporal offset and camera projection parameters.

Source: gait_lab_dataset/mocap_sync.py

Table Definition

-> Mocap
-> GaitTrialVideo
-> TopDownPerson              # 2D keypoints from pose detection
-> VideoInfo
---
toffset: float               # Temporal offset (seconds) between mocap and video
fx, fy: float                # Camera focal lengths (pixels)
cx, cy: float                # Principal point (image center)
tx, ty, tz: float            # 3D translation (mocap -> camera frame)
rx, ry, rz: float            # Rotation (Rodrigues / exponential map, stored at 1/1000 scale)
error: float                 # Huber loss of final projection fit (lower = better)
proj: longblob               # Projected mocap positions in 2D [n_frames, n_joints, 2]
frames: longblob             # Video frame indices where projection was valid

Synchronization Algorithm

The algorithm jointly optimizes temporal offset and camera parameters to align 3D mocap markers to 2D pose detections:

  1. Data extraction (get_dual_data()):

    • Fetches mocap 3D markers (hip, knee, ankle via Plug-In Gait bone model)
    • Fetches 2D keypoints from TopDownPerson (COCO format)
    • Filters to frames with high confidence 2D detections (>0.2) and valid mocap markers (visibility==0)
    • Swaps Y/Z axes to align mocap vertical with image vertical
    • Clips to temporal overlap with 200ms buffer at each end
  2. OpenCV initialization (cv2_calibrate()):

    • Uses cv2.calibrateCamera() with all points across time as one observation
    • Fixes principal point at image center, fixes all distortion coefficients
    • Initial focal length: ~1200 pixels
    • Must achieve Huber loss < 12 (quality gate)
  3. JAX/BFGS optimization (optimize_projection()):

    • 9 parameters: [tx, ty, tz, rx, ry, rz, fx, fy, toffset_raw]
    • Time offset bounded: toffset = tanh(toffset_raw) * 0.2 (max +/- 200ms)
    • Mocap is interpolated at timestamps + toffset for each candidate offset
    • Loss: Huber loss between detected 2D keypoints and projected 3D markers
    • Uses jax.scipy.optimize.minimize(method="BFGS", maxiter=1000)
    • Projection uses SO3.exp(rvec) for rotation (jaxlie)
  4. Storage: Optimized parameters stored in MocapProjection table. Note rx, ry, rz are stored at 1/1000 scale (divided by 1000 before insert).

Marker Mapping (Mocap <-> COCO)

The synchronization maps Plug-In Gait bone markers to COCO keypoints:

COCO KeypointPlug-In Gait MarkerDescription
Right HipRFEPRight Femur Proximal
Right KneeRFEORight Femur Origin
Right AnkleRTIORight Tibia Origin
Left HipLFEPLeft Femur Proximal
Left KneeLFEOLeft Femur Origin
Left AnkleLTIOLeft Tibia Origin

Data Statistics

  • ~8,568 successful MocapProjection rows computed
  • ~9,405 rows with Mocap + GaitTrialVideo + TopDownPerson available
  • ~1,000 potentially recoverable with improved alignment
  • ~15,359 videos with mocap present total

MMC-GaitLab Cross-System Matching

The gait_lab_dataset.mmc module compares markerless (MMC) and marker-based (GaitLab) motion capture data for the same participants (MAC_LAB sessions).

Source: gait_lab_dataset/mmc/

Core Workflow

from gait_lab_dataset.mmc import (
    get_gaitlab_session_for_participant,
    get_mmc_pelvis_data,
    get_gaitlab_pelvis_data,
    compute_session_times,
    find_optimal_matching,
    format_matching_report,
)

participant_id = "mac002"

# 1. Look up GaitLab session for this MMC participant (matches by date)
session_name = get_gaitlab_session_for_participant(participant_id)

# 2. Fetch pelvis trajectories from both systems
mmc_data = get_mmc_pelvis_data(participant_id)
gl_data = get_gaitlab_pelvis_data(session_name)

# 3. Compute session-relative timestamps
_, mmc_data = compute_session_times(mmc_data, "recording_time")
_, gl_data = compute_session_times(gl_data, "trial_time")

# 4. Find optimal trial matching (handles clock drift, spatial calibration)
result = find_optimal_matching(mmc_data, gl_data)

# 5. Human-readable report
report = format_matching_report(result, mmc_data, gl_data)
print(report)

MatchingResult Fields

FieldTypeDescription
matcheslist[tuple[int, int, float]](mmc_idx, gl_idx, offset_seconds) per matched trial
coveragefloatFraction of GaitLab trials matched (0-1)
r_squaredfloatCombined R-squared across all matched trials
offset_meanfloatMean temporal offset in seconds
offset_stdfloatTemporal offset std dev in seconds
cost_matrixndarray [n_mmc, n_gl]1 - R-squared for each trial pair
r_squared_matrixndarray [n_mmc, n_gl]Best R-squared for each pair
unmatched_mmclist[int]Indices of unmatched MMC trials
unmatched_gllist[int]Indices of unmatched GaitLab trials
affine_Andarray [2,2]Global affine transform
affine_bndarray [2]Affine offset

Synchronization Quality Interpretation

MetricThresholdInterpretation
R-squared > 0.95ExcellentHigh confidence in trial matching
R-squared 0.8-0.95GoodMatches likely correct
R-squared < 0.8PoorReview manually
Offset std < 2sConsistentClock alignment stable across trials
Offset std > 5sProblematicClock drift or matching errors

Joint Angle Comparison

from gait_lab_dataset.mmc import (
    get_mmc_joint_angles,
    get_gaitlab_joint_angles,
    align_joint_angles,
    compute_joint_r_squared,
    compute_joint_rmse,
    compute_axis_correlations,
)

mmc_angles = get_mmc_joint_angles(participant_id)
gl_angles = get_gaitlab_joint_angles(session_name)
_, mmc_angles = compute_session_times(mmc_angles, "recording_time")
_, gl_angles = compute_session_times(gl_angles, "trial_time")

# Compare matched trials
for mmc_idx, gl_idx, offset in result.matches:
    times, mmc, gl = align_joint_angles(mmc_angles[mmc_idx], gl_angles[gl_idx], offset)
    r2 = compute_joint_r_squared(mmc, gl)    # Per-joint R-squared (handles sign flips)
    rmse = compute_joint_rmse(mmc, gl)        # Per-joint RMSE in degrees

Joint angle mappings:

  • Sagittal plane joints: hip_flexion, knee_angle, ankle_angle (bilateral)
  • GaitLab markers: LHipAngles, RHipAngles, LKneeAngles, RKneeAngles, LAnkleAngles, RAnkleAngles
  • Each has 3 channels: sagittal (X), frontal (Y), transverse (Z) in degrees

Plug-In Gait Marker Names

Surface Markers (17)

MarkerFull Name
SACRSacral
LASI / RASILeft / Right Anterior Superior Iliac Spine
LPSI / RPSILeft / Right Posterior Superior Iliac Spine
LTHI / RTHILeft / Right Thigh
LKNE / RKNELeft / Right Knee
LTIB / RTIBLeft / Right Tibia
LANK / RANKLeft / Right Ankle
LHEE / RHEELeft / Right Heel
LTOE / RTOELeft / Right Toe

Bone Markers (Used for Anatomical Calibration)

Naming pattern: {SEGMENT}{SUFFIX} where suffix is O=Origin, P=Proximal, A=Anterior, L=Lateral.

Segments: PEL (Pelvis), RFE/LFE (Femur), RTI/LTI (Tibia), RFO/LFO (Foot), RTO/LTO (Toe)


Common Queries

Count sessions with c3d data

from gait_lab_dataset.dataset_dj import Trial, Session
sessions_with_mocap = Session & (Trial & 'mocap_present=1')
print(f"Sessions with c3d data: {len(sessions_with_mocap)}")

Find MAC_LAB sessions with corresponding c3d files

from multi_camera.datajoint.sessions import Session as MMCSession, Recording
from multi_camera.datajoint.multi_camera_dj import MultiCameraRecording
from gait_lab_dataset.dataset_dj import Session as GLSession, Trial, Mocap

# Get MAC_LAB MMC sessions
mmc_sessions = (MMCSession & (Recording & (MultiCameraRecording & "video_project='MAC_LAB'"))).fetch(as_dict=True)

# Get GaitLab sessions with mocap, compute dates from trial timestamps
gl_sessions = (GLSession & Mocap).fetch(as_dict=True)
gl_dates = {}
for gl in gl_sessions:
    timestamps = (Trial & gl).fetch('timestamp')
    if len(timestamps) > 0:
        gl_dates[gl['session_name']] = min(timestamps).date()

# Match by date
matched = sum(1 for mmc in mmc_sessions
              if any(mmc['session_date'] == d for d in gl_dates.values()))

Query MocapProjection synchronization error

from gait_lab_dataset.mocap_sync import MocapProjection
import numpy as np

errors = MocapProjection.fetch('error')
print(f"Entries: {len(errors)}, Median error: {np.median(errors):.4f}")

Get gait events for a trial

from gait_lab_dataset.dataset_dj import GaitEvents
events = (GaitEvents & key).fetch(as_dict=True)
for e in events:
    print(f"  {e['side']} {e['event']} at {e['time']:.3f}s")

Get synchronized training data

from gait_lab_dataset.fetch_training_dataset import get_synced_trial
# Returns aligned: [timestamps, keypoints3d, keypoints2d, mocap, mocap_weight, mocap_phase, mocap_phase_weight]
data = get_synced_trial(key)

Data Conventions

Units

DataUnits
Mocap marker positions (points)millimeters
GaitLab joint anglesdegrees
Heightmeters
Weightkg
Gait event timesseconds (relative to mocap start)
MocapProjection toffsetseconds
MocapProjection rx, ry, rzRodrigues vector / 1000 (stored scaled down)
Analog timestampsseconds (synced to mocap clock)
MMC pelvis datameters (from KinematicReconstruction)

Coordinate Axes

SystemXYZ
Mocap raw (Vicon)ForwardLateralVertical
After sync transformForwardVerticalLateral

The synchronization code swaps Y/Z to align the mocap vertical axis with the image height axis: points[:, :, [0, 2, 1, 3, 4]]

Visibility Flags

  • Mocap: points[:, :, 4] == 0 means visible/valid; non-zero means occluded
  • Video 2D keypoints: keypoints[:, :, 2] > 0.2 indicates high-confidence detection

Known Issues

IssueDetails
Knee angle sign conventionGaitLab and MMC use opposite signs (~-0.9 correlation). compute_joint_r_squared handles this via linear regression.
Unit mismatchGaitLab points in mm; MMC pelvis_xyz in meters. Data loading functions handle conversion.
find_optimal_matching performanceBuilds O(n*m) cost matrix with offset search. ~30s per participant.
rx, ry, rz storage scalingStored at 1/1000 scale in MocapProjection (divided before insert). Multiply by 1000 to recover original optimization values.
Missing TopDownPerson~5,959 videos lack 2D pose detections, preventing MocapProjection computation.
cv2_calibrate duplicatemocap_sync.py contains two definitions of cv2_calibrate. The second (line 232) is the active one.

Quick Reference: Imports

# Core schema tables
from gait_lab_dataset.dataset_dj import (
    Subject, Session, Trial, Mocap, GaitTrialVideo,
    GaitEvents, GaitParameters, Analog, Height, Weight,
    SessionConditions, MocapAnalysis, MPParam, SubjectTrainTest,
    ValidGaitEvents, PTEval, Assessment,
)

# Synchronization
from gait_lab_dataset.mocap_sync import MocapProjection

# MMC-GaitLab cross-system matching
from gait_lab_dataset.mmc import (
    get_gaitlab_session_for_participant,
    find_optimal_matching,
    format_matching_report,
    get_mmc_pelvis_data,
    get_gaitlab_pelvis_data,
    compute_session_times,
    get_mmc_joint_angles,
    get_gaitlab_joint_angles,
    align_joint_angles,
    compute_joint_r_squared,
    compute_joint_rmse,
    compute_axis_correlations,
)
from gait_lab_dataset.mmc.types import MatchingResult

# Training data
from gait_lab_dataset.fetch_training_dataset import get_synced_trial

# Associated tables from other systems
from pose_pipeline.pipeline import Video, VideoInfo, TopDownPerson
from multi_camera.datajoint.sessions import Session as MMCSession, Recording
from multi_camera.datajoint.multi_camera_dj import MultiCameraRecording

File Paths

ComponentFile
Schema definitiongait_lab_dataset/dataset_dj.py
Video-mocap syncgait_lab_dataset/mocap_sync.py
Gait event processinggait_lab_dataset/mocap_processing.py
Training data fetchgait_lab_dataset/fetch_training_dataset.py
Gait phase estimationgait_lab_dataset/gait_phase_kalman.py
MMC data loadinggait_lab_dataset/mmc/data_loading.py
MMC trial matchinggait_lab_dataset/mmc/matching.py
MMC joint comparisongait_lab_dataset/mmc/joint_comparison.py
MMC typesgait_lab_dataset/mmc/types.py

Gait Metrics Reference

Overview

The gait analysis pipeline detects walking segments from kinematic reconstructions, extracts gait events (foot contacts), and computes spatiotemporal metrics, joint kinematics, and the Gait Deviation Index (GDI). It works with both multi-camera (MMC) and monocular (PBL) data.

Schemas: project_body_models_gait_cycles (walking detection, gait events) and project_gdi (GDI computation)

Source: BodyModels/body_models/datajoint/gait/ and BodyModels/body_models/biomechanics_mjx/gait/


Finding Valid Walking Segments

Walking segments are detected by GaitTransformer (MMC) and GaitTransformerMonocular (PBL). Both use a GaitTransformer neural network + Kalman filter pipeline to identify when a person is walking.

Detection Pipeline

  1. Keypoint extraction from sites (17 keypoints mapped to GastNet order)
  2. GaitTransformer inference produces 8-channel gait phase output
  3. Kalman filter smoothing refines phase estimates, produces error signal
  4. Walking probability = prob_err * prob_speed where:
    • prob_err = exp(-errors * 2) (Kalman filter prediction error)
    • prob_speed = sigmoid of phase velocity * consistency with median
  5. Hysteresis thresholding (0.35/0.65) + median filter deglitching
  6. Minimum segment length filter (>100 frames)

Querying Walking Segments

from body_models.datajoint.gait.gait_cycles_dj import GaitTransformer, GaitTransformerMonocular

# Get the longest walking segment for a trial
key = {
    'participant_id': '104', 'session_date': date(2023, 7, 21),
    'kinematic_reconstruction_settings_num': 137,
    'transformer_method_name': 'default',
}

# WalkingSegment part table has aggregate stats per segment
segments = (GaitTransformer.WalkingSegment & key).fetch(as_dict=True, order_by='num_frames DESC')
best_segment = segments[0]  # Longest segment

# Aggregate stats available per segment:
#   num_frames, cadence, velocity,
#   stance_left, stance_right, ss_left, ss_right, dst,
#   length_left, length_right, width_left, width_right,
#   cycles_left, cycles_right (interpolated gait cycles, 100 timepoints each)

Filtering for Walking Trials

from multi_camera.datajoint.annotation import VideoActivity

# Only trials annotated as walking
walking_trials = (
    GaitTransformer
    & (VideoActivity & {'video_activity': 'Overground Walking'})
    & {'kinematic_reconstruction_settings_num': 137}
)

# From a specific project
from multi_camera.datajoint.multi_camera_dj import MultiCameraRecording
walking_controls = walking_trials & (MultiCameraRecording & {'video_project': 'GAIT_CONTROLS'})

Monocular Walking Segments

from portable_biomechanics_sessions.session_annotations import VideoActivity as PBLVideoActivity

monocular_walking = (
    GaitTransformerMonocular
    & (PBLVideoActivity & {'video_activity': 'Overground Walking'})
    & {'monocular_reconstruction_settings_num': -108}
)

Gait Events

GaitTransformer stores four event arrays as longblobs:

FieldDescription
left_downLeft foot initial contact timestamps
left_upLeft foot toe-off timestamps
right_downRight foot initial contact timestamps
right_upRight foot toe-off timestamps
events = (GaitTransformer & key).fetch1()
left_down = events['left_down']    # numpy array of timestamps
right_down = events['right_down']

# Filter events to longest walking segment
timestamps = (KinematicReconstruction.Trial & key).fetch1('timestamps')
start, end = (GaitTransformer.WalkingSegment & key).fetch(
    'segment_start', 'segment_end', order_by='num_frames DESC', limit=1
)
t0, te = timestamps[start[0]], timestamps[end[0]]
valid_left_down = [t for t in left_down if t0 <= t <= te]

Gait Cycle Definition

A gait cycle runs from one foot contact to the next same-foot contact:

  • Stride = foot_down[i] to foot_down[i+1] (one full cycle)
  • Stance phase = foot_down to foot_up (foot on ground)
  • Swing phase = foot_up to next foot_down (foot in air)
  • Step = contralateral foot_down to ipsilateral foot_down

Spatiotemporal Metrics

Step-by-Step Metrics (Steps Part Table)

GaitTransformer.Steps stores per-step metrics:

FieldDescription
sideLeft or Right
step_timeTimestamp of the step
velocityWalking velocity (m/s) at this step
cadenceSteps/min at this step
ss_leftLeft single support (%)
ss_rightRight single support (%)
dstDouble support time (%)
lengthStep length (m), NULL for monocular
widthStep width (m), NULL for monocular
steps = (GaitTransformer.Steps & key).fetch(as_dict=True)

Computing Detailed Metrics from Raw Data

For more detailed analysis beyond the stored tables, use gait_analysis.py:

from body_models.datajoint.gait.gait_analysis import compute_gait_analysis, summarize_gait_metrics

results = compute_gait_analysis(key)
# Returns dict with: timestamps, reoriented_qpos, reoriented_joints, events_raw, metrics

# metrics contains DataFrames:
#   step_metrics: step_time, step_length per step
#   gait_cycle_metrics: cycle_time, stride_length, walking_speed per cycle
#   phase_metrics: stance%, swing% per cycle
#   joint_rom: ROM per joint per cycle (radians)
#   knee_swing_rom: knee ROM during swing only
#   ankle_at_foot_down: ankle angle at each foot contact

summary = summarize_gait_metrics(results)
# Returns DataFrame with Mean/Std per side + asymmetry indices

Or use the standalone function (does not require GaitTransformer table):

from body_models.biomechanics_mjx.gait.gait_metrics import compute_gait_analysis

results = compute_gait_analysis(key, remove_first_cycle=True)
# metrics dict: step_metrics, phase_metrics, cycle_rom, phase_rom, ankle_at_foot_down

Metric Definitions

Step Metrics

MetricDefinitionUnits
Step lengthDistance between opposite heels at foot contactm or mm
Step widthLateral distance between heels at foot contactm or mm
Step timeTime between contralateral contactss
Stride lengthPelvis displacement over one gait cyclem
Cycle timeDuration of one full gait cycles
Walking speedStride length / cycle timem/s
CadenceSteps per minute derived from phase velocitysteps/min

Phase Metrics

MetricDefinitionNormal Range
Stance phase% of stride foot is on ground~60%
Swing phase% of stride foot is in air~40%
Single support% of stride only one foot on ground~40%
Double support% of stride both feet on ground~20%

Joint Angles Analyzed

JointIndex in qposDescription
hip_flexion_r/lvia ForwardKinematicsHip flexion/extension
hip_adduction_r/lvia ForwardKinematicsHip adduction/abduction
hip_rotation_r/lvia ForwardKinematicsHip internal/external rotation
knee_angle_r/lvia ForwardKinematicsKnee flexion/extension
ankle_angle_r/lvia ForwardKinematicsAnkle dorsiflexion/plantarflexion

Asymmetry Index

asymmetry = |left_mean - right_mean| / max(left_mean, right_mean) * 100

Computed for: step time, step length, cycle time, stride length, walking speed, stance phase, hip flexion ROM, knee angle ROM.


Gait Deviation Index (GDI)

GDI quantifies overall gait pathology by comparing joint angle patterns to a control group using PCA.

Schema: project_gdi

Tables

TableDescription
GDIJointsLookupDefines which joints to use (method 0 = 13 joints: pelvis tilt/list/rotation + bilateral hip/knee/ankle)
GDICyclesMethodLinks GaitTransformer trials to GDI computation (MMC)
GDICyclesStores parsed gait cycles per trial: parsed_cycles_right, parsed_cycles_left (Joint x 50 timepoints x Steps)
GDICyclesMethodMonocularLinks GaitTransformerMonocular trials to GDI (PBL)
GDICyclesMonocularSame as GDICycles for monocular data

Computing GDI

from body_models.biomechanics_mjx.gait.gdi import get_gdi_matrix_session, pca_gdi

# Get control group matrix (aggregate per session)
ctrl_keys = (GDICyclesMethod & {'video_project': 'GAIT_CONTROLS'}).fetch('KEY')
ctrl_matrix, ctrl_session_keys = get_gdi_matrix_session(ctrl_keys)

# Get treatment group matrix
trtmt_keys = (GDICyclesMethod & {'video_project': 'PROSTHETIC_GAIT'}).fetch('KEY')
trtmt_matrix, trtmt_session_keys = get_gdi_matrix_session(trtmt_keys)

# Compute GDI scores
gdi_ctrl, gdi_trtmt, ctrl_pca, trtmt_pca, n_components = pca_gdi(ctrl_matrix, trtmt_matrix)
# GDI ~ 100 = normal, lower = more pathological (each 10-point drop = 1 SD from normal)

GDI Interpretation

GDI ScoreInterpretation
>= 100Normal gait pattern
90-100Mild deviation
80-90Moderate deviation
< 80Severe deviation

Walking Segment Validation

Stride Length Filter

gait_analysis.py filters cycles by minimum stride length (default 0.1m) using pelvis displacement. It also discards the first valid cycle per side to remove gait initiation artifacts.

# Internal logic in _filter_events_by_valid_cycles:
# 1. For each foot-down-to-foot-down cycle, compute pelvis displacement
# 2. Keep cycles where displacement >= 0.1m (walking forward)
# 3. Drop first valid cycle per side (initiation artifact)

Quality Filtering with MultiTrackingDetection

from multi_camera.datajoint.quality_metrics import MultiTrackingDetection

# Filter for trials with no tracking breaks
good_track = (MultiTrackingDetection & "top_down_method=12 and reconstruction_method=2") & "num_breaks=0"
clean_gait = GaitTransformer & good_track

GaitAnalytics (Per-Step/Cycle Detail)

Schema: idjuraskovic_gait_analytics Source: BodyModels/body_models/datajoint/gait/gait_analytics_dj.py Branch: gait_metrics_rebase (in development, not merged to main)

GaitAnalytics stores detailed per-step and per-cycle metrics in normalized part tables. Preferred over StepMetrics when available — provides individual step/cycle data rather than trial-level aggregates.

Table Structure

GaitAnalytics (master) → KinematicReconstruction.Trial, steps_count

Part TablePrimary DataKey Fields
StepsPer-step spatiotemporalside, step_length, step_width, step_time, t_down, t_up
PhaseMetricsPer-cycle phase breakdownside, phase (stance/swing/single_support/double_support), percent_of_stride, duration_s
CycleROMJoint ROM per gait cycleside, joint, rom, min_angle, max_angle
PhaseROMJoint ROM per phaseside, joint, phase (stance/swing), rom
AnkleAtFootDownAnkle at contactside, ankle_angle, foot_down_time

Querying GaitAnalytics

from body_models.datajoint.gait.gait_analytics_dj import GaitAnalytics

key = {'participant_id': '104', 'session_date': date(2023, 7, 21),
       'kinematic_reconstruction_settings_num': 137}

# Per-step data
steps = (GaitAnalytics.Steps & key).fetch(as_dict=True)

# Phase breakdown per cycle
phases = (GaitAnalytics.PhaseMetrics & key).fetch(as_dict=True)

# Joint ROM per cycle
rom = (GaitAnalytics.CycleROM & key).fetch(as_dict=True)

Utility Functions

from body_models.utils.gait_analytics_utils import (
    fetch_all_gait_analytics,          # All part tables as dict of DataFrames
    compute_step_summary_statistics,    # Median/std/min/max per side
    compute_all_asymmetry_indices,      # Signed: 100 * (R - L) / avg(L, R)
    compute_normative_step_statistics,  # Control group step stats
    compute_normative_walking_speed,    # Control group walking speed
    highlight_outliers,                 # Styler coloring outside normative range
)

# Fetch everything for a trial
data = fetch_all_gait_analytics(key)
# Returns: {'steps': df, 'phases': df, 'cycle_rom': df, 'phase_rom': df, 'ankle_at_foot_down': df}

# Summary statistics
summary = compute_step_summary_statistics(data['steps'], aggregation='median')

# Asymmetry (signed: positive = right > left)
asym = compute_all_asymmetry_indices(data['steps'], data['phases'], aggregation='median')

Control group filter: participant_id LIKE '6%'


Quick Reference: Imports

# Walking detection tables
from body_models.datajoint.gait.gait_cycles_dj import (
    GaitTransformer, GaitTransformerMonocular,
    TransformerMethod, TransformerMethodMonocular,
    TransformerMethodLookup,
)

# GDI tables
from body_models.datajoint.gait.gdi_dj import (
    GDICycles, GDICyclesMonocular,
    GDICyclesMethod, GDICyclesMethodMonocular,
    GDIJointsLookup,
)

# Detailed gait analysis (DataJoint-based, uses GaitTransformer)
from body_models.datajoint.gait.gait_analysis import (
    compute_gait_analysis, summarize_gait_metrics, plot_gait_analysis, StepMetrics,
)

# Standalone gait metrics (direct from KinematicReconstruction)
from body_models.biomechanics_mjx.gait.gait_metrics import (
    compute_gait_analysis as compute_gait_analysis_standalone,
    calculate_step_metrics,
)

# GDI computation
from body_models.biomechanics_mjx.gait.gdi import (
    get_gdi_matrix_session, get_gdi_matrix_steps, pca_gdi,
)

# GaitAnalytics tables (branch: gait_metrics_rebase, not merged)
from body_models.datajoint.gait.gait_analytics_dj import GaitAnalytics
from body_models.utils.gait_analytics_utils import (
    fetch_all_gait_analytics, compute_step_summary_statistics,
    compute_all_asymmetry_indices, compute_normative_step_statistics,
    compute_normative_walking_speed, highlight_outliers,
)

# Walking activity annotations
from multi_camera.datajoint.annotation import VideoActivity
from portable_biomechanics_sessions.session_annotations import VideoActivity as PBLVideoActivity

Common Mistakes

MistakeFix
Querying GaitTransformer without transformer_method_nameAdd 'transformer_method_name': 'default' to key
Using all events instead of filtering to walking segmentFilter events to [t0, te] of best WalkingSegment
Forgetting kinematic_reconstruction_settings_num=137Always specify method number
Computing metrics on gait initiationUse remove_first_cycle=True or the built-in stride-length filter
Mixing gait_metrics.py and gait_analysis.pygait_metrics.py is standalone; gait_analysis.py uses GaitTransformer events
Monocular GaitTransformer uses different settingsDefault monocular_reconstruction_settings_num=-108 for monocular

GitHub Activity Review

Periodic cross-repository review of GitHub activity, organized thematically with appropriate depth. Produces a structured report in scratch/.

Parameters

  • Period: Default 2 weeks. User can override (e.g., "last month", "since March 1").
  • GitHub user: Detect from gh auth status. User can override.
  • Orgs: Detect from gh api user/orgs. User can specify which are relevant.

Process

digraph activity_review {
  rankdir=TB;
  node [shape=box];

  discover [label="Pass 1: Discover\nFind all active repos"];
  triage [label="Pass 2: Triage\nCount commits, rank by activity"];
  gather [label="Pass 3: Gather\nParallel subagents per repo cluster"];
  synthesize [label="Pass 4: Synthesize\nThematic organization"];
  deepen [label="Pass 5: Deepen\nFill gaps, read PROGRESS files"];
  finalize [label="Pass 6: Finalize\nTeam report, cross-cutting themes"];

  discover -> triage -> gather -> synthesize -> deepen -> finalize;
}

Pass 1: Discover Active Repos

# Get orgs
gh api user/orgs --jq '.[].login'

# For each relevant org, find repos pushed in period
gh api "orgs/ORG/repos?sort=pushed&per_page=30&type=all" \
  --jq '.[] | select(.pushed_at > "SINCE_DATE") | .full_name'

# Also check personal repos
gh api "users/USER/repos?sort=pushed&per_page=20&type=owner" \
  --jq '.[] | select(.pushed_at > "SINCE_DATE") | .full_name'

Pass 2: Triage by Commit Count

For each discovered repo, count commits by the user and by others:

gh api "repos/REPO/commits?since=SINCE&author=USER&per_page=100" --jq 'length'
gh api "repos/REPO/commits?since=SINCE&per_page=100" --jq 'length'

Rank repos into tiers:

  • Heavy (20+ user commits): Full investigation with PROGRESS files
  • Moderate (5-19): Commit log analysis with theme grouping
  • Light (1-4): Brief summary
  • Others only (0 user, >0 total): Team activity section

Pass 3: Gather Data (Parallel Subagents)

Dispatch subagents in parallel. Group small repos into a single agent. For each:

  1. README (first 80 lines): gh api repos/REPO/readme --jq .content | base64 -d | head -80
  2. User's commits: gh api "repos/REPO/commits?since=SINCE&author=USER&per_page=100" --jq '.[] | "\(.commit.committer.date) \(.sha[0:8]) \(.commit.message | split("\n")[0])"'
  3. Others' commits: same query filtering select(.author.login != "USER")
  4. PRs: gh pr list --repo REPO --state all --search "created:>SINCE" --limit 20 --json number,title,author,state,mergedAt
  5. PROGRESS.md / CHANGELOG.md: gh api repos/REPO/contents/PROGRESS.md --jq .content | base64 -d | head -200

IMPORTANT: Subagents often lack Bash permissions. Gather all raw data yourself in the main conversation, then use subagents only for analysis/writing if needed.

Pass 4: Synthesize Thematically

Do NOT organize the report repo-by-repo only. The first sections should be thematic:

  1. Read all gathered data
  2. Identify cross-cutting themes (shared techniques, common goals, connected projects)
  3. Look for mono-repos, submodules, and shared libraries that tie projects together
  4. Group commits by intellectual contribution, not just repository

Common themes to look for:

  • Same technique applied across repos (e.g., contrastive learning, JIT compilation)
  • Pipeline stages (data prep -> training -> evaluation -> deployment)
  • Shared infrastructure or library development
  • Factored/disentangled representations across different model types
  • Validation against gold-standard comparators

Pass 5: Deepen (Critical -- Do Not Skip)

After the initial synthesis, actively look for gaps:

  • PROGRESS.md files: These contain versioned experimental results with quantitative outcomes, ablation tables, and methodological decisions. They are far more informative than commit messages alone. Read them for any repo with 10+ commits.
  • Submodule relationships: Check .gitmodules for mono-repos. Trace how subprojects relate to each other (shared data pipelines, common evaluation frameworks).
  • Cross-repo patterns: If the same function/technique name appears in multiple repos (e.g., "phaser", "contrastive", "STP"), trace how it was developed in one and applied in others.
  • Quantitative results: Pull specific numbers (error rates, speedups, SRM values, accuracy) -- these make the report concrete rather than vague.
  • Negative results: Architecture ablations that failed, approaches that were rejected. These are often the most interesting findings.

Pass 6: Finalize

Write the report to scratch/YYYY-MM-DD-github-activity-review/README.md.

Report structure:

# Cross-Repository Activity Report: USERNAME

**Period:** DATE_RANGE
**Organization:** ORG_NAMES

## Executive Summary
3-5 bullet points on major thrusts with commit counts.

## Thematic Analysis

### Theme 1: [Descriptive Name]
Cross-cutting narrative tying together work across repos.
Include quantitative results, key decisions, connections.

### Theme 2: ...

## Per-Repository Details

### Repo Name (N commits by user, M by others)
**What it is:** 1-2 sentences from README.
**Key work:** Grouped by feature/theme, not chronological.
**Results:** Specific numbers from PROGRESS files.

## Team Activity

| Contributor | Repos | Activity Summary |
|---|---|---|
| name | repo list | What they did |

Common Mistakes

MistakeFix
Organizing only by repo, not by themeLead with thematic sections; repos are reference
Ignoring PROGRESS.md filesThese have the real results; always check heavy repos
Missing submodule relationshipsCheck .gitmodules; mono-repos hide connected work
Shallow commit-message-only analysisRead PRs, PROGRESS files, and README for context
Skipping team member activityAlways include "others only" repos in team section
Not tracing cross-repo techniquesIf same keyword appears in 2+ repos, it's a theme
Vague summaries without numbersPull specific metrics, error rates, speedups
Single-pass reportThe deepen pass catches 30-50% of the interesting content

Golden / Parity Test Harness

Overview

Integration and parity tests compare a port (or refactor) against a reference (PyTorch model, prior implementation, multi-view ground truth). The reference outputs are "golden" data. This skill codifies the preferred way to manage that golden data across this environment, so every repo does it the same way.

The one rule that matters:

Golden data is generated, not committed. A config-driven test harness regenerates it on demand; tests/golden/ is .gitignored. The only test inputs checked into git are tiny, public-domain fixtures (e.g. one image) or a config that points at a DataJoint trial.

Committing golden NPZs (or trial-derived tests/data/) is an anti-pattern by default — it bloats the repo, goes stale silently, and hides which reference the test actually depends on. Vendor goldens only when explicitly chosen to make tests runnable offline, and document that choice (see "When committing is OK").

Why generated, not committed

  • Golden NPZs are large (component-level captures, video E2E). They bloat clones.
  • A committed golden is a frozen number with no provenance — when the reference model changes, the test passes against a stale golden and the regression hides.
  • A config that names the trial (DataJoint subject/session) or the fixture image makes the test's input explicit and reproducible from source.

Preferred input sources (in order)

  1. DataJoint trial via config (ideal for E2E/integration). The config names a standard test subject/session; the harness fetches synchronized video + ground truth from the database. Nothing trial-derived is committed. See the /pose-datajoint and /fetching-synchronized-data skills for query patterns.
  2. Checked-in public-domain fixture (fine for component/unit tests). One small image (COCO val or similar, ~200 KB) under tests/fixtures/, committed. Use for deterministic component-level parity that must run without a database.
  3. Vendored golden NPZ (last resort, must be documented). Only when the test must run with neither torch nor the database available (e.g. offline CI).

File layout

tests/
├── conftest.py              # shared fixtures, flags, load-or-generate logic
├── fixtures/
│   └── test_image.png       # public-domain input — CHECKED IN (~200 KB)
├── golden/                  # .gitignored — auto-generated, NEVER committed
│   ├── component_golden.npz # per-component: encoder, decoder layers, heads
│   └── e2e_golden.npz       # E2E outputs on the configured DataJoint trial
├── test_component_parity.py # uses checked-in fixture + component golden
└── test_integration.py      # uses DataJoint trial + e2e golden (+ benchmark)

Add to .gitignore (every repo with parity tests):

tests/golden/

Keep a tests/golden/.gitkeep if you want the directory to exist on a fresh clone, but never commit the .npz files themselves.

The load-or-generate fixture pattern

The harness generates from the reference on first run, caches the NPZ, and reuses it on subsequent reference-free runs. No standalone generate_golden.py that must be remembered and run by hand.

# tests/conftest.py
from pathlib import Path
import numpy as np
import pytest

GOLDEN_DIR = Path(__file__).parent / "golden"
FIXTURE_DIR = Path(__file__).parent / "fixtures"


def pytest_addoption(parser):
    parser.addoption("--run-comparison", action="store_true", default=False,
                     help="Run reference-comparison tests (generates golden if missing)")
    parser.addoption("--run-integration", action="store_true", default=False,
                     help="Run DataJoint integration tests")
    parser.addoption("--regenerate-golden", action="store_true", default=False,
                     help="Force re-generation even when the golden NPZ exists")


@pytest.fixture(scope="module")
def component_golden(request, test_image):
    """Load cached golden, or generate it from the reference (needs torch)."""
    path = GOLDEN_DIR / "component_golden.npz"
    if path.exists() and not request.config.getoption("--regenerate-golden"):
        return dict(np.load(path, allow_pickle=False))
    pytest.importorskip("torch", reason="Golden missing — run with PyTorch to generate.")
    outputs = _generate_component_golden(test_image)   # project-specific
    GOLDEN_DIR.mkdir(exist_ok=True)
    np.savez_compressed(path, **outputs)
    return outputs

The same shape works for e2e_golden, fed by an integration_video_frames fixture that pulls the configured DataJoint trial.

Behavior matrix

Golden existsReference (torch) available--regenerate-goldenResult
Yes—NoLoad from disk, run comparison
YesYesYesRegenerate, save, run comparison
YesNoYesSkip with clear message
NoYes—Generate, save, run comparison
NoNo—FAIL with "run with PyTorch installed to generate"

Never let a missing golden silently skip — a parity test that quietly no-ops is worse than no test. Skip only when the reference is unavailable and the golden is cached; otherwise fail loudly.

Test marking and usage

@pytest.mark.comparison
class TestComponentParity:
    def test_encoder(self, model, component_golden, test_image):
        cos = cosine_similarity(model.encode(test_image), component_golden["encoder_features"])
        assert cos > 0.9999

@pytest.mark.integration
class TestEndToEnd:
    def test_e2e(self, model, e2e_golden, integration_video_frames):
        out = model.predict(integration_video_frames[0])
        for k in ("points", "conf", "camera_poses"):
            np.testing.assert_allclose(out[k], e2e_golden[k], atol=0.01)
pytest tests/ --run-comparison                      # generates golden first time, caches after
pytest tests/ --run-comparison --regenerate-golden  # after the reference model changes
pytest tests/ --run-comparison --run-integration -s # full suite incl. DataJoint trial + benchmark

When committing a golden IS acceptable

Only when all of these hold, and it is documented in the README:

  • The test must run where neither the reference (torch) nor the database is available (e.g. an offline CI lane), AND
  • A regeneration script/command exists and is referenced next to the data (e.g. a .gitignore comment: # regenerate with scripts/capture_pt_stages.py), AND
  • The committed artifact is small. Large captures stay generated/.gitignored even in this case.

If you vendor a golden, add a README note stating why it is committed and how to regenerate it. Silent vendoring is the anti-pattern, not vendoring per se.

Anti-patterns to flag and fix

  • git ls-files shows committed tests/golden/*.npz → should be generated + ignored.
  • Committed tests/data/<subject>/… arrays derived from a DataJoint trial → replace with a config that names the trial and fetches at test time.
  • A parity test that pytest.skips silently when the golden is missing.
  • A standalone generate_golden.py that must be run by hand and is easy to forget — fold generation into the conftest.py load-or-generate fixture instead.

Guards that cannot fail

A parity suite is only worth its gates. Four shapes recur, and each produces a test that passes for the wrong reason — worse than no test, because nobody distrusts a green one. Prove every guard by MUTATION: break what it guards, watch it fail, revert.

  • Asserting a value against the constant it was derived from. assert jax.config.jax_default_matmul_precision == config.MATMUL_PRECISION is an identity when the fixture set the flag from that constant — it holds for every value, including the wrong one. Pin the required VALUE to a literal in the test, and assert the constant against it, so the constant is what is guarded rather than its plumbing.
  • A guard conditioned on the work being incomplete. assert unfinished_stages(), assert len(STUBS) > 0, "this hazard is still unwritable" — each goes inert or red the moment the port finishes, i.e. exactly when it starts mattering. Make the precondition the check's REACH, not the work remaining.
  • A set defined by subtraction, then asserted to partition. If captured = all_keys - uncaptured, then captured | uncaptured == all_keys is true by construction and cannot fail.
  • A test whose subject never enters the assertion — e.g. a "the port does not sort" test that inspects only the reference array.

See also the jit-cache hazard in jax-memory-and-retracing: a mutation absorbed by a cached trace makes any of the above look proven.

Provenance: what a stamp must cover

  • Hash the GENERATOR SOURCE into the golden, not just the inputs, and re-check it on load. A golden whose recipe is unknown cannot be reproduced later. Never backfill a stamp after the fact — that converts an honest gap into a false claim.
  • Every file the generator READS is a recipe source, including the config module it pulls constants from. A config edit that changes what a golden MEANS while its stamp still reads "current" is undetectable.
  • A stale or missing stamp should FAIL LOUDLY, not silently regenerate mid-suite: an unattended rebuild voids every gate measured from the old numbers, and on a shared GPU it is an unannounced multi-GB job.
  • Pin the capture DEVICE explicitly. "cuda" if available else "cpu" is ambient: one capture in a shell without a visible card lands on CPU, and a CPU and a CUDA capture differ exactly in reduction order — so one wrong-device golden makes its siblings' floors describe a different computation.
  • Commit the generator before trusting its output. A generator fix left uncommitted while goldens are built against it leaves HEAD rejecting every golden.

Measure the floor on every backend the suite runs on

A tolerance derived on one backend is a tolerance for that backend. Measured on one port: the CPU arm of the same comparison was 3.5x wider than the GPU arms, so a GPU-only floor is too tight for a CPU run of the same suite. The same holds across input scales — an arithmetic floor that grows with sequence length makes a constant measured at one size a gate for that size only. Derive from an envelope over the configurations the suite actually runs, and have the suite report which backend produced its numbers.

Related skills

  • /scaffold-repo — creates the tests/ layout and .gitignore (ensure tests/golden/).
  • /pose-datajoint, /fetching-synchronized-data — fetch the configured trial for E2E goldens.
  • The workspace eqx-porting skill (Step 6, parity tests) drives what to capture from the reference; this skill governs how the golden data is stored and regenerated.

Set up an investigation

Instructions

  1. Create a folder in {REPO_ROOT}/scratch/ with the format {YYYY-MM-DD}-{descriptive-name}:
    mkdir scratch/$(python -c "import datetime; print(datetime.date.today().isoformat())")-{descriptive-name}
    
  2. Create a README.md in this folder with: task description, background context, task checklist. Update with findings as you progress.
  3. Create scripts and data files as needed for empirical work.
  4. For complex investigations, split into sub-documents as patterns emerge.

Investigation Patterns

These are common patterns, not rigid categories. Most investigations blend multiple patterns.

Tracing - "trace from X to Y", "what touches X", "follow the wiring"

  • Follow call stack or data flow from a focal component to its connections
  • Can trace forward (X → where does it go?) or backward (what leads to X?)
  • Useful for: assessing impact of changes, understanding coupling

System Architecture Archeology - "document how the system works", "archeology"

  • Comprehensive documentation of an entire system or flow for reusable reference
  • Start from entry points, trace through all layers, document relationships exhaustively
  • For complex systems, consider numbered sub-documents (01-cli.md, 02-data.md, etc.)

Bug Investigation - "figure out why X happens", "this is broken"

  • Reproduce → trace root cause → propose fix
  • For cross-repo bugs, consider per-repo task breakdowns

Technical Exploration - "can we do X?", "is this possible?", "figure out how to"

  • Feasibility testing with proof-of-concept scripts
  • Document what works AND what doesn't

Design Research - "explore the API", "gather context", "design alternatives"

  • Understand systems and constraints before building
  • Compare alternatives, document trade-offs
  • Include visual artifacts (mockups, screenshots) when relevant
  • For iterative decisions, use numbered "Design Questions" (DQ1, DQ2...) to structure review

Best Practices

  • Use subagents for parallel exploration to save context
  • Write small scripts to explore APIs interactively
  • Generate figures/diagrams and reference inline in markdown
  • For web servers: npx serve -p 8080 --cors --no-clipboard &
  • For screenshots: use Playwright MCP for web, Qt's grab() for GUI
  • For external package API review: clone to scratch/repos/ for direct source access

Important: Scratch is Gitignored

The scratch/ directory is in .gitignore and will NOT be committed.

  • NEVER delete anything from scratch - it doesn't need cleanup
  • When distilling findings into PRs, include all relevant info inline
  • Copy key findings, code, and data directly into PR descriptions
  • PRs must be self-contained; don't reference scratch files

ISR Scientific Paper Writing, Pre-2026 Voice Alignment & RAE MyST Workflow (isr-paper-writing)

Use this skill whenever scaffolding, drafting, revising, critiquing, or auditing scientific manuscripts (isr single-column preprints and ieeetrans / conference submissions) across the Intelligent Sensing & Rehabilitation (ISR) research portfolio.


1. Grounding in R. James Cotton's Authentic Pre-2026 (Non-AI-Written) Voice

CRITICAL ANTI-ECHO-CHAMBER RULE: Never calibrate prose style, paragraph structure, or rhetorical transitions against 2026 manuscripts (B3D, PBL, CAMBR, FASCIA, BiomechGPT, etc.), which were co-drafted or edited with AI agents. Aligning to AI-assisted papers creates a self-reinforcing echo chamber of bold-lead bullet lists, em-dash chains, and formulaic transitions. Always anchor voice alignment strictly to R. James Cotton's pre-2026 (2013–2025) human-written papers.

1.1 Querying the Pre-2026 Voice Corpus via Local Zotero API (http://localhost:23119/api)

Before drafting or editing any section, query the local Zotero HTTP API (available on the host and tunneled into every devcontainer at http://localhost:23119/api) and filter strictly to year <= 2025:

# Fetch R. James Cotton's pre-2026 (non-AI-written) publications from local Zotero
curl -s "http://localhost:23119/api/users/0/items?q=Cotton&limit=100" | python3 -c "
import json, sys, re
for it in json.load(sys.stdin):
    d = it.get('data', {})
    if d.get('itemType') in ('attachment', 'note'): continue
    m = re.search(r'(20\d\d)', d.get('date', ''))
    yr = int(m.group(1)) if m else 0
    if 2013 <= yr <= 2025:
        print(f\"{it['key']} ({yr}) [{d.get('publicationTitle') or d.get('proceedingsTitle') or 'Preprint'}]: {d.get('title')}\")
"

# Export BibTeX directly for citation keys
curl -s "http://localhost:23119/api/users/0/items?q=Cotton&format=bibtex"
Canonical Pre-2026 Human-Written Exemplars (2013–2025)
DomainZotero Key (Year, Venue)Title & Full-Text Reference Location (if in /workspaces/Notes/Manuscripts/ or /workspaces/Manuscripts/)
Probabilistic Computation & Structured Statistics (2013–2021)23SPKLY6 (2013, PNAS)
E22PCNTX (2015, PLoS Comp Biol)
J5B3JRRF (2020, Nature Neurosci)
VF2QWPFD (2021, Nature Comms)
Trial-to-trial, uncertainty-based adjustment of decision boundaries
Improved Estimation and Interpretation of Correlations in Neural Circuits (sparse + low-rank covariance)
A neural basis of probabilistic computation in visual cortex
Revealing nonlinear neural decoding by analyzing choices
Clinical Rehab, Outcomes & Precision Rehab (2020–2024)SE8LVGCI (2020, IEEE BioRob)
EUWZZ49R (2021, Sci Reports)
GAMD4ML8 (2023–2024)
8TGRW2I6 (2024, Sci Reports)
RWKKTS5E (2024, JNER)
Smartphone Control for People with Tetraplegia by Decoding Wearable EMG
Portable, open-source solutions for estimating wrist position during reaching in people with stroke
A Causal Framework for Precision Rehabilitation (2023 - PrecisionRehab/Causal Framework for Precision Rehab.md — tag jner-submission-2025)
Validation of portable in-clinic video-based gait analysis for prosthesis users (MarkerlessMocapGaitRite)
Transforming modeling in neurorehabilitation: clinical insights for personalized rehabilitation
Markerless Mocap, Calibrated Pose & Differentiable Biomechanics (2020–2025)84ENQZZ6 (2022)
L7F5RXTR (2022, Digit Biomark)
6TWZYM4B (2022/2023)
5MZ5V29K (2023)
BYMFMV2C (2023, IEEE BHI)
35LAPNK8 (2024)
N2PK29BL (2024, IEEE BioRob)
S3BSZYID (2024/2025, ICORR)
PXQCMRJA (2025, EMBC)
PosePipe: Open-Source Human Pose Estimation Pipeline for Clinical Research
Video-Based Pose Estimation for Gait Analysis in Stroke Survivors
Optimizing MPJPE promotes miscalibration in multi-hypothesis human pose lifting
Improved Trajectory Reconstruction for Markerless Pose Estimation
Optimizing Trajectories and Inverse Kinematics for Biomechanical Analysis of Markerless Motion Capture Data (MarkerlessMocapBiomechanics)
Platypose: Calibrated Zero-Shot Multi-Hypothesis 3D Human Motion Estimation
Advancing Monocular Video-Based Gait Analysis Using Motion Imitation with Physics-Based Simulation (Biomechanical Follower)
Differentiable Biomechanics Unlocks Opportunities for Markerless Motion Capture (DifferentiableBiomechanics)
Biomechanical Reconstruction with Confidence Intervals from Multiview Markerless Motion Capture (BiomechanicalConfidenceIntervals)

1.2 Five Stylistic Pillars of Pre-2026 James Cotton Prose (vs. AI-Written Slop)

Inspecting James's human-written manuscripts (2013–2025) reveals five concrete stylistic traits that distinguish his authentic voice from LLM-generated scientific prose:

  1. Cohesive Narrative Paragraphs Over "Bold-Lead Bullet Lists":

    • How James writes (pre-2026): Every section of the Introduction, Methods, Results, and Discussion is written in complete, flowing paragraphs where each paragraph develops one unified idea. Bullet lists appear only at the end of the Introduction for the 3–4 headline contributions (or when defining a formal taxonomy in a review paper).
    • AI habit to strip: Never write Methods, Results, or Limitations as a sequence of - **Bold Phrase:** sentence fragment bullets or numbered pseudo-paragraphs. Convert bulleted body text back into connected narrative prose.
  2. Plainspoken, Direct Scientific Motivation ("What We Truly Need"):

    • How James writes (pre-2026): Before introducing any formalism, James states the practical clinical or measurement bottleneck in plain, unpretentious English:
      • "However, this does not provide the information truly needed -- how accurate is a particular measurement with a novel camera configuration on the next current participant?" (BiomechanicalConfidenceIntervals, 2025)
      • "These confidence intervals are much more useful than the quality metrics we previously used, such as the mean reprojection error. They directly reflect what we want - how accurately we are tracking joint angles and body segment positions at each moment of time." (BiomechanicalConfidenceIntervals, 2025)
      • "We intentionally recruited a diverse rehabilitation population both to ensure the system was robustly validated for all types of patients, and because this is the population we intend to apply this final system to." (MarkerlessMocapBiomechanics, 2023)
  3. Pedagogical 3-Part Loss & Equation Introductions (Brief Intuition $\to$ Equation $\to$ Term Details):

    • How James writes: Whenever introducing a loss function or optimization objective, follow a crisp 3-part rhythm:
      1. Brief intuitive explanation (1–2 sentences): State directly what the objective/loss achieves at a high level before displaying math (avoiding multi-paragraph storytelling preambles).
      2. Displayed equation: Present the clean equation, using \underbrace{...}_{\text{...}} annotations where helpful so the reader can see the role of each term at a glance.
      3. Details of the equation: Immediately after the equation, concisely explain the symbols and what each term does (moving low-level numerical/implementation minutiae to the Appendix).
    • Example from pre-2026 prose:
      • "The pose parameters predict the detected keypoint locations through two non-linear steps. First, the marker locations can be predicted in 3D space from the pose parameter using the forward kinematic model of the body, $\mathcal{M}_\beta: \boldsymbol{\theta} \rightarrow \mathbf{x}$... Next, these marker locations are projected into the image plane using the calibrated camera model, $\Pi_c: \mathbf{x} \rightarrow \hat{\mathbf{y}}^c$... Because $p(\mathbf{Y}, \mathbf{S})$ is a constant, we can maximize the Evidence Lower Bound (ELBO) to minimize the KL divergence:" (BiomechanicalConfidenceIntervals, 2025)
  4. Measured, Understated Empirical Claims (Zero Marketing Adjectives):

    • How James writes (pre-2026): Quantitative improvements speak for themselves, framed with measured scientific restraint:
      • "Our fairly low ECE scores suggest our confidence intervals are fairly well calibrated, meaning 95% of the time, the ground truth should fall within our 95% confidence intervals."
      • "We also found some indications that it is less inclined to fall into local minima than our prior deterministic method..."
      • "Fairly unsurprisingly, policies driven by keypoints from video produced greater tracking errors and errors for spatial gait parameters than policies driven by ground truth data."
    • Banned AI hype vocabulary: Strip words like "paradigm-shifting", "unprecedented", "seamlessly", "crucially", "remarkably", "delves into", "testament to", "pivotal", "synergistic", "holistic", and "underscores".
  5. Candid Operational Transparency on Negative Results, Compute Costs & Model Mismatches:

    • How James writes (pre-2026): Rather than hiding what failed or writing generic AI boilerplate ("Future work will explore larger datasets"), James candidly reports negative experimental branches, real GPU wall-clock costs, and biomechanical model simplifications:
      • "We also implemented short horizon actor-critic in Brax as an optimizer, given the claim it can produce more precise tracking for differentiable environments. However, we found that it was extremely slow to train and could not obtain good policies." (Biomechanical Follower, 2024)
      • "However, our method also has limitations. Optimizing the probabilistic representation is several times slower than our non-probabilistic implicit representation; for example, 10-20 trials can take 1-2 hours on an A100. Our synthetic testing also revealed that certain modeling discrepancies, such as errors in the calibration, can produce overly confident results by attributing too much uncertainty to the likelihood function." (BiomechanicalConfidenceIntervals, 2025)
      • "We focused on only sagittal plane joint angles as we anticipated the vast differences between our simplified humanoid model and the accurate biomechanical model used in the mocap data would be too significant of a model mismatch." (Biomechanical Follower, 2024)
Quick Reference: AI-Drafted Slop vs. Authentic Pre-2026 James Cotton Voice
DimensionAI-Drafted Slop (Rewrite Immediately)Authentic Pre-2026 James Cotton Prose
Body StructureWall of - **Bold Header:** sentence bullets in Methods, Results, and LimitationsCohesive narrative paragraphs; bullets reserved for the 3–4 Intro contributions
PunctuationHeavy em-dash chains (— ... —) and rhetorical colon headers (Key Insight: ...)Clean declarative sentences, natural subordinate clauses (although, while, because)
Math PresentationGiant 7-term $\mathcal{L}_{\text{total}} = \sum \lambda_i \mathcal{L}_i$ dump + symbol listStep-by-step inline prose walkthrough of the forward model $\to$ why naive inference fails $\to$ compact objective
Tone & Claims"Crucially, our novel framework seamlessly achieves unprecedented accuracy...""This end-to-end approach produced lower reprojection errors than a traditional multistage pipeline..."
LimitationsVague boilerplate ("Future work will expand to larger cohorts and real-time deployment")Specific physical/compute candor ("10–20 trials take 1–2 hours on an A100; calibration errors can produce overconfident posteriors by inflating the learned likelihood width")

1.3 Strict Ban on Internal Code, Config, & Lab-Notebook Jargon

Never leak internal codebase identifiers, run tags, checkpoint filenames, or config flags into manuscript prose, tables, or figure labels:

Banned Internal JargonClean Scientific Terminology
O_v48, O_v48_s0, v4.7, fits_v47Pointwise Regression (FASCIA-Pointwise), CT registration archive
whole_v47_c6p, c6b, flow_calibConditional Flow Matching (FASCIA-Flow)
humanoid_simple_arm_20261021.xml, correspondence_20261026CT-calibrated anatomical skeleton, calibrated surface-to-skeleton correspondence model
"CT skeleton" (when applied to non-CT subjects like amputees or gait patients)Anatomically calibrated reference skeleton (never imply participants had CT scans when they did not!)
dualhead_C_20261026, pseudoanchor_scratch_tgoffUnified dual-head multi-view transformer, online pseudo-anchor supervision
carry3sh, carry3shac, dvltqftCarried Gaussian decoder with spatial-hash query selection, velocity-continuity regularization
NLF v0.2.0 vs. NLF v0.3.2Report only one NLF baseline (the faithful port of the original NLF paper, called simply NLF) alongside MAMMA and CSE
unified_v4_31b_grpo_perc_nf_v2, c0 + c1 * z, M5 b3dGRPO reinforcement learning with verifiable kinematics rewards, affine calibration, monocular mesh baseline

2. Canonical RAE MyST Paper Template (templates/myst-paper/)

Every ISR manuscript across all repositories must be authored in MyST Markdown (mystmd) using the canonical template in reproducible_agent_environment/templates/myst-paper/ (also mirrored in .agents/skills/isr-paper-writing/templates/myst-paper/). Never hand-roll standalone LaTeX .tex article files.

2.1 Scaffolding a New Paper in One Command

# From any repository or devcontainer with RAE installed:
/path/to/reproducible_agent_environment/templates/myst-paper/scripts/scaffold_myst_paper.sh \
  docs/papers/<paper_slug> \
  "Full Jargon-Free Paper Title" \
  "<paper_slug>"

2.2 Canonical Directory Layout (docs/papers/<paper_slug>/)

docs/papers/<paper_slug>/
├── myst.yml                     # Project config, Eq. %s numbering, references.bib, toc
├── manuscript.md                # Dual ISR + IEEE export frontmatter + modular {include} directives
├── abstract.md                  # Structured Background / Methods / Results / Conclusions abstract
├── introduction.md              # 4-paragraph clinical/scientific hook -> gap -> unified method -> contributions
├── methods.md                   # Best unified method first, with <!-- CODE: file:line --> anchors
├── results.md                   # Primary benchmark -> physical/clinical validation -> ablations that mattered
├── discussion.md                # Synthesis + forthcoming "Limitations and Future Directions" subsection
├── acknowledgments.md           # Funding, cohort, and mandatory AI usage acknowledgment
├── appendix.md                  # Exhaustive per-bone/per-joint tables, derivations, sensitivity grids
├── references.bib               # Populated via local Zotero API (http://localhost:23119/api)
├── BUILD.md                     # Git commit provenance table + exact reproduction commands
├── build_isr.sh                 # MyST -> XeLaTeX + Biber -> 2x XeLaTeX + unresolved-citation gate
├── build_ieee.sh                # MyST -> pdflatex + Biber -> 2x pdflatex + unresolved-citation gate
├── scripts/
│   ├── generate_paper_figures.py # Deterministic Tufte serif figure generator
│   └── diff_commits_pdf.py       # Word-level redline PDF diff between any 2 Git commits/tags/WORKTREE
└── templates/
    ├── isr/                     # isrpaper.cls, isr-logo.pdf, template.tex, template.yml (single/two-col, orange/blue)
    └── ieeetrans/               # IEEEtran.cls, template.tex, template.yml

2.3 Key Build & Provenance Rules in the RAE MyST Template

  1. Modular {include} Architecture (manuscript.md):
    • Keep manuscript.md as a thin frontmatter + {include} spine (abstract.md, introduction.md, methods.md, results.md, discussion.md, acknowledgments.md, appendix.md). If a conference submission requires a separate supplementary PDF (manuscript_conf.md + supplementary.md), both include the exact same section .md files so numbers and text can never drift between ISR and conference builds.
  2. Preventing MyST .bib Culling (cp references.bib main.bib):
    • myst build --tex writes a culled main.bib. Both build_isr.sh and build_ieee.sh automatically overwrite main.bib with references.bib and copy templates/isr/{isrpaper.cls,isr-logo.pdf} into the working directory before running xelatex + biber + xelatex $\times 2$, then grep the .log to hard-fail on any undefined citation (?).
  3. Code-to-Paper Provenance Anchors (<!-- CODE: path:Lstart-Lend -->) & BUILD.md:
    • Every architectural claim, loss term, and evaluation metric in methods.md and results.md should carry an HTML comment anchoring it to the exact source file and line range (<!-- CODE: src/pkg/model.py:L120-L155 -->), and BUILD.md must record the git commit SHA and exact CLI invocations that produced every table and figure.

3. Organizational Strategy: "Best Method First $\to$ Ablations That Mattered"

Never organize a paper as a chronological lab notebook ("First we tried X, then we fixed bug Y, then in v3 we tried Z..."). Structure every manuscript around the strongest unified method first, followed by targeted ablations and lesions that isolate why each component is necessary:

  1. Introduction (1–1.25 pages, 4–5 narrative paragraphs):
    • Paragraph 1 (Clinical / Scientific Need): What measurement or control capability does rehabilitation or movement science need, and why is it inaccessible at scale?
    • Paragraph 2 (Why Existing Silos Fall Short): Contrast traditional laboratory optical mocap / CT imaging (accurate but restricted) against commodity computer vision pose/mesh estimators (scalable but biomechanically uncalibrated, lacking joint-axis fidelity, calibrated uncertainty, or physical consistency).
    • Paragraph 3 (Core Insight & Unified Method): Walk through how our unified formulation bridges this gap in plain, concrete terms.
    • Paragraph 4 (Contributions & Validation Summary): 3–4 crisp bullets summarizing the method, the validation cohort, and the headline findings.
    • Strict Introduction Hygiene Rules (No Unintroduced Acronyms/Math & No Results Data Dumps):
      • Spell Out Every Acronym on First Use: Never use an acronym (e.g., LOSO for leave-one-subject-out, SynX, FSM, CMT) before defining it in full on first occurrence in the text.
      • Zero Unintroduced Mathematical Notation in the Introduction: Never drop un-introduced parameter symbols or subscripted math ($\uparrow\kappa_{\text{SOL,GAS}}^V$, $\downarrow\alpha_{\text{SOL,GAS}}^{\text{strength}}$, $\theta_{\text{reflex}}$) into the Introduction. Explain physiological and algorithmic mechanisms intuitively in plain prose in the Introduction, and introduce formal parameter vectors ($\theta \in \mathbb{R}^P$) sequentially in the Methods where the circuit/model is defined.
      • No Quantitative "Data Dumps" in the Introduction: You may preview the study design and state the high-level conclusions in the Introduction, but never dump laundry lists of correlation coefficients ($r=0.682\text{--}0.804$), $p$-values, or unmotivated muscle-specific breakdowns (e.g., naming individual deep muscles before the reader knows why they matter). Save detailed quantitative numbers for the Results section.
      • No Overly Busy Diagrams as Figure 1: Keep Figure 1 focused and immediately legible as a high-level paper overview; move dense multi-module wiring/block diagrams to the Supplementary or graphical abstract.
  2. Methods (Lead with What We Actually Did — Never Belabor What We Didn't Do):
    • Present the complete, best-performing formulation as a cohesive pipeline from inputs to outputs in flowing narrative paragraphs.
    • Make the Loss, Targets, & Stage Rationale Unmistakable to a Naive Reader: Always state explicitly what the parameter vector $\theta \in \mathbb{R}^P$ controls before differentiating with respect to it, what exact loss function is being minimized, and what measured signals are being fit. Whenever a multi-stage procedure is used (e.g., teacher-to-student distillation), state up front why the first stage cannot be deployed directly at test time (or, if it can, apply Occam's Razor and remove the extra stage). When synthetic validation data is used, explain in Methods how synthetic controllers/trials are parameterized so the validation is falsifiable and non-circular.
    • Do Not Belabor Pruned Pathways or Replaced Precursors in Main Methods: Never spend main-text Methods paragraphs or equations explaining pathways we pruned (e.g., collinear static spindle length feedback $\kappa_i^L$), legacy four-sphere contact models we replaced, or speculative mono- vs. bi-articular arguments we did not experimentally test. Focus the main Methods 100% on what we actually did; mention pruned components in 1–2 sentences of prose in the Discussion and relegate collinearity/Fisher pruning math to appendix.md.
    • Distinguish Native Simulator Physics vs. Custom Intercessions: Clearly state what is provided natively by the physics engine (e.g., standard MJX contact/COP readout used strictly as a validation signal) versus where we interceded with custom biomechanics (e.g., a custom muscle or reflex model, which should be backed by an ablation).
  3. Results (Standard # Results Heading: Primary Benchmark $\to$ Physical/Clinical Validation $\to$ Ablations & Lesions):
    • Always use the standard filename results.md and top-level heading # Results (not # Experiments and Results or results_and_lean.md) so MyST Studio parses section keys cleanly.
    • 3.1 Primary Head-to-Head Comparison: Compare our unified method against established external baselines.
    • 3.2 Physical, Anatomical, or Clinical Validation: Validate against independent gold-standard measurements (CT holdout scans, dual AMTI forceplates, C3D optical markers, whole-body DEXA, GaitRite, or clinical scales).
    • 3.3 Ablation & Lesion Studies ("What Mattered?"): Systematically remove or swap key components (or compare against simpler formulations tested during our simplification cycle) so the reader sees empirically why each component earns its keep.
  4. Discussion & Candid Limitations (Limitations and Future Directions):
    • Include a dedicated, concrete subsection candidly exposing: (a) physical observability limits (what cannot be inferred from the modality alone), (b) training data / field-of-view coverage gaps (e.g., truncated distal forearms in clinical trunk CTs, adult body priors on infants), (c) biomechanical/sensor simplifications & compute costs (e.g., rigid prosthetic foot segments, optimization wall-clock time), and (d) how each limitation directly motivates the next step.

3.1 Direct, Punchy, Point-Driven Prose & The "3–4 Main Points Spine" (Zero Throat-Clearing)

AI drafts often eliminate bullet lists only to replace them with winding, 45-word multi-clause paragraphs that bury the point in passive throat-clearing. R. James Cotton's authentic pre-2026 (<= 2025) prose is direct, punchy, and tightly organized around a small number of explicit points:

  1. Define the 3–4 Main Points Before Writing (methods_outline.md):
    • Every paper must state its 3–4 core points (punchlines) at the top of methods_outline.md, paired with: (a) the exact dataset, subject/trial count ($N$ participants / $n$ trials), and fraction of the full cohort vs. pilot data used, (b) the primary metric/effect size supporting the point, and (c) the specific figure that visually proves it.
    • Frame engineering papers around an empirical discovery arc: "Before starting this study, we did not know how to do $X$ optimally (or whether $A$ was necessary over $B$); through targeted experiments and ablations, we now know $Y$."
  2. One Point Per Paragraph (Topic Claim $\to$ Mechanism/Evidence $\to$ Punchy Takeaway):
    • Sentence 1 (Direct Point, $<22$ words): Open every paragraph with a direct, active-voice claim or question. Cut all throat-clearing ("In order to address the aforementioned challenges associated with...", "It is important to note that within the context of...").
      • Bad (Diffuse): "In order to mitigate the sensitivity of monocular 3D human mesh recovery to partial clinician-hand occlusions during clinical spasticity assessments, we incorporate an uncertainty-aware canonical landmark weighting scheme."
      • Good (Direct & Punchy): "Clinician hands frequently occlude the patient's forearm during passive stretch exams. Weighting each canonical landmark by its predicted Laplace uncertainty prevents occluded vertices from corrupting the elbow angle trajectory."
    • Middle Sentences: Walk through the math or empirical numbers step by step without stacked participial tails ("..., leveraging X, thereby enabling Y, while simultaneously mitigating Z").
    • Final Sentence: State the concrete takeaway or transition cleanly to the next step.
  3. Ban Unnatural Framework Buzzwords, Inline Number Dumps in Prose, & Walls of Tables (ICK!):
    • Never invent pompous structural labels like "Three-Pillar Architecture", "Dual-Paradigm Synergistic Framework", or "Four-Tiered Holistic Pipeline". Describe what the code does in plain biomechanics and optimization language.
    • Put Numbers in Programmatically Generated Tables (and Figures), Not Inline Prose Dumps: There should not be tons of numbers in running text—most quantitative results belong in compact, programmatically generated tables (generated by committed scripts in docs/papers/<paper_slug>/scripts/ from committed evaluation JSONs) or data-driven figures. Running prose should focus on the scientific insight, physical mechanism, and what the comparison proves (referencing {numref}\tab-xyz`/{numref}`fig-xyz`, with at most a single headline anchor number where helpful), rather than stringing parenthetical metric dumps ((RMSE = X ± Y, r = Z, p = W vs. A ± B)`) through every sentence. This guarantees that numbers are programmatically generated and stay 100% synchronized while producing much cleaner, more readable prose.
    • Distinguish True Scientific Ablations From Trivial Bug Fixes / Early Suboptimal Choices: Not every iteration needs to be an ablation table or figure. Reserve formal ablations strictly for core architectural components where high-quality quantitative metrics are needed to prove why our final algorithm requires that component. Early engineering lessons or suboptimal parameterizations with hindsight (e.g., that an early oversized bofoot.stl foot mesh was bad) belong as at most a single concise sentence in the Discussion, never a dedicated ablation table.
    • 100% Locked-Down methods_outline.md & Zero scratch/ Code Dependencies: Every paper must maintain a 100% locked-down methods_outline.md (with exact file:line pointers, formulas, hyperparameters, and inputs/outputs matching the manuscript and production code). All code discussed in the manuscript must be migrated out of scratch/ into committed production modules (src/) or into docs/papers/<paper_slug>/scripts/ (if it only drives paper-specific evaluations, tables, or figures) and committed to git.
    • No Walls of Tables: Main-text tables must be compact ($\le 6\text{–}8$ rows, $\le 5\text{–}6$ columns) and answer one focused question. If a table exceeds 8 rows or 6 columns, convert the core comparison into a Tufte figure (slope graph, forest plot, or complexity ladder) and move the exhaustive breakdown to appendix.md.
  4. Canonical Fleet Terminology & Intellectual Lineage:
    • Use CSE (CanonicalSapiensEqx) strictly for the vision backbone that predicts dense canonical surface queries and uncertainties, and CAMBR strictly for the bilevel monocular and multi-view biomechanical reconstruction method that optimizes MHR meshes and BodyModels skeletons from canonical queries.
    • Ground CAMBR in the intellectual lineage of our prior differentiable biomechanics (MuJoCo/bilevel IK) and PBL (Probabilistic Body Localization) papers: keypoints are robust to appearance shifts, and optimizing a biomechanical skeleton through a bilevel framework yields substantially higher anatomical accuracy than feed-forward mesh regression (B3D) alone.

3.2 Point-Driven Visual Cadence: Overview (TikZ) $\to$ Qualitative Biomechanical Overlays $\to$ Tufte Quantitative & Occam's Razor Figures (Flexible, Non-Rigid)

Every figure in a manuscript must exist to prove one of the paper's main points to a reader who skims all figures before reading the text. Often in our papers it makes sense to follow a 3-tier visual cadence—while keeping this pattern explicitly non-rigid whenever a paper's specific claims align better a different way:

  1. Tier 1 — Method / Architecture Overview Figure (Often Standalone TikZ / Vector Schematic + Optional Mini-Renders):
    • Show the conceptual and mathematical pipeline at a glance (Inputs -> Canonical Queries / Latent Representations -> Physics / Optimization -> Outputs), explicitly contrasting the Primary Best Pathway against any Ablation Pathway with inline loss symbols ($\mathcal{L}{\text{NLL}}$, $\mathcal{L}{\text{prior}}$).
    • Exemplars: fig1_cambr_architecture_pathways.png (2-pane CAMBR vector schematic contrasting Pathway A vs. Ablation Pathway B), fig_fascia_flow_architecture_and_samples.png (Panel a Eulerian–Lagrangian flow diagram paired with anatomical samples), and bga_fig1_architecture_overview.png.
    • Hard Rule: Build pipeline overviews in TikZ/vector PDF or tightly audited SVG—never ship boxy matplotlib rectangle diagrams with clipped text or arrow collisions (avoid right-edge text clipping).
  2. Tier 2 — Qualitative Biomechanical Overlay & Physical Convincer Figure:
    • Immediately show that the method works visibly on real movement data across views, occlusions, or clinical conditions.
    • Mandatory Visual & Geometric Standards:
      • Camera Distortion Correction (multi_camera JAX): Project 3D meshes, skeletons, and wheels using full OpenCV radial/tangential camera distortion coefficients ($[k_1, k_2, p_1, p_2, k_3]$ and exact principal point $(c_x, c_y)$) via multi_camera's high-performance JAX camera model—and/or undistort the video images first—so overlays align pixel-accurately out to image edges.
      • Standard GPU Cycles Mesh-Over-Skeleton Recipe: Render the native forward-kinematics MHR skin as a semi-transparent pearl envelope (alpha = 0.28–0.32, roughness = 0.45), internal .stl bones with Bilateral Bone Hue Coding (Left = Steel-Blue #7FA8D4, Right = Coral-Red #D98880, Axial/Midline = Warm Ivory #E8E2D5), and CT-calibrated skull offset (skull_offset_from_mhr + gmat). Never use frame-0 posed-space cKDTree proximity skinning.
      • Three-Gate PHI & Publication-Image Policy (sheprd-publication-consent skill): (1) Verify REDCap publication-image consent (recording == "agree") via Sheprd $\to$ PosePipeAgent / 1 (eqx wK:p1) (redcap_consent skill, cached in <repo>/data_cache/publication_image_consent.json, or use public-consent ASB participants; never include disagree / not_answered clinical participants); (2) blur all visible faces even for consented participants (MHR 3D cranial projection); and (3) never commit participant images outside the specific publication figures directory (docs/papers/<slug>/figures/ or docs/manuscript/figures/).
      • Time-Locked Waveforms & Insets: Pair image overlays with 3D studio views, high-magnification anatomical insets (e.g., $2.6\times$ hand/finger or ribcage zoom), and synchronized kinematic/GRF/EMG/reflex waveforms that never crop at panel boundaries.
    • Exemplars: tremor_multiview_finger_overlay_keyframes.png, hop_test_triple_hop_skeleton_image_overlay_keyframes.png, fig_two_person_cse_mamma_contact.png, and fig_fascia_flow_architecture_and_samples.png (Panels b–d).
  3. Tier 3 — High-Density Tufte Quantitative Result & Occam's Razor Ablation Figures:
    • Present quantitative validation and complexity ablations with transparent background (transparent=True), serif typography (Palatino / GT Sectra Fine / TeX Gyre Pagella + stix math, pdf.fonttype = 42), despined axes, direct on-canvas labels, subject-level paired trajectories (N patients vs. trials to prevent pseudoreplication), and an explicit Occam's Razor Complexity Ladder whenever a simplification/ablation was tested.
    • Exemplars: fig_ssrp_occams_razor_ablation.png (M1–M5 complexity ladder showing CV $r$, BIC minimum, and paired $d_z$), fig_real_stroke_vs_nonstroke_ssrp.png (6-panel paired clinical validation), and fig1_clinical_gaitlab_emg_grf.png (8-panel EMG/GRF/CoP validation).
  4. Multi-Contrast Color Scheme Templates Across Manuscript PDF (tufte-print), Companion Interactive 3D Viewers (dark-studio), and Conference-Room TV / Slides (bright-room-tv & sralab-blue-deck in pasture/config/viz_color_themes.json):
    • Use tufte-print (transparent=True, #111111 primary ink, #333333 secondary ink, Left = #1655a1, Right = #c0392b, Measured = #0b8e93) for manuscript .svg/.pdf/.png figures.
    • Whenever generating companion interactive 3D HTML visualizers (slot1_*.html modeled on gaitlab/slot1_gaitlab_m5_flagship_viewer.html and hop_test/slot1_hop_test_m5_viewer.html) or Deckwerk presentations, wire the 1-click theme switcher (🌙 Dark | ☀️ TV Bright | 🏛️ SRALab Blue) using pasture/config/viz_color_themes.json and the June 2026 Shirley Ryan AbilityLab Brand Guidelines (PMS 2145 Deep Blue #1655a1, PMS 185 Red #ed1c2c, PMS 1505 Orange #f36c21, Custom Dark Teal #0b8e93, PMS Warm Gray 11 #6e6259). In bright-room-tv mode, strictly ban low-contrast #64748b/#94a3b8 grays, use 3.0 px plot strokes, and switch the 3D scene floor to warm alabaster (#f8fafc -> #dbe4ee) to eliminate conference-room TV mirror glare.
  5. Non-Rigidity Guardrail:
    • Never force a figure that does not serve the paper's core points. If a paper's 3–4 main points do not align with a TikZ overview + qualitative overlay sequence, adapt the figure architecture directly to the points being communicated.

4. Multi-Cycle Paper Development & Improvement Workflow (Cycles 0–5)

We develop and polish papers through an iterative, six-stage loop that tightly couples pre-2026 voice alignment, code-to-paper verification, and experimental simplification:

Cycle 0 (RAE MyST Scaffold & BUILD.md Provenance)
  │
  ▼
Cycle 1 (Pre-2026 Zotero Voice & Literature Grounding — year <= 2025 only)
  │
  ▼
Cycle 2 (Round-1 Three-Subagent Review Triad: Code/Jargon + Pre-2026 Voice + Peer-Reviewer/Simplification)
  │
  ▼
Cycle 3 (Experimental Occam's Razor Loop: Methods Smell-Test -> Fast GPU Simplification Ablation)
  │
  ▼
Cycle 4 (Round-2 Deep Audit: Code/Eval Bugs Exposed by Writing, Participant-Level DoF & Label-Blindness)
  │
  ▼
Cycle 5 (Tufte Serif Figures, Dual ISR/IEEE PDF Build, Visual Page Inspection & Redline Diff PDF)

Cycle 0 — Scaffold from RAE templates/myst-paper & Stamp Provenance

  • Initialize docs/papers/<paper_slug>/ using templates/myst-paper/scripts/scaffold_myst_paper.sh.
  • Record the baseline git commit SHA, branch, and evaluation script commands in BUILD.md.

Cycle 1 — Pre-2026 Voice Calibration & Mandatory Zotero Library Grounding (127.0.0.1:23119)

  • Query http://localhost:23119/api/users/0/items?q=Cotton&limit=100 filtered to year <= 2025 and read 1–2 closest pre-2026 human-written manuscripts by R. James Cotton (e.g., BiomechanicalConfidenceIntervals.md, Differentiable Biomechanics.md, Biomechanical Follower.md, Causal Framework for Precision Rehab.md) before drafting or rewriting prose.
  • Mandatory Zotero Library Grounding & Unambiguous Zotero Item Key (zotero_key):
    • Every reference cited in our manuscripts must be grounded in the user's Zotero library (http://localhost:23119/api/users/0/items).
    • Populate references.bib directly from Zotero (http://localhost:23119/api/users/0/items/<ITEM_KEY>?format=bibtex) and include the unambiguous 8-character Zotero item identifier on every .bib entry (zotero_key = {<ITEM_KEY>}) so every reference can be deterministically audited against Zotero (curl -s "http://localhost:23119/api/users/0/items/<ITEM_KEY>").
    • If a relevant external paper is not yet in Zotero, you must: (1) verify that the paper is 100% real via Crossref / PubMed / Semantic Scholar / arXiv (confirming exact DOI/PMID/arXiv ID, authors, title, venue, and year—never guess or synthesize BibTeX fields), (2) tag the entry with zotero_status = {pending_user_import} + verified doi = {...}, and (3) explicitly report the verified reference + DOI/URL to the user so they can add it to Zotero.

Cycle 2 — Round-1 Three-Subagent Review Triad

Spawn three parallel subagents with non-overlapping responsibilities:

  1. Subagent 1 — Code-to-Paper, Statistical-Unit & Jargon Auditor:
    • Verifies every equation, symbol, sample size $N$, and table number against the exact production Python scripts (<!-- CODE: file:line -->) and JSON evaluation outputs.
    • Strips all internal run tags (O_v48, fits_v47, carry3sh, "CT skeleton" on non-CT cohorts) and checks that external baselines are faithfully represented.
    • Code-Complexity Smell Test: Flags any Methods passage where describing the code requires listing multi-stage heuristics, magic thresholds, or unablated sub-modules.
  2. Subagent 2 — Pre-2026 James Cotton Voice, Narrative Flow & Candid Limitations Editor:
    • Explicitly compares the draft against James's pre-2026 (<= 2025) human-written papers (never 2026 AI drafts).
    • Converts - **Bold Header:** bullet-list slop in Methods, Results, and Discussion back into cohesive narrative paragraphs.
    • Rewrites cold equation dumps into step-by-step inline prose introductions, strips AI hype adjectives, enforces "Best Method First $\to$ Ablations That Mattered", moves secondary tables to appendix.md, and ensures the Limitations section is candid and specific.
  3. Subagent 3 — Adversarial Peer-Reviewer, Simplification Experiment Designer & Figure-to-Point Critic:
    • Visually inspects every figure and rendered PDF page via read_file to verify that each figure proves one unmistakable scientific claim with zero text overlaps or aspect-ratio distortions.
    • Simplification Mandate: For every complex component or multi-stage rule in Methods, asks: "Have we tested whether a simpler, single-objective formulation works just as well or better?" and designs a fast GPU ablation (15-Minute Rule).

Cycle 3 — The Experimental Occam's Razor Loop ("Don't Guess—Test It")

  1. Methods-Drafting as an Architectural Smell Test:
    • When a Methods paragraph feels clunky or apologetic ("First we apply heuristic A with threshold $\tau_1$, then for stationary tracks we switch to heuristic B..."), treat it as an architectural smell in the code, not merely a writing problem.
  2. Run the Fast GPU Simplification vs. Complexity Experiment:
    • We are not dogmatically anti-complexity—we are against untested, unearned complexity.
    • Implement the cleaner single-objective formulation alongside the complex one and run a targeted head-to-head GPU evaluation.
  3. Promote on Win/Tie, or Document as an "Ablation That Mattered":
    • If the simpler formulation matches or beats the complex one: Promote the simpler code to production, delete the dead heuristics, and present the clean formulation as our primary Method.
    • If the richer formulation wins significantly: Keep it in the primary Method and report the simpler baseline in the Ablation Study ("What Mattered?") subsection of Results so every piece of complexity is empirically justified.

Cycle 4 — Round-2 Deep Verification & Anti-Lab-Notebook Clutter Purge (Bugs & Administrative Clutter Exposed by Writing)

Writing the paper alongside the code routinely exposes both subtle evaluation bugs and "lab-notebook administrative clutter" that would never appear in a peer-reviewed journal article. Before finalizing any manuscript (or its supplement), run a dedicated Round-2 verification pass checking:

  1. Statistical Unit of Analysis & Degrees of Freedom: Never treat repeated trials, strides, or frames from the same participant as independent samples in $t$-tests or $p$-values (e.g., 59 trials across $N=12$ participants must use participant-level paired statistics or mixed-effects models with $df = N - 1$, never trial-level $df = 58$).
  2. Strict Label-Blindness at Inference: Verify in code that no evaluation metric or test-time inference path leaks ground-truth clinical labels, ground-truth body mass/height (when evaluating video-only prediction), or post-hoc per-trial bias subtraction without calling it out explicitly.
  3. Metric & Baseline Parity: Verify that baselines and our model are evaluated on the exact same frames, coordinate conventions (KAD1-IMR vs. CMR), and unit scales (mm vs. cm).
  4. Gate 5 — Mandatory Anti-Lab-Notebook & Irrelevant Administrative Detail Purge (Main Text AND Supplement):
    • Zero Raw Calendar Visit Dates (YYYY-MM-DD) or Session Logs: Never include tables or prose listing raw calendar visit dates (e.g., 2024-07-03, *2024-07-17*, TF70 2025-03-20... held only clinical tests) in either the main text or the supplement. Report clinical protocol intervals in elapsed days/weeks/months (e.g., "following a 6-week home acclimation period") and aggregate cohort flow counts ($N$ participants, $n$ visits, $n$ walking trials).
    • Zero Internal Recording IDs, Filenames, Commit Hashes, or Checkpoint Codenames: Strip raw trial recording hashes (20240915_112405_cam01, ps9n02cm, ASB_001), git commit hashes (4ebfbe9), CLI flags (--use-v49), and internal run tags (O_v48_s0, fits_v47, carry3sh, Method-137) from all prose, tables, and figure labels. Use clean participant pseudonyms (P1–P6 or TF1–TF6) and descriptive scientific names.
    • Zero Data-Cleaning / QA Diary Notes: Replace narrative data-wrangling logs ("Two trials had no accepted strides because the participant stood still, and three sessions had no multi-camera capture...") with a single standard CONSORT-style cohort accounting sentence in Methods/Results ("From 79 recorded walking trials across 6 participants, 72 met steady-state gait screening criteria (910 strides total).").
    • Mandatory Pre-Build Automated Regex Linter: Before compiling any ISR or IEEE PDF (./build_isr.sh / ./build_ieee.sh), run this automated regex check across all manuscript .md files and verify zero offending matches:
      grep -rnE \
        '(\b202[0-6]-[01][0-9]-[0-3][0-9]\b|[0-9]{4}_202[0-6][01][0-9][0-3][0-9]_[0-9]{6}|JLH000[0-9]{4}|Method[ -]137|fits_v[0-9]+|O_v4[0-9]|carry3sh|dvltqft|tf_newbase|_2026[01][0-9][0-3][0-9]\.(stl|npz|xml|json)|\b(TF|TR|KD|P0|p)[0-9]{2,3}\b|\*Status:\* PLANNED|\bTODO\b|\bTENTATIVE\b)' \
        docs/papers/<paper_slug>/*.md
      
  5. Gate 6 — Cross-Portfolio De-Duplication ("One Canonical Home Per Method"):
    • Every core mathematical formulation (ProbabilisticReconstruction TBME 2026 pointwise Laplace marginals, FASCIA CT/MRI anatomy & MhrFkOpt, CAMBR multi-view front-end & SequenceIK, GaitLabDataset Pointwise Probabilistic + Implicit Kinetic Rollout M5) has one primary owner manuscript.
    • Downstream or clinical papers (ProstheticComponentry, Spasticity, CUET, Tremor, WheelchairPropulsion, SARA-PULSE, KinTwin) must cite the upstream foundation paper in 2–4 concise sentences and focus 100% of their Methods space on their domain-specific methodological extension—never re-deriving shared pipeline equations across multiple papers.
  6. Gate 7 — Mandatory AI Usage Acknowledgment (acknowledgments.md):
    • Every 2026+ ISR manuscript must include an explicit AI usage statement in acknowledgments.md (or the Acknowledgments section of manuscript.md / sections/acknowledgments.md) acknowledging the use of Claude (Anthropic) and Antigravity for code development, running experiments, and manuscript preparation, and stating that the manuscript was extensively reviewed by the authors:

      "AI-assisted coding and research agents, including Claude (Anthropic) and Google Antigravity, were used for code development, running experiments, and assistance with manuscript preparation. The manuscript was extensively reviewed and verified by the authors, who take full responsibility for its content."

  7. Gate 8 — Mandatory Zotero Reference & Identifier (zotero_key) Audit:
    • Every .bib entry cited in the manuscript must carry zotero_key = {<8-char-Zotero-key>} verified against http://localhost:23119/api/users/0/items/<ITEM_KEY>. Any citation not yet in Zotero must be verified against Crossref/PubMed/arXiv for real DOI/metadata, tagged zotero_status = {pending_user_import}, and surfaced in a clean table of DOIs/BibTeX for the user to import into Zotero.
  8. Gate 9 — Mandatory IEEE Font Compliance (pdf.fonttype = 42, Zero Type 3 Bitmap Fonts, & pdffonts Audit):
    • IEEE Xplore / PDF eXpress strictly rejects Type 3 bitmap fonts (which Matplotlib emits by default when pdf.fonttype = 3) as well as non-embedded fonts (emb = no).
    • Every Matplotlib figure script must explicitly configure:
      plt.rcParams.update({
          "pdf.fonttype": 42,          # Embed TrueType (Type 42) outlines — NEVER Type 3 (3)
          "ps.fonttype": 42,           # Embed TrueType in PostScript/EPS
          "svg.fonttype": "none",      # Keep text as <text> elements using standard installed serif fonts
          "font.family": "serif",
          "font.serif": ["Times New Roman", "Liberation Serif", "DejaVu Serif"],
          "mathtext.fontset": "stix",  # STIX vector math glyphs (avoids Type 3 Computer Modern bitmaps)
          "text.usetex": False,        # Avoid dvipng/Type 3 TeX bitmap fallbacks
      })
      
    • Both ./build_isr.sh and ./build_ieee.sh must run pdffonts <output>.pdf after compilation and fail the build if any font has type == "Type 3" or emb == "no".

Cycle 5 — SVG-First Tufte Serif Figures, Inkscape PDF Pipeline, TikZ Policy, & Redline Diff PDF

  1. SVG-First Figure Generation (scripts/generate_paper_figures.py) & TikZ Preservation Policy:
    • Generate .svg + .pdf + .png for all programmatic figures: Save every Matplotlib / programmatic figure in all three formats (fig.savefig(f"{stem}.svg", transparent=True, bbox_inches="tight"), .pdf, and 300 DPI .png).
    • Reference .svg in MyST Markdown (![](figures/fig_name.svg)): Referencing .svg in .md files gives crisp, infinitely zoomable vector rendering in MyST Studio (http://127.0.0.1:7319) and HTML builds, while MyST's LaTeX/PDF exporter automatically converts .svg -> .pdf via inkscape (or our lightweight ~/.local/bin/inkscape CLI wrapper around rsvg-convert -f pdf) during ./build_isr.sh and ./build_ieee.sh.
    • Keep Existing TikZ Diagrams as .tikz.tex / .pdf (Do Not Convert TikZ $\to$ SVG): Standalone or inline TikZ schematics (*.tikz.tex compiled directly by pdflatex to .pdf + .png for web preview) are already native LaTeX vector graphics with Type 1/TrueType fonts. Converting TikZ -> PDF -> SVG -> Inkscape -> PDF introduces unnecessary double-conversion and can degrade math glyph embeddings or clipping boxes—leave TikZ diagrams untouched as .tikz.tex/.pdf.
    • Use serif typography (Times New Roman / Liberation Serif / DejaVu Serif, mathtext.fontset = "stix", pdf.fonttype = 42) with mathematical symbols identical to methods.md.
    • Use left-aligned 2-line panel headers (bold panel letter + claim on line 1; muted #475569 subtitle with $N$, units, and statistic on line 2), despined top/right axes, and 1:1 native pixel aspect ratios (aspect="equal") on all camera crops and 3D renders.
  2. Dual ISR & IEEE Compilation + pdffonts Audit + Page-by-Page Visual Inspection:
    • Run ./build_isr.sh and ./build_ieee.sh (ensuring inkscape or the rsvg-convert inkscape shim is on PATH), verify pdffonts reports zero Type 3 fonts and 100% emb = yes, render pages to PNG (pdftoppm -png -r 150 <paper>_isr.pdf page), and visually inspect every page (or preview live in Sheprd MyST Studio via ./sheprd myst on http://127.0.0.1:7319).
  3. Word-Level Redline Diff PDF (scripts/diff_commits_pdf.py):
    • Whenever completing an editing cycle or preparing a revision for co-author review, generate a publication-style inline word-diff PDF comparing the baseline commit/tag against the updated manuscript:
      python scripts/diff_commits_pdf.py <baseline_commit_or_tag> WORKTREE -f manuscript.md -o exports/manuscript_diff.pdf
      

Cycle 6 — Addressing Inline MyST Studio Comments (status: "addressed") & Finalization (myst-paper-finalize)

  1. Never Delete <!-- tandem-comments --> Blocks While Addressing Review Comments:
    • When addressing inline user comments stored in <!-- tandem-comments ... --> at the bottom of .md files (or queried via ./sheprd myst-comments <paper_id>), never strip or delete the comment entries during iterative drafting.
    • Instead, update each addressed comment entry in-place inside <!-- tandem-comments --> so MyST Studio renders an emerald-green addressed highlight (✓💬) over the revised prose together with your explanation:
      • Set "status": "addressed" and "resolved": true.
      • Preserve the original highlighted text in "originalQuote".
      • Update "quote" to a verbatim substring (60–180 characters) of the newly rewritten sentence in that .md file so the highlight attaches cleanly in Live Paper and RTF views.
      • Set "addressedBy" (e.g., "claude-pb124-fascia-flow (eqx w6:p3)"), "addressedAt" ("YYYY-MM-DD HH:MM"), and "agentReply" (concise 1–3 sentence explanation of the prose/figure change and any literature or metric verification).
  2. Finalizing and Stripping Comment History (myst-paper-finalize):
    • Only strip <!-- tandem-comments --> history when the user explicitly asks to clear addressed comments or finalize the manuscript for submission (see .agents/skills/myst-paper-finalize/SKILL.md):
      ./sheprd myst-comments <paper_id>                      # Inspect open & addressed comments
      ./sheprd myst-comments finalize <paper_id> --addressed # Strip only addressed comments
      ./sheprd myst-comments finalize <paper_id> --all       # Strip all comment history for submission
      

JAX-Compatible Configuration

Overview

This skill sets up typed configuration dataclasses that work cleanly with JAX tracing, Equinox modules, and tyro CLI parsing. The core challenge: JAX's jit traces code and caches compiled versions keyed on the static components of its inputs. Configs must cleanly separate dynamic values (JAX array leaves that can change without recompilation) from static values (Python values baked into the trace).

Refer to references/pytree-registration.md for the full pytree registration pattern and references/tyro-integration.md for CLI configuration with tyro.

Decision Flow

  1. Will this config be passed into a jit/filter_jit function?

    • YES → Register it as a JAX pytree (see references/pytree-registration.md)
    • NO → Plain @dataclass is fine
  2. Is the config part of an eqx.Module (model, layer, etc.)?

    • YES → Use eqx.field(static=True) on structural fields
    • NO → Use pytree registration with explicit dynamic/static field sets
  3. Does the config need CLI parsing?

    • YES → Use tyro (see references/tyro-integration.md)
    • NO → Plain @dataclass with defaults

The Golden Rules

Rule 1: Separate Static from Dynamic

Static fields (changing them SHOULD trigger recompilation):

  • int — loop bounds, array shapes, sequence lengths
  • bool — control flow switches
  • str — mode selectors, paths
  • Enum — architecture variants
  • None | float — nullable fields (structural change)

Dynamic fields (changing them MUST NOT trigger recompilation):

  • float — loss weights, learning rates, thresholds, hyperparameters
  • Anything used in pure arithmetic, not control flow

Rule 2: Wrap Scalar Floats into Arrays

CRITICAL: JAX does not properly trace bare Python float values as dynamic pytree leaves. A bare float in a pytree leaf position gets treated as a static value by the tracer, defeating the purpose of making it dynamic.

# WRONG — bare float stays static, recompiles on every change
children_dict[key] = value  # value is 0.5 (Python float)

# RIGHT — wrapped into a 0-d array, properly traced as dynamic
children_dict[key] = jnp.asarray(value, dtype=jnp.float32)

Always convert dynamic float fields to jnp.float32 arrays in the tree_flatten function. The tree_unflatten function receives JAX arrays and can pass them directly to the dataclass constructor (they behave like floats in arithmetic).

Rule 3: Validate Outside JIT

Config validation (range checks, incompatible option detection) MUST happen at startup, never inside a traced function. Call config.validate() once before entering any JIT boundary.

config = tyro.cli(MyConfig)
config.validate()  # Fail fast here, not inside jit
train(model, config)  # Now safe to pass into traced code

Quick Reference: Field Classification

Field TypeStatic/DynamicWhy
intStaticOften controls loops or shapes
boolStaticControls if branches in trace
strStaticMode selection, paths
EnumStaticArchitecture variants
floatDynamicHyperparameters, weights, thresholds
float | NoneStaticNullability is structural
tuple[float, ...]StaticLength is structural
jax.ArrayDynamicAlready a JAX type

Equinox Module Config Fields

When config values live inside an eqx.Module, use eqx.field(static=True):

class MyModel(eqx.Module):
    # Dynamic — learnable parameters (JAX arrays)
    encoder: eqx.nn.Linear
    decoder: eqx.nn.Linear

    # Static — architecture config (baked into trace)
    hidden_dim: int = eqx.field(static=True)
    num_layers: int = eqx.field(static=True)
    mode: ArchitectureMode = eqx.field(static=True)
    dropout_rate: float = eqx.field(static=True)  # Static if used in control flow

Note: In eqx.Module, even float fields that control dropout probability or similar structural behavior should be static=True. The dynamic-float rule applies to pytree-registered dataclasses where you explicitly want to tune hyperparameters without recompilation.

Common Mistakes

MistakeFix
Passing a plain @dataclass config into jitRegister it as a pytree or make it an eqx.Module
Bare float as pytree leafWrap with jnp.asarray(value, dtype=jnp.float32) in tree_flatten
Validating config inside jitMove validation to startup, before any traced calls
Using eqx.field(static=True) on a float you want to sweepRemove static=True; or use pytree registration instead
Changing an int field and expecting no recompilationInts are always static — this is correct behavior
float | None as dynamic fieldKeep it static — None vs float is a structural change
Not testing retrace behaviorWrite a test that changes dynamic fields and asserts no retrace

File Organization

src/my_package/
├── configs.py          # All config dataclasses + pytree registration
├── model.py            # eqx.Module with eqx.field(static=True)
└── train.py            # tyro.cli() entry point
scripts/
├── train.py            # tyro.cli(TrainConfig)
├── eval.py             # tyro.cli(EvalConfig)
└── sample.py           # tyro.cli(SampleConfig)

Keep all configs in a single configs.py. Pytree registration goes at the bottom of the same file, immediately after the dataclass definitions.

References

  • rae:jax-memory-and-retracing — Diagnosing OOMs, memory leaks, and JIT retracing at runtime (the diagnosis-side complement to this skill's design-time rules)
  • references/pytree-registration.md — Full pattern for registering configs as JAX pytrees
  • references/tyro-integration.md — CLI configuration with tyro + JAX pytree configs
  • enforce-guidelines/references/python-standards.md — Python coding standards
  • enforce-guidelines/references/coding-standards.md — TDD, fail-fast

JAX JIT & Fusion Hygiene: Don't Run Hot Code Eager or Under-Fused

Overview

XLA only optimizes what you hand it inside a single jit. Code outside a jit runs eager: every primitive op dispatches separately from Python, with no fusion, no kernel-launch coalescing, and full host round-trips. On a deep model this is dramatically slower than the same computation compiled once — yet it is easy to miss, because eager code works, produces correct numbers, and shows no error. The only symptom is that the GPU is starved and the wall-clock is high.

Three failure families:

  1. Eager hot path — an expensive forward (a ViT/backbone, a decode head) is called as a plain model(x) per item instead of through eqx.filter_jit. Per-op Python dispatch dominates. Worked example below: a deep ViT head called eager once per input crop; wrapping it in eqx.filter_jit was ~11x faster.
  2. Under-fused — the work is jitted, but split across many tiny jit calls (or a Python for loop calling a small jit per element), so XLA never sees enough to fuse and you pay a launch/dispatch tax per op or per iteration.
  3. Wrong boundary — jit is present but placed too deep (inside the loop body only) or too wide (across a batch axis that then OOMs). Boundary placement is a speed↔memory trade, not a free win.

This skill gives symptom → diagnosis → fix recipes for each, plus the filter_jit partition rules, static_argnums/donate_argnums, how to measure honestly, and the parity caveat (jit ≠ eager bit-for-bit).

Quick Triage Table

SymptomMost likely causeSection
Hot forward called as plain model(x) per item; GPU ~idle, wall highEager hot path — wrap in eqx.filter_jit1
for item in items: jit_small(item) — per-iteration dispatch taxUnder-fused loop — fuse body, scan, or vmap2
Dozens of @jit helpers each called once per stepToo-fine jit boundaries — hoist into one outer jit2
jit is inside the loop body only; outer glue is eager jnp opsBoundary too deep — jit the largest pure block3
Fused a batch axis with vmap, now OOMBoundary too wide — chunk it; see memory skill3, 6
Numbers shifted slightly after adding jit; a bit-equality test failsExpected: XLA fusion reorders fp — gate on the real metric5
"Is this actually faster?" unclear from one timerAsync dispatch / cold compile not controlled4
Every call recompiles; latency spikesStatic arg or shape varies → retracejax-memory-and-retracing §1

1. Eager Hot Path — Wrap the Forward in eqx.filter_jit

How it happens

An Equinox Module is callable, so out = model(image, mask) just works and returns correct values. But with no enclosing jit, every primitive inside the forward dispatches separately from Python — for a deep ViT that is thousands of host-driven kernel launches per call, no two of which XLA is allowed to fuse. The result is correct, silent, and slow.

JAX's own guidance: "give the XLA compiler as much code as possible, so it can fully optimize it" (jit-compilation guide). A bare model(x) gives XLA nothing.

Worked example — a deep ViT head called eager per item

A common anti-pattern: a deep landmark/segmentation head called eager, once per input crop, in a Python loop over items (crops, frames, detections). A representative microbench of such a head — same GPU / same process, warm, block_until_ready:

eager  model(image, mask):              ~138 ms/crop
eqx.filter_jit(model)(image, mask):      ~12 ms/crop   ->  ~11x faster

The fix is a one-liner — wrap the per-item forward:

@eqx.filter_jit
def _jit_forward(model, image, mask):
    return model(image, mask)

When the input shape is fixed across items, it compiles once and every subsequent item reuses the fused executable. filter_jit caches on the model's static structure, so a stable model object across calls compiles exactly once.

Diagnosis

  • Grep the hot path for model( / self.<submodule>( / .forward( calls that are not lexically inside an @eqx.filter_jit / @jax.jit function.
  • Look for any Module.__call__ invoked inside a Python for/while over items (crops, frames, people, cameras, hypotheses).
  • Confirm it's hot: tie it to the wall-share. Wrapping a 0.1%-of-wall helper buys nothing; wrapping the per-item forward of a phase that dominates wall is the win.

Fix

Wrap the largest pure forward in eqx.filter_jit (not the tiny inner layers). Keep the model object stable across calls so the trace is reused — don't rebuild or re-partition the module each iteration. Inputs that vary in value but not shape stay traced automatically (§4 partition rules).

2. Under-Fused — Fuse Many Small Ops / Per-Element Calls into One Block

How it happens

Two shapes of the same mistake:

  • Tiny-jit soup. Each helper is individually @jit-decorated and called once per step. XLA optimizes each in isolation and you pay a dispatch/launch tax at every boundary between them — work that one enclosing jit would fuse.
  • Per-element Python loop around a small jit. for x in xs: y = jit_f(x) drives the GPU one tiny launch at a time from Python, leaving it idle between launches (the classic "low GPU util, host-bound" profile).

JAX measured the dispatch cost directly: redefining/re-dispatching trivial jitted functions in a loop ran ~454 ms vs ~2.7 ms for the cached/fused form — a

100x gap that is pure overhead (jit-compilation guide).

Fix — pick by loop shape

  • Independent elements, same shape → vmap inside one jit. Batch the element axis so XLA fuses across it: eqx.filter_jit(jax.vmap(f))(xs). One launch, full occupancy. Caveat (§6): vmapping a heavy forward over a large batch axis materializes every element's activations at once and can OOM — e.g. a large ViT/backbone vmapped over a wide batch axis (many parallel streams/views) can exceed device memory at every batch size. Chunk the batch if it doesn't fit; see jax-memory-and-retracing.
  • Sequential dependence (state carried step→step) → lax.scan. A Python loop inside jit is unrolled into a giant graph (slow to compile, large); lax.scan compiles to one XLA While and "is useful for reducing compilation times" (lax.scan docs). Note scan's main win is compile time / graph size, not always runtime — on some hardware the rolled loop blocks a few XLA optimizations.
  • Many small jitted helpers → one outer jit. Drop the inner @jits and wrap the composite at the top. Define jitted functions once at module scope, never inside a loop or inside a per-call function/method body (def decode(...): @eqx.filter_jit def fwd(...):) — the cache keys on the function object's identity and closure, so a function-local @eqx.filter_jit or jax.jit(jax.vmap(lambda ...)) builds a brand-new function object on every outer call and recompiles from scratch (guide). Worst of all, if the outer function loops over $N_{\text{batches}}$ batches (e.g. $T=120$ frames in batches of 8), the ~9 s cold XLA compile on batch 0 is amortized over the 120 frames (9000 ms / 120 + 5 ms ≈ 80 ms/frame), masquerading as a plausible steady-state per-frame cost (~18x slower than true steady state) and rendering any warmup call of the outer function useless! Pass call-local arrays (query_points, camera intrinsics K, extrinsics) as explicit traced arguments to a module-scope @eqx.filter_jit function instead of capturing them in a function-local closure.
# WRONG — host-bound: one tiny launch per element, GPU idle between them.
results = [jit_f(model, x) for x in xs]          # xs all same shape

# RIGHT — fuse the element axis into one jit (watch peak memory, §6).
results = eqx.filter_jit(jax.vmap(jit_f, in_axes=(None, 0)))(model, jnp.stack(xs))

3. Boundary Placement — Largest Pure Block, but Mind Memory

The right boundary is the largest contiguous pure-functional block: ideally one jit wrapping preprocess-compute-postprocess, with only true I/O / host control (decode, DB, Python branching on data) outside it.

  • Too deep — jit only the innermost layer while the surrounding glue stays eager jnp ops. Each glue op dispatches from Python. Hoist the boundary outward until it encloses the whole pure block.
  • Too wide — jit (or vmap) across an axis whose fused activations don't fit. Speed and memory pull opposite ways here. Don't keep widening blindly; chunk the axis (lax.map over chunks) so peak ∝ chunk, not ∝ batch. Cross-reference jax-memory-and-retracing §"Reduce-after-fan-out".
  • Block differentiates a deep sub-forward — if the pure block trains through a large frozen network (perceptual / teacher / discriminator loss), the backward stores that network's full activation tape and the block can OOM at jit_step even on a 2x card. jax.checkpoint (remat) the sub-forward to recompute it in backward. See jax-memory-and-retracing §"Backward-tape OOM".
  • Value-dependent control flow can't go inside jit directly — restructure with jnp.where/lax.cond, or keep that branch outside and jit the two pure arms.

4. filter_jit Partition Semantics, static_argnums, donate

What eqx.filter_jit does

"All JAX and NumPy arrays are traced, and all other types are held static", at the PyTree-leaf level — so a single argument can mix traced array leaves and static non-array leaves (Equinox transformations). This is why you wrap an eqx.Module directly: its array params are traced, its static structure (and any eqx.field(static=True) config) is cached as part of the compile key.

Consequences to internalize:

  • A Python scalar that varies per call is static → it retraces on every new value. To make it traced, pass jnp.asarray(x). (This is the #1 retrace leak — see jax-memory-and-retracing §1.)
  • eqx.field(static=True) is for genuine config only. Making a JAX array static is almost always a bug; it bakes the array's values into the compile key and recompiles when they change. Keep weights/data as normal (traced) leaves.
  • Keep the model object identity/structure stable across calls so the cached trace is reused. Re-partitioning or rebuilding the module each step defeats it.

static_argnums / static_argnames (plain jax.jit)

Mark an argument static to allow value-based control flow inside the function — but "JAX will have to re-compile the function for every new value", so it "only [works] if the function is guaranteed to see a limited set of static values" (guide). Use it for a handful of fixed configs; never for a per-frame index or per-call count.

donate_argnums / buffer donation

If an input is not needed after the call and matches an output's shape and dtype, donate its buffer so XLA reuses that memory for the output — "This will reduce the memory required for the execution by the size of the donated buffer" (buffer donation). The hard rule: "It is not allowed to donate a buffer that is used subsequently" — reuse raises an error. Donation is positional-args only, and follows pytrees. Best fit: in-place-style state carried through a step (optimizer/tracker state) where the old buffer is dead after the update. eqx.filter_jit(donate="all") / "all-except-first" apply this with leaf-level filtering.

5. Parity Caveat — jit ≠ eager Bit-for-Bit

Adding jit changes the numbers slightly. XLA fuses and reorders floating-point ops, so a jitted forward is not byte-identical to the eager one. Typical magnitude for a deep vision forward: outputs shift by ~1e-4–1e-3 in their native units (e.g. sub-pixel on normalized coordinates, ~1e-3 on probabilities) — negligible for the task, fatal for a == test.

Therefore:

  • Do not gate a jit change on bit-equality — it will fail for a correct change. An "it'll be bit-equal" expectation is usually wrong.
  • Gate on the metric that matters end-to-end — your task's accuracy/error measure (e.g. a reprojection error, an MPJPE, a downstream consumer's values) — within a tolerance, not ==.
  • Flag any jit change that feeds a downstream exact-match assertion or a byte-identical stored/cache key; loosen those to metric tolerances.

6. How to Measure (so the win is real, not noise)

Eager-vs-jit and fused-vs-unfused A/Bs are only trustworthy if you control the confounds — same hardware, warm, synchronized:

  1. Same GPU, same process. Never compare GPU-A eager vs GPU-B jit (arch/clock confound). Run both arms back-to-back on one card.
  2. Warm, post-compile. The first call pays a one-time XLA compile (seconds to minutes). Do a warmup call, then time. Report warm and cold separately.
  3. block_until_ready() — mandatory. JAX dispatch is async; without it you time the dispatch, not the compute (guide).
    jax.block_until_ready(jit_f(model, x))          # warmup + compile
    t0 = time.perf_counter()
    for _ in range(reps):
        out = jax.block_until_ready(jit_f(model, x))
    ms = (time.perf_counter() - t0) / reps * 1e3
    
  4. Watch GPU util as the inefficiency signal, not the score. Low util on a high-wall phase = host-bound / under-fused (the thing to fix); the score is still wall-clock. (nvidia-smi dmon, or a profiler trace via jax.profiler.trace → TensorBoard for a per-op breakdown.)
  5. Confirm zero compilations during the timed window (compiles_during_timed == 0) — Doing a warmup call sync(fn()) before for _ in range(reps): sync(fn()) does not guarantee that the timed calls are warm if fn() internally builds a function-local @eqx.filter_jit or retraces! Attach a programmatic jax_log_compiles logging handler (_CompileCounter) and assert compiles_during_timed == 0 over the timed repetitions. Repeated compiles mean a retrace or function-local @jit rebuild, not steady-state compute (→ jax-memory-and-retracing).

Pre-flight Checklist for a Hot Inference or Training Path

  • The expensive forward is inside eqx.filter_jit / jax.jit, not eager
  • No Module.__call__ / heavy op in a bare Python per-item loop
  • One jit wraps the largest pure block; defined at module scope (never inside a per-call function/method body, loop, or fresh lambda/partial)
  • Independent same-shape elements vmapped (chunked if the batch OOMs)
  • Sequential state loops are lax.scan, not Python-unrolled; state-independent gathers, goal encodings, and PRNG splits are hoisted outside lax.scan as xs (rae-jax-profiling §11)
  • Multi-step training/collection loops fuse $K \in [20, 100]$ iterations + on-device ring-buffer updates (dynamic_update_slice_in_dim) inside one @eqx.filter_jit(donate="all") lax.scan (rae-jax-profiling §12)
  • Mixed-precision MLP hidden layers set preferred_element_type=jnp.bfloat16 (not float32) so backward cotangent GEMMs (dX, dW) stay on bf16 tensor cores while master weights, normalizers, state integrators, heads, and losses stay float32 (rae-jax-profiling §8)
  • Kinematic tree Jacobians use @jax.custom_jvp with a static binary ancestor mask and Precision.HIGHEST on $3\times 3$ matmuls rather than jax.jacfwd through the tree recurrence (rae-jax-profiling §9)
  • No per-call Python scalar crossing the jit boundary (pass jnp.asarray)
  • static=True only on real config; weights/data stay traced
  • Benchmarked warm, same-GPU, same-process, with block_until_ready
  • Gated on your task metric within tolerance, NOT bit-equality
  • Programmatic compile counter (jax_log_compiles + jax.monitoring listener on "/jax/core/compile/backend_compile_duration") asserts compiles_during_timed == 0 during warm timing

Related Skills

  • rae-jax-profiling — canonical RAE operational checklist covering preferred_element_type=jnp.bfloat16 backward GEMMs (§8), @jax.custom_jvp analytic tree Jacobians with Precision.HIGHEST (§9), jax.monitoring pmap recompile detection & sharding-flip fixes (§10), loop-invariant lax.scan hoisting & 1-GEMM unrolled BPTT weight gradients (§11), and multi-iteration lax.scan program granularity + persistent cache hygiene (§12).
  • jax-memory-and-retracing — the complement to this skill. Fusion buys speed but raises peak memory (a wide vmap/jit materializes all activations at once → OOM; chunk it) and a mis-set boundary can retrace (a static scalar/shape that varies). Use that skill when fusing makes a path OOM or recompile, and for the no-retrace assert_max_traces regression test that locks the win in.
  • jax-config — static/dynamic config separation and pytree registration; the design-time complement that prevents most static-vs-traced boundary mistakes.

JAX Memory & Retracing: Diagnose and Avoid OOMs, Leaks, and Recompiles

Multi-device runs add their own OOM and hang modes — unreplication residency (an "unreplicated" tree that is still resident on every device), collective deadlocks, and schedule scaling. If the OOM lands at a checkpoint, validation or rendering boundary, or the process is alive but doing no work, start at jax-multi-gpu.

Overview

Three failure families account for most "mysterious" crashes in long-running JAX/Equinox programs:

  1. Retrace leaks — a jit'd function silently recompiles on every call, and every compiled variant is retained, holding device and host memory.
  2. Host-RAM OOM — Python-side accumulation (frames, masks, caches) grows until the Linux OOM-killer SIGKILLs the process with no traceback.
  3. GPU OOM — the device allocator fails, either loudly (RESOURCE_EXHAUSTED) or disguised as an unrelated error raised mid-operation.

This skill gives symptom → diagnosis → fix recipes for each, plus the compilation-cache, CUDA/cuDNN packaging, and precision-tiering hazards that feed them.

Quick Triage Table

SymptomMost likely causeSection
Every call to a jit'd fn is slow; occasional 10-100x latency outliersRetracing (static Python scalar, changing shapes, or function-local @jit)1, 2
Batch/sequence helper reports ~50–100 ms/frame "warm" when the inner model is ~5 ms/frameFunction-local @eqx.filter_jit / jax.jit(vmap(lambda ...)) rebuilt on every outer call, amortizing a multi-second compile over the batch loop1, 2
GPU memory grows step over step inside a "fixed-size" loopRetained compiled executables from retraces, or an unbounded feature cache1, 2
Process disappears mid-run, log ends abruptly, no Python tracebackHost OOM-killer (SIGKILL) — usually unbounded host-side accumulation4
XlaRuntimeError: RESOURCE_EXHAUSTEDGPU OOM — JAX raised it; catchable and diagnosable5
OOM at jit_step/first backward; worsens with more views/loss terms; still OOMs on a 2x-bigger cardUn-checkpointed backward tape through a large sub-forward (perceptual/teacher/discriminator) in the loss5
OOM during model loading, error blames a checkpoint leaf / tree mismatchDevice allocator died mid-deserialize; wrapped as a structure error5
Worked yesterday; today jit'd fns fail with version-mismatch / deserialize errorsStale persistent compilation cache after a cuDNN/driver/jaxlib change3
libcudnn load errors or cudnn-graph build failures after adding a dependencyTwo cuDNN wheels (cu12 vs cu13) colliding, or cuDNN below what jaxlib needs6
Memory roughly doubled with no code changejax_enable_x64 flipped on (float64 default)7

1. Retracing / Recompilation Leaks

How the leak happens

jax.jit (and eqx.filter_jit) cache compiled executables keyed on the function object identity plus the static parts of the call: array shapes, dtypes, pytree structure, and every non-array Python value. Three common ways to leak compiles:

  • Python scalar treated as static. eqx.filter_jit treats anything that is not a JAX/NumPy array as static — including plain Python ints such as a frame index, sequence length, or count. Every new value → a fresh trace, a fresh XLA compile, and a new retained executable. (jax.jit does the same for static_argnums arguments, and for any int that changes a shape.)
  • Pytree leaf whose shape changes between calls. E.g. a detector returns (N_kept, H, W) where N_kept depends on a score threshold; every novel N_kept recompiles all downstream jit'd consumers (resize, IoU, matching).
  • Function-local @eqx.filter_jit / @jax.jit rebuilt on every outer call (and hidden by inner-loop amortization). A sequence or batch helper (def decode(head, images, K, ...):) defines @eqx.filter_jit def fwd(...) or solve = jax.jit(jax.vmap(lambda ...)) inside its body to capture call-local variables (qp, k_jax) in a closure. Within a single decode() call over $T=120$ frames ($15$ batches of $8$), batch 0 pays a full XLA compile (~9 s) and batches 1..14 hit the local cache (~4.5 ms/frame). Averaged over $T=120$ frames, decode() reports ~80–85 ms/frame — hiding a 9-second cold XLA compile inside what looks like a plausible per-frame forward time (~19x slower than true steady state)! Worse, as soon as decode() returns, the local fwd is garbage-collected, so every subsequent call to decode() (every video clip, trial, or "warm" benchmark repetition) builds a fresh function object and recompiles from scratch.

Each retained executable holds device memory (baked-in constants, workspace) and host memory (the lowered program). In a per-frame loop this looks exactly like a memory leak — GPU usage stair-steps upward until OOM — and each new variant also costs hundreds of ms to seconds of compile time, visible as latency spikes.

Symptoms

  • Per-step time never reaches a fast steady state, or shows sporadic ~10-100x outliers on specific iterations.
  • GPU (and host) memory grows monotonically in a loop that should have a fixed working set.
  • JAX_LOG_COMPILES=1 prints a compilation line on calls after the first.

Fixes

  • Make varying values traced, not static. Pass jnp.asarray(x) instead of a bare Python scalar. If the value feeds Python control flow, restructure with jnp.where / lax.cond masks so the trace is value-independent.
  • Keep leaf shapes static: pad to a fixed maximum + validity mask. Instead of returning (N_kept, ...), return top-K with fixed K = N_MAX and a (N_MAX,) bool valid mask; downstream ops gate results on valid so padded slots are inert. Select top-K in pure JAX (jnp.argsort(-scores)[:N_MAX]), not np.nonzero on host.
  • Never branch shapes on Python ints. Anything like x[:n] or jnp.zeros((n, d)) with n varying per call must either be bucketed (round n up to a small set of fixed sizes, e.g. (8, 16, 32)) or padded to a max. Bucketing bounds compiles to the bucket count.
  • State objects carried across steps must have static shape. A tracker/accumulator pytree passed in and out of a jit'd step must keep identical leaf shapes every step — preallocate fixed-capacity slots and a ring-buffer index rather than appending.
  • Define @eqx.filter_jit / @jax.jit functions once at module scope (or once in __init__), never inside a per-call function or method body. Pass call-local arrays (query_points, camera intrinsics K, extrinsics) as explicit traced arguments instead of capturing them in a function-local closure.
  • If you intentionally discard a set of compiled variants (e.g. after a resolution change), call jax.clear_caches() to release them.
# WRONG — num_frames is a Python int: filter_jit treats it as static,
# so every distinct value compiles (and retains) a new executable.
@eqx.filter_jit
def step(model, frame, num_frames):
    ...

# RIGHT — traced 0-d array; one compile covers all values.
step(model, frame, jnp.asarray(num_frames))
# WRONG — output shape depends on data: every novel count retraces consumers.
keep = np.nonzero(scores >= threshold)[0]
return masks[keep]

# RIGHT — fixed (N_MAX, H, W) + valid mask; shape never changes.
order = jnp.argsort(-scores)[:N_MAX]
return masks[order], scores[order] >= threshold
# WRONG — @eqx.filter_jit defined inside decode() to close over qp and k_jax:
# every call to decode() creates a brand-new `fwd` function object and pays a
# full ~9 s XLA compile on batch 0 (amortized across the batch loop to look like
# ~85 ms/frame instead of the true ~4.5 ms/frame steady state!).
def decode(head, images, K, boxes, valid, *, batch_size=8):
    qp, k_jax = head.query_points, jnp.asarray(K, jnp.float32)

    @eqx.filter_jit
    def fwd(model, ims, bxs, ok):
        return jax.vmap(lambda im, b, v: forward_one(model, qp, im, b, v, k_jax))(ims, bxs, ok)

    return [fwd(head.device_model, ...) for ...]


# RIGHT — hoisted to module scope with qp and k_jax passed as traced arguments;
# compiles once across the entire process lifetime.
@eqx.filter_jit
def _decode_batch_jit(model, qp, k_jax, ims, bxs, ok):
    return jax.vmap(lambda im, b, v: forward_one(model, qp, im, b, v, k_jax))(ims, bxs, ok)

1b. The Inverse: a Cached Trace Silently Absorbing a Change

Section 1 is about retracing too often. The same cache key causes the opposite failure, and it is far harder to see: jit does not retrace when you needed it to, so a change you made has no effect and the code appears to confirm it.

How it happens

The cache is keyed on the static parts of the call — shapes, dtypes, pytree structure, non-array Python values. It is not keyed on module-level constants, config values, or any other global the traced code read. Change one of those and call the function again with the same argument structure, and you get the previous executable back.

# WRONG -- the mutation is absorbed by the cache; the assertion tests nothing.
config.EPS = 1e-5                     # was 1e-6
out = model(x)                        # same shapes -> cached trace, old EPS
assert allclose(out, golden)          # PASSES. Nothing was mutated in the executable.

Where it bites hardest: mutation tests

A mutation test exists to prove a guard has teeth: break the thing it guards, watch it fail. If the mutation is absorbed by a trace, the test passes and is recorded as proven when it is inert — the worst outcome, because a test nobody distrusts is worse than no test.

Fix

Force a re-trace, then confirm the mutation is observable before trusting any arm that passes:

# Reach the undecorated body, so no cache is consulted at all.
type(model).__call__.__wrapped__(model, x)

# Or clear the caches between arms.
jax.clear_caches()

# Or build a fresh module/closure per arm, so the cache key differs.
# And prove the harness can see a mutation at all -- run one arm you KNOW must
# fail, and watch it fail, before crediting an arm that passes.

The same cache defeats memory measurement

A module baked into a closure is a compile-time constant; the same module passed as a traced argument is not. The two produce different executables and different peak-memory figures for the same computation. Measure in the calling convention you ship, or the number describes a program you do not run.

Relatedly, jax.device_get-style peak counters (peak_bytes_in_use, and torch's max_memory_allocated) are process-lifetime high-water marks. Several arms in one process all report the maximum over every arm run so far, which silently yields an identical figure for arms that differ. One arm, one process.

2. Detecting Retraces (Equinox/JAX tooling)

JAX_LOG_COMPILES=1 — locate and count compilations

JAX_LOG_COMPILES=1 python my_script.py 2>&1 | grep -c "Compiling"

Every line after warmup is a retrace. The logged function name tells you which jit boundary is unstable; correlate the timestamps with your loop index to find what changed.

eqx.debug.assert_max_traces — hard-fail on unexpected retraces

Wrap the inner function, then jit. Order matters: assert_max_traces counts traces of the function it directly wraps, so jit-then-assert would count outer dispatches (every call), not real compilations.

inner = eqx.debug.assert_max_traces(step_fn, max_traces=1)
jit_step = eqx.filter_jit(inner)
...
eqx.debug.get_num_traces(inner)  # introspect the count at any time

Gate it behind an env var so production pays zero overhead and CI can opt in:

def _maybe_guard(fn, registry, name):
    if not os.environ.get("MYPKG_ASSERT_NO_RETRACE"):
        return fn
    inner = eqx.debug.assert_max_traces(fn, max_traces=1)
    registry[name] = inner  # so tests can query get_num_traces(inner)
    return inner

self._jit_step = eqx.filter_jit(_maybe_guard(step_fn, self._guards, "step"))

A no-retrace regression test

The highest-value test in any per-frame/per-step pipeline: exercise the path that historically changed shapes or scalars, and assert the trace count stayed at 1. Run it on CPU with a synthetic model — it needs no weights.

def test_step_compiles_once_across_lengths(monkeypatch):
    monkeypatch.setenv("MYPKG_ASSERT_NO_RETRACE", "1")
    runner = make_runner()  # small synthetic config, CPU is fine
    for num_frames in (3, 5, 9):          # values that used to retrace
        runner.run(synthetic_frames(num_frames))
    assert all(n == 1 for n in runner.trace_counts().values())

assert_max_traces makes the failure loud at the moment of the second trace (with both call signatures in the error), which is far easier to debug than a downstream OOM hours later.

Gotcha — assert_max_traces on a function-local definition is a guard that cannot fail across outer calls. If a helper builds @eqx.filter_jit inside its own function body (def decode(...): @eqx.filter_jit def fwd(...)), wrapping fwd with assert_max_traces(max_traces=1) inside decode() creates a brand-new guard instance on every decode() call with its trace counter reset to 0. Within one decode() call, all batches have identical shape so the counter reaches 1 and passes — completely blind to the fact that 100 calls to decode() compiled 100 separate XLA executables! Always hoist both the @eqx.filter_jit function and its assert_max_traces wrapper to module scope, and pair them with a process-wide XLA compile counter in benchmarks and integration tests:

import logging
import jax


class _CompileCounter(logging.Handler):
    """Process-wide XLA compile counter (catches even function-local @jit rebuilds)."""

    n = 0

    def emit(self, record):
        msg = record.getMessage()
        if "Finished XLA compilation" in msg or "Compiling " in msg:
            _CompileCounter.n += 1


def enable_compile_count():
    jax.config.update("jax_log_compiles", True)
    h = _CompileCounter()
    for name in ("jax", "jax._src.dispatch", "jax._src.interpreters.pxla", "jax._src.compiler"):
        lg = logging.getLogger(name)
        lg.addHandler(h)
        lg.setLevel(logging.WARNING)
        lg.propagate = False

Assert _CompileCounter.n - c0 == 0 across all post-warmup calls or timed benchmark repetitions.

3. Persistent Compilation Cache Pitfalls

jax.config.update("jax_compilation_cache_dir", ...) persists compiled executables across processes — a big win when cold compiles take minutes. Two hazards:

  • Stale-cache poisoning. The cache key does not capture every runtime dimension — notably the cuDNN/driver runtime can change without changing the key. After upgrading cuDNN (or jaxlib pulling a different one), the cache can serve an incompatible executable, failing with deserialize / version-mismatch errors that look like corruption. Fix: delete the cache directory after any CUDA/cuDNN/jaxlib/driver change. Treat "wipe the JAX cache dir" as a standard step in upgrade SOPs.
  • Libraries must not enable it on import. Setting jax_compilation_cache_dir is a global config flip and a filesystem side effect (it silently accumulates GBs). A library that does this on import hijacks application policy. Fix: make it opt-in via an env var or explicit setup call owned by the application:
def setup_compilation_cache(cache_dir: str | None = None) -> Path | None:
    """Opt-in persistent compile cache. Off unless the caller asks."""
    cache_dir = cache_dir or os.environ.get("MYPKG_JAX_CACHE_DIR")
    if not cache_dir:
        return None
    path = Path(cache_dir).expanduser()
    path.mkdir(parents=True, exist_ok=True)
    jax.config.update("jax_compilation_cache_dir", str(path))
    jax.config.update("jax_persistent_cache_min_compile_time_secs", 1.0)
    return path

A persistent cache only pays off if nothing retraces — pin that first with the no-retrace tests from section 2, otherwise the cache just grows.

4. Host-RAM OOM: the Traceback-less SIGKILL

Recognize it

The process disappears — log ends mid-line, shell reports Killed or exit code 137 (128+SIGKILL), and there is no Python traceback. This is the Linux/cgroup OOM-killer. PYTHONFAULTHANDLER cannot help: SIGKILL is not interceptable, so faulthandler prints nothing. Do not burn time on batch-size or GPU mem-fraction knobs — this class is host RAM, not GPU.

Confirm:

dmesg -T | grep -i "killed process"        # if you have access
# Otherwise: watch host RAM during a repro
while sleep 5; do free -g | awk '/Mem:/{print strftime("%T"), $3" GB used"}'; done
# Or track the process itself:
while sleep 5; do ps -o rss= -p <PID> | awk '{print strftime("%T"), $1/1048576" GB RSS"}'; done

Monotonic growth toward the machine's limit = accumulation leak. A flat plateau that is simply too high = oversized working set (datasets, workers).

The two canonical causes

(a) Accumulating O(T) full-resolution arrays over a sequence. The natural consumer pattern for a streaming model — "append every frame's masks/frames/ features to a list" — buffers the entire video in host RAM. A single full-HD float32 mask is ~8 MB; per-object, per-frame, this caps runs at roughly a thousand frames and then SIGKILLs.

Fixes, in order of preference:

  1. Stream, don't accumulate. Consume each frame's output inside the loop (write to disk, update running stats) and let it go. Provide an iterator API so O(1)-memory consumption is the easy path.
  2. Store compactly when you must keep everything. Binary masks: threshold → np.packbits → zlib.compress is lossless, stdlib-only, and ~1000x smaller than float32; decode lazily on access. Frames/features: uint8, downsampled, or memory-mapped to disk.
  3. Bound every cache with a real budget. An "it's only a few MB" LRU becomes hundreds of GB when shard sizes grow 100x. Size caches as entries x measured_entry_size x num_workers against actual host RAM, and re-check the bound whenever data size changes. Never ship an unbounded dict cache on a data path.

(b) Compiled-executable buildup from retracing (section 1) — XLA compilation itself can also transiently spike host RAM by many GB on large models, so an almost-full host dies during a compile, which is confusing: the leak was the dataset cache, but the killer struck inside JIT.

5. GPU-OOM Diagnosis Discipline

Two distinct presentations

  • Catchable: jaxlib.xla_extension.XlaRuntimeError: RESOURCE_EXHAUSTED. JAX raised it; you get a traceback and can reason about which allocation failed.
  • Disguised: the allocator fails midway through a larger operation and surfaces as an unrelated error. Classic case: model deserialization OOMs partway through the leaf list and presents as a tree-structure / leaf-mismatch error naming whatever leaf it died on. If a "structure mismatch" appears only on busier GPUs or at lower memory fractions, suspect OOM, and check the exception's __cause__ chain for RESOURCE_EXHAUSTED.

Tell GPU from host, leak from working set

Sample both memories over time; the divergent one is your problem:

# GPU, every 5 s:
nvidia-smi --query-gpu=timestamp,memory.used --format=csv -l 5
# Host, every 5 s:
while sleep 5; do free -m | awk '/Mem:/{print strftime("%T"), $3}'; done
  • GPU grows stepwise per iteration → retained executables (section 1) or a per-step device-array cache you forgot to clear.
  • GPU flat but high, fails only at one phase → working set too large for that phase; reduce batch/resolution or free the previous phase's arrays (drop references, then jax.clear_caches() if compiled variants pin them).
  • Host grows, GPU flat → section 4.

Reduce-after-fan-out: scan/chunk, never one wide vmap

A common phase-OOM is a fan-out you immediately reduce: run the same model on N variants (augmentations, hypotheses, MC samples, ensemble members) and collapse the N outputs to a statistic (mean, std, max, top-k). The natural jax.vmap(f)(variants) materializes all N forwards' activations at once, so the peak scales with N even though the result is tiny. On a large model (a ViT/backbone × a real batch) a modest N (e.g. 8) is enough to exhaust the device — the failing allocation names the fan-out's executable and is N-proportional.

The fix is to stream the fan-out and reduce online, so only chunk forwards co-reside:

# WRONG — all N augmented forwards live at once; peak ∝ N.
outs = jax.vmap(lambda k: model_forward(augment(x, k)))(keys)   # (N, B, ...)
sigma = jnp.std(outs, axis=0)

# RIGHT — lax.map over chunks; peak ∝ chunk, result identical.
def chunk_fwd(ck):                       # ck: (chunk,) keys
    return jax.vmap(lambda k: model_forward(augment(x, k)))(ck)
outs = jax.lax.map(chunk_fwd, keys.reshape(n_chunks, chunk, *keys.shape[1:]))
sigma = jnp.std(outs.reshape(-1, *outs.shape[2:])[:N], axis=0)

chunk is a pure memory↔throughput knob — it does not change the numerical result (same N samples), so set it small enough to fit the phase and leave N at whatever the statistic needs. For a true O(1)-peak reduction (no (N, …) stack at all), carry a running accumulator through lax.scan (e.g. Welford for mean/variance) instead of stacking then reducing.

Gotcha — a chunk default >= N silently collapses to the wide vmap. If the chunking helper defaults chunk to the same value as the fan-out count (or the caller omits it), n_chunks == 1 and lax.map runs a single group over all N — i.e. the exact wide vmap you were avoiding, with no error, just the OOM. Default chunk to a small constant (2–4), and treat chunk >= N as the unguarded wide-vmap case in the docstring so callers know it OOMs large models.

Backward-tape OOM: gradient-checkpoint large frozen sub-forwards used in a loss

A loss term that forwards a large network on the prediction — a perceptual / feature loss (VGG / DINO / CLIP), a frozen teacher, a discriminator — is differentiable wrt that prediction, so reverse-mode AD must store the sub-network's entire activation tape for the backward pass. The weights being frozen does not help: freezing zeroes the weight gradient, but the input-gradient path still needs every intermediate activation. Run the sub-forward K times per step (once per view, timestep, MC sample, or loss target) and the backward-tape peak scales with K x depth x tokens, dwarfing the rest of the step.

The tell that distinguishes this from a working-set OOM: the failure strikes at jit_step (the first backward), worsens as you add loss terms or views, and does not fit even on a much larger card — a 2x GPU OOMs the same way, because the cost is the tape, not a one-off buffer. (A working-set OOM, by contrast, shrinks with batch/resolution and does fit when you upsize the card.) If a perceptual/feature loss makes a run need >2x its base memory, suspect this before reaching for a bigger GPU.

Fixes, in order:

  1. jax.checkpoint (remat) the sub-forward. Wrap the frozen network's forward so its activations are recomputed in the backward instead of stored — trading compute for memory. Peak drops from K x full-tape to roughly K x one-layer. Value-identical; the single highest-leverage fix.
  2. stop_gradient the target branch. In d(f(pred), f(gt)), f(gt) needs no tape — wrap it so only the pred branch is differentiated.
  3. Cut K. Score the term on fewer views/samples; a perceptual loss on the single matched view is usually as effective as on all of them at a fraction of the memory.
# WRONG — frozen backbone forward on every view; full tape per view -> peak ∝ K.
def feat_loss(pred, gt, mask):
    return jnp.mean(1 - cos(backbone(pred * mask), backbone(gt * mask)))
total = sum(feat_loss(pred[v], gt[v], mask[v]) for v in range(K))   # OOMs at jit_step

# RIGHT — remat the differentiated forward, stop-grad the target, score one view.
backbone_remat = jax.checkpoint(backbone)
def feat_loss(pred, gt, mask):
    f_gt = jax.lax.stop_gradient(backbone(gt * mask))               # no tape
    return jnp.mean(1 - cos(backbone_remat(pred * mask), f_gt))     # recomputed in bwd
total = feat_loss(pred[0], gt[0], mask[0])                          # K = 1

This is the backward-pass dual of the forward-peak fan-out above: §"Reduce-after-fan-out" chunks the forward activations of N parallel variants; remat recomputes the backward activations of a deep sub-forward. A loss with both (a perceptual term under a vmap over views) needs both fixes.

Standardized Cross-Project Toolchain & XLA Flags (body_models.environment_check)

Rather than scattering ad-hoc os.environ["XLA_FLAGS"] blocks across training scripts, projects in the biomechanics ecosystem share a single, pure-metadata environment checker and configurator in body_models.environment_check:

  • Before JAX backend initialization (top of entrypoints / conftest.py): Call configure_environment() (or apply_recommended_flags()) to idempotently set XLA_PYTHON_CLIENT_PREALLOCATE=false, append version-specific XLA workaround flags (such as --xla_gpu_experimental_gemm_fusion_v2=false on jaxlib >= 0.12 to prevent the Triton GEMM nested-vmap backward miscompile openxla/xla#51186), and configure partitioner workarounds (JAX_USE_SHARDY_PARTITIONER=false on jaxlib == 0.11.2).
  • Passive runtime verification (check_environment() / find_gotchas()): Automatically invoked once per process when ForwardKinematics is constructed (and callable directly by any library). Reads importlib.metadata and os.environ without importing jax or touching devices, emitting a ToolchainWarning if any known JAX / XLA / cuDNN / NCCL gotcha is active and unmitigated.

Allocator hygiene

XLA_PYTHON_CLIENT_PREALLOCATE=false     # see true usage; play nice on shared GPUs
XLA_PYTHON_CLIENT_MEM_FRACTION=0.65     # leave headroom for transients & cohabitants

Preallocation off makes nvidia-smi meaningful for diagnosis. Choose a mem fraction that leaves room for allocator transients — peak usage during loading/compile can briefly exceed steady state by 2x.

Two loading/lifecycle patterns that prevent avoidable GPU OOMs:

  • Deserialize weights to CPU first, then upload once. Loading a checkpoint directly onto the device while a randomly-initialized skeleton of the same model is still resident doubles peak device memory. Build / deserialize under jax.default_device(jax.devices("cpu")[0]), drop the skeleton, then device_put the array leaves.
  • Own your background jobs. Backgrounded GPU processes that lose their launcher frequently orphan and keep their device memory. Before relaunching, find and kill your own stale PID (never broad pkill on a shared machine); before blaming contention, check whether the squatter is yours.

6. CUDA/cuDNN Dependency Hazards

There is one libcudnn.so soname per process — two providers cannot coexist.

  • Torch pulling CUDA-13. Default GPU torch wheels can be CUDA-13 builds that depend on nvidia-cudnn-cu13, which file-collides with jax-cuda12's nvidia-cudnn-cu12 and breaks JAX's cudnn-graph paths. Fix: if torch is only used for weight conversion or host-side I/O, resolve it from the CPU wheel index (carries no cuDNN) and put the conversion deps behind an optional extra. Only co-install GPU torch when it genuinely runs on the GPU, and then match CUDA majors.
  • cuDNN floor below what jaxlib needs. JAX's declared constraint (e.g. cudnn>=9.x) can be looser than what its bundled XLA actually needs to build cudnn graphs (e.g. flash attention). The resolver then picks a too-old patch release and JIT fails at runtime. Fix: pin/floor nvidia-cudnn-cu12 (via a constraint, not a direct dep) at a version verified against your jaxlib, so fresh clones resolve a working runtime.
  • After any change in this layer, wipe the persistent compilation cache (section 3).

7. Precision / Memory Tiering

Precision is a memory knob with sharp numerical edges. Default tiering:

  • float32 is the global default. Never enable jax_enable_x64 globally: it silently doubles every array and the autodiff tape — a fit that used to OOM at 2x memory is the classic signature. If some component needs float64 (reference-equivalence tests, a numerically delicate solver), gate it behind an explicit opt-in flag scoped to that component.
  • bfloat16 for big NN forwards (backbones, encoders) is usually safe — validate once against f32 (cosine similarity / max-abs on outputs) and you halve activation memory and often gain throughput.
  • Never run ill-conditioned optimization in bf16 — second-order / line-search fits (LBFGS, IK, bundle-adjust) need f32 minimum; bf16's 8-bit mantissa destroys convergence. Tier within the program: bf16 forwards feeding an f32 optimizer is a sound and common split.

Rule of thumb: dtype choices are per-subsystem policy, set explicitly at each boundary — never a global config flip made on import.

Pre-flight Checklist for Long Sequence/Video Loops

  • All jit'd step functions defined at module scope (never inside a per-call function/method body) and guarded by a module-scope assert_max_traces(max_traces=1) + a no-retrace test over the shape-varying inputs
  • Warm benchmarks and multi-call integration tests assert compiles_during_timed == 0 via a process-wide jax_log_compiles handler (catching any hidden function-local @jit rebuilds)
  • No Python scalar that varies per call crosses a filter_jit boundary
  • Variable-count outputs padded to a static max with a valid mask
  • Per-frame outputs streamed or compressed, never appended raw
  • Every cache has an explicit, measured budget
  • XLA_PYTHON_CLIENT_PREALLOCATE=false + mem fraction with headroom
  • Persistent compile cache opt-in only; wiped after CUDA/cuDNN changes
  • x64 off globally; bf16 only on validated NN forwards, never on fits
  • Large frozen sub-forwards in a loss (perceptual / teacher / discriminator) wrapped in jax.checkpoint; target branch stop_gradient'd; scored on the fewest views/samples that suffice

Related Skills

  • rae:jax-config — static/dynamic config separation and pytree registration; the design-time complement to this skill's diagnosis recipes (a correctly-classified config prevents most retrace leaks up front).
  • eqx-porting (project skill in repos that port PyTorch models to JAX/Equinox) — porting workflow, weight conversion, and parity testing; use this skill when a ported model's inference loop OOMs or retraces.

JAX Multi-GPU: Unreplication, Collective Hangs, and Whether Sharding Pays

Overview

Data-parallel multi-device training fails in ways single-device training does not, and the failures are quiet. Three families cover most of it:

  1. Unreplication / placement bugs. Pulling one replica of a replicated pytree back to one device is the most common multi-GPU operation and the easiest to get wrong. Done naively it leaves a full copy resident on every device, permanently — or, under the modern sharding API, silently returns the wrong numbers.
  2. Collective hangs. When one replica dies, the others block forever. The process stays alive, holds its GPUs, writes nothing, and never returns an exit code.
  3. Scaling that silently changes the experiment. Every step-counted hyperparameter means something different when a step consumes N times the data.

Underneath all three: your usual instruments lie on multi-GPU. Process existence, exit codes, log tails, GPU-utilisation snapshots and even the op name in the error each report "healthy" — or point at the wrong place — for a job that has been dead for hours.

Evidence tags. Claims below are marked [M] measured (reproduced on jax/jaxlib 0.10.1, CUDA plugin, in an environment you can re-run the snippet in), [C] cited (upstream docs or source, not re-measured here), or [F] field report (observed once on one project's hardware — directionally true, do not treat the number as a constant).


0. pmap is in maintenance mode — know which API you are in

pmap is in maintenance mode — not formally deprecated (it emits no DeprecationWarning [M]), but the JAX 0.8.0 changelog is explicit [C]:

"jax.pmap is in maintenance mode and we encourage all new code to use jax.shard_map directly."

and jaxlib 0.10.1's own jax.pmap docstring opens with [M]:

"Old way of doing parallel map. Use jax.shard_map instead." "pmap is now implemented in terms of jit and shard_map." — with a pointer to the pmap migration guide.

Note the recommended successor is shard_map, not "jit + sharding" generically. JEP 14273 gives the reason that matters here [C]: "the sharding of pmap outputs can't be expressed as Shardings/OpShardings, due to pmap's stacking rather than concatenative semantics." That mismatch is the root of §1.

[M] jax.device_put_replicated and jax.device_put_sharded are already gone from jax 0.10.1; code that calls them will AttributeError. The replacement is an explicit mesh:

devices = jax.devices(); n = len(devices)
mesh = jax.sharding.Mesh(np.asarray(devices), ("devices",))
row  = jax.sharding.NamedSharding(mesh, jax.sharding.PartitionSpec("devices"))
replicated = jax.tree.map(
    lambda x: jax.device_put(jnp.broadcast_to(x[None], (n, *jnp.shape(x))), row), tree)

[M] Also present in 0.10.1 and worth knowing before you write a shim: jax.shard_map, jax.smap, jax.make_mesh, jax.set_mesh, jax.sharding.reshard, jax.sharding.make_single_device_sharding, jax.sharding.AxisType / auto_axes / explicit_axes.

Which API you are in changes the shape of the bug, not whether you have one — see §1.4. Nothing in §1 or §2 is pmap-specific: pmap now runs on the same jit+shard_map machinery, and the residency behaviour reproduces identically through a plain NamedSharding [M].


1. Unreplication: x[0] does not do what it looks like

Under pmap (or a pmap-shaped mesh layout) parameters carry a leading device axis. To checkpoint, validate, or render you need one replica on one device. The obvious code is wrong:

single = jax.tree.map(lambda x: x[0], replicated)   # WRONG

x[0] selects replica 0's values, but the output inherits the input's multi-device sharding: PartitionSpec("devices") becomes PartitionSpec() on the same mesh — i.e. fully replicated, a complete copy resident on every device in the job. It is not "uncommitted and therefore free"; the bytes are really there.

Upstream documents exactly this, in the pmap migration guide's appendix [C]:

"JAX does not track semantic replication. It does not 'know' that the shard on device 1 is identical to the shard on device 0. Therefore, when you call x[0], JAX must satisfy the strict semantics of array indexing… This triggers a global gather (or broadcast) of the entire array to all devices before slicing."

and explains why the idiom used to be free: PmapSharding "had a fast-path optimization in jax.Array's __getitem__ allowing it to return an array with a SingleDeviceSharding" — a fast path that was removed along with PmapSharding in jax 0.10 [C], because it "breaks the semantics of array indexing". So code that was correct-and-cheap under old pmap became correct-and-ruinous without a single line changing.

1.1 Prove it in twelve lines (no GPU needed)

[M] Runs on CPU with fake devices; the semantics are platform-independent.

import math, jax, jax.numpy as jnp, numpy as np
# XLA_FLAGS="--xla_force_host_platform_device_count=4" JAX_PLATFORMS=cpu
D = jax.devices(); mesh = jax.sharding.Mesh(np.asarray(D), ("d",))
row = jax.sharding.NamedSharding(mesh, jax.sharding.PartitionSpec("d"))

def resident(a):  # device bytes actually held, summed over every device it sits on
    return math.prod(a.sharding.shard_shape(a.shape)) * a.itemsize * len(a.sharding.device_set)

x = jax.device_put(jnp.zeros((len(D), 1024, 1024)), row)      # one replica per device
for name, a in (("x[0]", x[0]), ("device_put(x[0], D[0])", jax.device_put(x[0], D[0]))):
    print(f"{name:24s} logical {a.nbytes/2**20:5.1f} MiB  resident {resident(a)/2**20:5.1f} MiB "
          f" devices {len(a.sharding.device_set)}")
x[0]                     logical   4.0 MiB  resident  16.0 MiB  devices 4
device_put(x[0], D[0])   logical   4.0 MiB  resident   4.0 MiB  devices 1

jax.debug.visualize_array_sharding says the same thing in one line [M]:

x                        x[0]                device_put(x[0], D[0])
┌──────────┐             ┌───────────┐       ┌───────┐
│  CPU 0   │             │CPU 0,1,2,3│       │ CPU 0 │
├──────────┤             └───────────┘       └───────┘
│  CPU 1   │  …

1.2 The fix, and why each guard is load-bearing

def unreplicate_to_single_device(tree, device=None):
    """Take replica 0 AND commit it to one device."""
    target = device if device is not None else jax.devices()[0]

    def take_replica_zero(leaf):
        # Non-array and 0-d leaves carry no device axis. `3.0[0]` is a TypeError and
        # `jnp.float32(3.0)[0]` is an IndexError, so an unguarded tree.map over a real
        # Equinox module or config pytree crashes before it ever leaks anything. [M]
        if not hasattr(leaf, "shape") or getattr(leaf, "ndim", 0) == 0:
            return leaf
        return jax.device_put(leaf[0], target)

    return jax.tree.map(take_replica_zero, tree)
  • jax.device_put(x, device) is the whole fix. Its docstring: "If given, then the result is committed to the device(s)" [C]. Verified to produce SingleDeviceSharding(device=0) [M].
  • The pin must happen outside jit. A device_put inside a traced function does not commit: with a Device or a NamedSharding it is silently ignored and the result comes back on all devices [M] (upstream: jax#6040, "sticky device (device_put) ignored in jit context"); with a SingleDeviceSharding it raises ValueError: … device_put inside jit with device ids [0] [C]. A maintainer's summary [C]: "the recommendation is to use device_put outside jit, and if you want to constrain shardings of an intermediate inside jit, then with_sharding_constraint is the API." Single-device extraction is strictly a top-level operation — jit(..., out_shardings= SingleDeviceSharding(d0)) under a multi-device mesh raises the incompatible-devices error too [C].
  • Per leaf, not per tree. jax.device_put(whole_tree, device) raises TypeError: Argument 'x' of type <class 'str'> is not a valid JAX type on the first non-array leaf [M] — so you need the map and the guards anyway, and mapping keeps the replicated intermediate one leaf wide instead of one tree wide.
  • donate=True does not rescue you. jax.device_put(x[0], D[0], donate=True) left the source array not deleted (is_deleted() == False) [M, CPU backend]; donation is documented as best-effort. Drop the reference instead.

1.3 Two symptoms, one bug

Loud. Mixing a properly pinned array with a naively "unreplicated" one inside a jit fails immediately [M]:

ValueError: Received incompatible devices for jitted computation.
  Got argument a … with device ids [0] on platform CPU and
      argument b … with device ids [0, 1, 2, 3] on platform CPU

This is the good outcome — it names the bug. In practice it appears wherever a host-side consumer combines a device-0 batch slice with a replicated model: an eager sanity forward, a lax.cond / select_n over a validity mask, an overlay renderer [F].

Quiet. If nothing ever mixes device sets, the copies just sit there and shrink the margin until something else fails. [F] On a 2-GPU run, params + Adam state unreplicated this way charged ~10.5 GiB to each card, half of it on the card that never reads the result.

The leak is the Python local, not the copy

The copy is transient only if you let it be. This pattern pins memory for the rest of the run:

if global_step % save_every == 0:
    model_to_save = unreplicate(model)        # rebound only on the NEXT save
    opt_to_save   = unreplicate(opt_state)    # ...so both stay live in between
    save(model_to_save, opt_to_save)

Each local holds its arrays until reassigned, so between one save and the next the copy stays live.

Do not over-claim the magnitude — one field investigation measured it and got a much smaller number than it first estimated [M]. Bracketing the allocator cap on a trainer whose step wanted ~80.7 GB of a ~135 GB pool, with save / overlay / validation all firing on the same step, the window in which a boundary tipped an otherwise-passing step was under ~1.5 GB — not the 12-25 GB the docstring arithmetic above suggests. The persistent cost was small; the boundary behaved as a brief transient spike on a margin that was already knife-edge (the production failure map read 59% free against 58.4% needed). Its earlier "~25 GB/card pinned for the whole run" estimate was retracted.

The lesson generalises past this bug: the residency mechanism is real and worth fixing, but on a system this tight the thing that eventually kills you may be slow allocator drift over tens of thousands of steps rather than any single branch. In that investigation the production memory fraction was GREEN in a fresh short run at the same setting the dead arms had used, and the arms had survived several saves and a validation before dying — so no single event explained it. Fix the placement because it is correct; do not promise a specific margin recovery unless you measured one.

Rules

  • Route every unreplication through one helper that commits to a device. Never open-code x[0].
  • del the locals at the end of the branch, or scope them in a function so they fall out.
  • Unreplicate the buffers too. Large constants replicated at setup (mesh templates, lookup tables, per-step constants) carry a device axis as well. A host-side consumer receiving a (D, G, V, C) array where it expects (G, V, C) fails deep inside an einsum, far from the cause.

1.4 Under the modern API the same mistake is worse

[M] In a jax.jit + NamedSharding data-parallel program, parameters are replicated (PartitionSpec()), not stacked — there is no leading device axis to index. x[0] then returns the first element of the data:

p = jax.device_put(jnp.arange(4.), NamedSharding(mesh, P()))
p.shape        # (4,)  — no device axis
p[0]           # 0.0   — a DATA element, not a replica. Silently wrong.

The fix needs no indexing at all, and it is the single most useful line in this skill:

single = jax.device_put(params, jax.devices()[0])                      # or
single = jax.device_put(params, jax.sharding.SingleDeviceSharding(jax.devices()[0]))

Both produce SingleDeviceSharding [M]. So: migrating off pmap removes the residency trap entirely — but only if you also delete the x[0]. Porting the indexing forward converts a memory bug into a correctness bug. Grep for [0] in every unreplication path as part of the migration.

The mode matters too [M]. On an Auto mesh (what Mesh(devices, ("d",)) gives you by default) x[0] silently performs the gather described above. Give the mesh Explicit axis types and the same expression raises instead:

ShardingTypeError: … out sharding could not be resolved unambiguously
  (or would require collectives on inputs). Use .at[...].get(out_sharding=) …

That is a good reason to adopt explicit axis types on new code: the failure becomes loud. In Explicit mode the resharding API is jax.sharding.reshard / jax.reshard, and with_sharding_constraint degrades to an assertion rather than a constraint [C].

1.5 Do not unreplicate at all if your checkpointer is distributed

§1 assumes a single-writer serialiser that reads one complete tree from one device. That is a property of your checkpointing code, not a law. A distributed checkpointer (Orbax) inverts it: "each process first writes its own data independently to its corresponding subdirectory" and then merges metadata, and save is "called on all available controller processes" [C]. device_get / process_allgather / gathering to one device appear nowhere in its idiom.

So decide once, and write it down: either you gather to one device and pay §1's attention tax, or you write shards in place and never unreplicate. Doing the first while believing you are doing the second is how the residency bug survives review.

1.6 Assert placement; don't hope for it

Put this in a test, not in a comment. It runs on CPU fake devices in seconds.

def assert_single_device(tree, what):
    for path, leaf in jax.tree_util.tree_leaves_with_path(eqx.filter(tree, eqx.is_array)):
        n = len(leaf.sharding.device_set)
        assert n == 1, f"{what}{jax.tree_util.keystr(path)} lives on {n} devices"
    resident = sum(resident_bytes(l) for _, l in ...)
    assert resident == sum(l.nbytes for _, l in ...)   # no hidden replicas

Then take the same measurement across a whole save cycle: building the checkpoint copies must add exactly one tree's worth of device bytes to jax.live_arrays(), and dropping them must return you to the baseline. That catches the lifetime half of the bug, which the per-leaf assertion cannot see.

Gotcha [M]: compute resident bytes from arr.sharding.shard_shape(arr.shape) × itemsize × len(arr.sharding.device_set), not from arr.addressable_shards — reading addressable_shards materialises single-device Array objects that then show up in jax.live_arrays() and corrupt the very accounting you are doing.

Use x.sharding.num_devices == 1 as the assertion. is_fully_replicated is not a single-device test — it is True both for a one-device array and for an N-device replicated one [M], which is precisely the state you are trying to detect. x.devices() works but is undocumented [C]. Other useful diagnostics: x.sharding, x.committed, x.device (returns a Device for single-device arrays and a Sharding otherwise), len(x.addressable_shards), and jax.debug.visualize_array_sharding(x) [M]. And jax.transfer_guard("disallow") names the destination sharding of any transfer it blocks, which is a cheap way to catch an unintended host round-trip [M]:

JaxRuntimeError: INVALID_ARGUMENT: Disallowed host-to-device transfer:
  aval=ShapedArray(float32[3]), dst_sharding=SingleDeviceSharding(device=CpuDevice(id=0) …)

1.7 Why it surfaces as a giant unrelated allocation

The leaked memory does not fail on its own — it shrinks the margin until something else fails. [F] In one case the failing request was the pmapped train step's peak workspace, ~58% of the whole pool, dying with 59% free. Diagnostic tells:

  • the request size varies with model/loss configuration → it is a computation's workspace, not a fixed-size copy of parameters
  • the allocator map shows plenty free (****____...____****) → not exhaustion, a margin problem
  • the op name in requested by op is blank → no attribution; static reading beats more runs

Do not chase this by shrinking the memory fraction. That makes the failure move earlier, not away — the same request then fails at first-step validation instead of at step 40,000.

Related trap: JAX dispatch is async, so a device error is raised at the next synchronisation point and blamed on whatever executable is being awaited there, which is often a small, innocent function. [F] A failing linear-algebra op inside a pmapped train step surfaced as JaxRuntimeError … [executable_name='jit_extract_single_device_model'] — a function that is a pure per-leaf slice and contains no such op. Before believing the named op, dump the HLO of the step (--xla_dump_to) and enumerate its custom calls, or force synchronisation (block_until_ready) immediately after each stage to localise it.


2. Collective hangs: one replica dies, the job does not

When a replica OOMs or aborts mid-collective, the surviving replicas block in the rendezvous forever:

E rendezvous.cc:116] [rank=1] Acquire clique: devices=2:[0,1]; local_participants=2
  ... has been waiting for 10 seconds and may be stuck.
  Expected 2 threads to join the rendezvous, but not all of them arrived on time.

The warning prints once. Then nothing. The process stays alive, holds every GPU, writes no metrics and never exits. XLA's default is to warn and then wait forever.

Note the vocabulary: in a single-process job (one process, several local devices, no jax.distributed) "rank=0/rank=1" are XLA's per-device threads, not processes. There is no second process to kill.

2.1 Arm a timeout — but arm the RIGHT one

One function (xla/service/rendezvous.cc) emits that message for four different rendezvous points, each with its own timeout pair. The name in the message is the whole triage [C]:

name in the logterminate flagdefault
Acquire clique / Initialize clique--xla_gpu_nccl_termination_timeout_seconds−1 = never
first call … collective operation--xla_gpu_first_collective_call_terminate_timeout_seconds40 s
thunk initialization completion …--xla_gpu_executable_terminate_timeout30 s
whole-execution deadlock (not a rendezvous)--xla_gpu_execution_terminate_timeoutinf

So an Acquire clique hang lasts forever by default while a first-collective-call hang already self-terminates. [F] That is why, on an Acquire clique stall, only --xla_gpu_nccl_termination_timeout_seconds ended the process and the other *_terminate_* flags looked inert — they arm rendezvous the job never reached. Match the flag to the name in the log; do not bisect the family.

A set that parses on jaxlib 0.10.1 [M] and covers all four:

export XLA_FLAGS="\
--xla_gpu_nccl_termination_timeout_seconds=600 \
--xla_gpu_execution_terminate_timeout=10m \
--xla_gpu_execution_progress_tracking=16 \
--xla_gpu_executable_warn_stuck_timeout=60 --xla_gpu_executable_terminate_timeout=600 \
--xla_gpu_first_collective_call_warn_stuck_timeout_seconds=60 \
--xla_gpu_first_collective_call_terminate_timeout_seconds=600"

Four things to know before you touch these — the details, defaults and verbatim help strings are in references/collective-hang-toolkit.md:

  • The terminate timeout dumps every thread's stack (DumpAllThreadStacksToLog) before its LOG(FATAL) [C]. That is the best artefact you will get; an external kill gets you none.
  • A malformed flag VALUE and an unknown flag NAME are different fatals, and both kill the job at first device use with a F… line that reads like a driver fault [M]. The single most expensive instance: --xla_gpu_execution_terminate_timeout takes a duration string (10m, 600s) — passing =60 fails, and reading that failure as "not a flag" discards the one knob whose stated purpose is "to prevent undetected deadlocks".
  • NCCL_TIMEOUT / NCCL_ASYNC_ERROR_HANDLING / NCCL_BLOCKING_WAIT are PyTorch variables. Absent from NCCL's documented variable list [C] and from the bundled libnccl.so.2 [M]; in a JAX job they do nothing. Use NCCL_DEBUG / NCCL_DEBUG_SUBSYS / NCCL_DEBUG_FILE.
  • Not every stuck warning is a hang. The same code logs "Warning above was a false-positive. Perhaps the timeout is too short" when threads merely arrived late [C] — uneven per-device work under pmap produces exactly this. Raise the warn timeout; do not chase a bug.

When to set them. XLA_FLAGS is parsed once, at the first XLA backend initialisation — not at import jax [M]: with a bogus flag set, import jax succeeds and the abort only arrives when the first array touches a device. Setting it before importing jax is still the right rule, but audit the guard from the program's own log of its effective XLA_FLAGS, never from import order. Also drop RLIMIT_CORE to 0 first — the abort writes a core dump, one measured at 17.5 GB into a repo root [F], unreadable without driver symbols.

2.2 A dead process is not a released process

[M/F] An exception escaping main does not reliably end a multi-device job: normal interpreter shutdown joins the PJRT worker threads, and a thread parked in a rendezvous never joins. A run that "crashed" can hold every GPU indefinitely. Exit hard:

def run_entrypoint(main):
    try:
        main()
    except BaseException:
        traceback.print_exc()
        sys.stderr.flush(); sys.stdout.flush()
        os._exit(1)          # sys.exit() would block joining a parked PJRT thread

2.3 Detecting a hang: the only signal that does not lie

signalverdict
process existsuseless — a hung job lives forever
exit codeuseless — never arrives
GPU memory allocateduseless — a hung job holds it all
GPU utilisation snapshotuseless — a healthy train loop reads 0% between steps
metrics-file mtimemisleading — healthy runs batch writes and can look 80+ min stale
CPU-tick deltareliable — /proc/<pid>/stat fields 14+15 (utime+stime)

[F] A hung process burned ~1 tick per 15 s with both GPUs at 0% util while still holding 165 GB and 131 GB; a healthy trainer burns thousands in the same window. The regimes are three orders of magnitude apart, so the threshold is not delicate.

Three properties a watchdog needs that the tick delta alone does not give you:

  • Debounce. Require several consecutive quiet polls (e.g. 40 × 15 s). One quiet poll is a long checkpoint write or a host-side pause, not a hang.
  • A startup grace period. Shuffle-buffer fill plus compilation can exceed 20 minutes; a watchdog armed at t=0 kills healthy launches.
  • A fast-failure cap. A death inside the grace period is a launch failure (bad config, OOM at step 1), not a stall. Stop relaunching after a few.

Prefer an in-process heartbeat to an external prober. A daemon thread that watches a beat(position) timestamp knows where the loop was, and can os._exit() — releasing the GPUs even though a PJRT thread will never join. Layer it: (1) the XLA termination flag for rendezvous stalls, (2) the heartbeat watchdog for every other stall, (3) the hard-exit entrypoint of §2.2.

Before you kill a hung run, ask it where it is. Register faulthandler.register(signal.SIGUSR1, all_threads=True, chain=False) at startup; then kill -USR1 <pid> dumps every thread's stack to stderr without killing the process. That dump is usually the only record of where a hang was. (chain=False matters: chaining re-raises to the default handler, which terminates.)

2.4 Multi-process adds a heartbeat — but not a cure

If you run multi-process, jax.distributed does "health checking, ensuring that all processes shut down if any process dies" [C]. Read the default carefully anyway — JAX's fault tolerance page [C]:

"By default, if a process fails while participating in a distributed computation (like jnp.sum), then the rest of the processes participating in the computation will get stuck forever."

Heartbeats fate-share the processes; they do not unwedge an in-flight collective. You still want §2.1. The knob is heartbeat_timeout_seconds (default 100) — not heartbeat_timeout [M] — and the survivors' fatal message carries the most useful operational hint in the stack: "Disable Python buffering, i.e. python -u, to be sure to see all the previous output" [C]. Without -u the dying process's traceback, the only thing that says why, is lost unflushed.

Prevention beats detection: "it's crucial that all processes… run the same computation in the same order… beware of any control flow that depends on jax.process_index() and includes communication" [C]. multihost_utils.assert_equal on shapes/step counters/config hash before entering a jit, plus named sync_global_devices barriers to bisect divergence, is the documented mismatch check. Single-process pmap on local devices gets none of this — hence §2.1–2.3.

For the full API, the coordinator's own error strings, live inspection of a wedged job (NCCL RAS on localhost:28028, gdb thread apply all bt, profiler TraceMe stall counts, per-rank HLO diffing) and what upstream issues actually say, see references/collective-hang-toolkit.md.


3. Scaling a recipe without changing the experiment

Every step-counted quantity must be divided

If a step consumes N times the samples, everything counted in steps reaches its boundary on N times the data. Halving two of three is worse than halving none, because the arm looks subtly wrong rather than obviously wrong.

steps_per_epoch                    ÷ N
unfreeze/freeze schedules          ÷ N
lr_decay_updates / cosine horizon  ÷ N     <-- the one that gets missed
warmup_steps                       ÷ N (usually)
eval/checkpoint cadences           ÷ N (if you want them at the same data volume)

Missing the LR horizon means the schedule completes on N× the data, so at any matched sample count the arm sits at a higher LR — less decayed and often a scaled-up peak. It presents as "multi-GPU is mysteriously less sample-efficient".

Verify from the run's own effective config, not the launch flags. Whether a resume honours a config file is a property of your trainer — many rebuild config from the checkpoint and re-apply only a whitelist of CLI overrides, so a passed --config is silently ignored. Read the value the process logs at startup (or logs to your experiment tracker), never the command line.

Divisibility is a hard startup gate

Every batch bucket's B must divide the device count — including eval buckets, which are easy to forget and fail only at startup on the larger device count.

Decide whether sharding pays before you build for it

Two numbers settle it, and neither is throughput.

Critical batch size. Above it, more parallelism buys wall-clock and loses sample efficiency. If your critical batch fits on one device, more devices cannot help — and if data is your binding constraint rather than time, going past it is strictly bad.

The standard model is McCandlish, Kaplan & Amodei, An Empirical Model of Large-Batch Training (arXiv:1812.06162) [C]. Its printed tradeoff law is (S/S_min − 1)·(E/E_min − 1) = 1 with B_crit ≡ E_min/S_min, equivalently (Appendix D, fixed batch) the more usable S/S_min = 1 + B_crit/B and E/E_min = 1 + B/B_crit. At exactly B = B_crit you pay 2× the minimum steps and 2× the minimum data — a turning point, not a free lunch. Two conditions a naive fit violates [C]: re-tune the LR at every batch size before fitting (and fit in log space), and remember B_crit "typically increases by an order of magnitude or more over the course of training" — so it exists only relative to a stated performance target.

You can measure it instead of fitting it, almost for free. The gradient-noise estimator needs only the gradient norm before and after the cross-device average [C]:

S ≡ (|G_small|² − |G_big|²) / (1/B_small − 1/B_big)      # B_small = per-device, B_big = global

— "we can compute these with minimal effort by computing the norm of gradient before and after averaging between devices." Two extra scalars in a pmap/shard_map step, and you get B_crit from one run instead of a sweep. Say measured if you did this and curve fit if you fitted the tradeoff law to (arm, target) observations; they are different claims.

Learning rate does not scale with a single rule — it scales with an exponent that depends on the optimizer, in the saturating form ε(B) = ε*/(1 + B*/B)^α: α = 1 (linear) for SGD/momentum, 0.5 < α < 1 for Adam/RMSProp, α = 0.5 for the classic variance-preserving √ rule [C]. Because it saturates, an unbounded lr *= sqrt(ratio) overshoots at large B. Warmup is not optional and should be gradual, because "the linear scaling rule breaks down when the network is changing rapidly, which commonly occurs in early stages of training" [C]. The largest study of this (Shallue et al., 35 workloads) found the optimum "did not always follow either a linear or square root scaling heuristic" and that LR and momentum must be tuned jointly [C] — so treat the rule as a starting point for a re-tune, never as a substitute.

Two traps around the comparison itself [C]:

  • The budget is not neutral. An epoch/example budget only ever probes the perfect-scaling regime and favours small batches; a step budget favours large ones. Say which you used.
  • Check you are not simply IO-bound first. Tuning Playbook: "if the batch size doubles, the training throughput should also double… If this is not the case then the training pipeline has a bottleneck such as I/O or synchronization between compute nodes." Its usual conclusion is worth hearing before you build anything: "the largest batch size supported by the available hardware will be smaller than the critical batch size… a good rule of thumb is to use the largest batch size possible." (Gradient accumulation buys none of this — it "does not provide any throughput benefits".)

Citations, exact equations and the recent CBS literature: references/scaling-and-batch-size.md.

Decompose any speedup you quote. "2 GPUs gave 1.71×" is usually two changes at once — the extra devices and a bigger batch. Measure the batch increase on one device first:

1.71  =  2 (the doubled batch)  /  1.16 (the sharding tax)

[F] That decomposition showed the second device contributed a 16% overhead and nothing else, and that the whole useful batch range fitted on a single card. Comparing an N-GPU arm at batch 2B against a 1-GPU arm at batch B measures the batch, not the hardware.

An N-GPU arm is not the same objective as a 1-GPU arm

  • Masked means. Gradients are pmean'd, but a loss term computed as sum / max(count,1) makes mean-of-per-device-means ≠ global masked mean whenever counts differ across shards.
  • Percentiles are wrong. A pmean over per-device percentiles is not a global percentile. Means are fine; treat p50/p90/p99 as approximate.
  • Contrastive losses change meaning. Gathering across devices changes the negative-pool size, which changes the objective. Check whether your resume path force-enables such a gather — resume and fresh launch then optimise different things.
  • The random stream is not device-count invariant. The standard idiom derives a per-device key inside the mapped function, fold_in(rng, step) then fold_in(step_key, lax.axis_index(...)) (axis_index is free — it lowers to ReplicaId plus arithmetic, no communication [C]). It is reproducible at a fixed device count and not across a change in device count, so a 2-GPU and a 4-GPU arm see different augmentation/dropout noise even at matched global batch. Note this is community practice: no official JAX page documents PRNG handling across devices [C, negative]. Two rules regardless: init must use an identical key on every device or the replicas diverge on step 0; dropout/augmentation must use a varying key or you lose batch diversity.
  • There is a noise floor even at N=1. [F] Two runs of the same step implementation with identical model/state/batch/key diverged after two steps: ~4e-7 absolute on parameters, ~1e-3 relative on losses. Run an A-vs-A control to establish that floor before reading any small 1-GPU-vs-N-GPU gap as signal. OpenXLA's determinism page names two GPU sources — autotuning picking a different winning kernel across compiles (pin with --xla_gpu_autotune_level=0 or persisted AOT autotune results) and nondeterministic ops accumulating via atomics — and offers --xla_gpu_exclude_nondeterministic_ops at "substantial runtime throughput loss", which hard-fails compilation for ops with no deterministic lowering [C]. --xla_gpu_deterministic_ops also exists in the plugin [M] but is absent from that page [C, negative].
  • Do not say "NCCL is nondeterministic". The reduction tree is fixed at communicator init, so an all-reduce is generally run-to-run reproducible on an unchanged configuration; what it is not is invariant to configuration — change the device count, rank order, algorithm, protocol or message size and float non-associativity gives you different bits [C, inferred from NCCL issue reports; no NVIDIA guarantee found]. For 2-GPU vs 4-GPU that is the operative statement: the arms are not bitwise comparable even with identical seeds and data order.

4. Instruments and knobs that behave differently on several devices

Buffer donation matches on the sharded aval

Donation (jax.jit(donate_argnums=...), eqx.filter_jit(donate=...)) aliases an input buffer to an output of the same shape and dtype. On several devices the match is made on the per-device shard, not the global shape [C, read from jax/_src/interpreters/mlir.py]. Consequences that produce no error at all:

  • Same global shape but a different sharding → different shard shapes → no aliasing, no memory saving, only UserWarning: Some donated buffers were not usable: ….
  • Memory kinds must match (device vs pinned_host); the source comment is explicit that cross-memory donation is not supported yet.
  • An AUTO-layout input donated toward a concrete-layout output raises.
  • Keyword arguments are never donated — f(params=p) silently donates nothing [C].
  • jit + shard_map + donate_argnums was genuinely broken until jax 0.4.23 (jax#15368, fixed 2024-02) [C].

Because every one of these degrades to a warning, promote it in CI: warnings.filterwarnings("error", message="Some donated buffers"). Otherwise a sharding change silently removes an optimisation you are still budgeting memory for.

jax_log_compiles misses pmap — use jax.monitoring and normalize _device_put_sharded vs. 'i'-mesh

[M] Under modern JAX (>= 0.4.30 / 0.10+), jax_log_compiles=True does not emit log records for jax.pmap compilations. Always attach a jax.monitoring listener on "/jax/core/compile/backend_compile_duration" (rae-jax-profiling §10) to count every jit, pjit, shard_map, and pmap backend compile.

Furthermore, legacy jax.device_put_replicated(x, devs) tags leaves with NamedSharding(mesh={'_device_put_sharded': N}), whereas jax.pmap(..., axis_name='i') returns leaves with NamedSharding(mesh={'i': N}):

  • Epoch-2 recompile: Epoch 1 compiles pmap for _device_put_sharded inputs, and Epoch 2 recompiles the entire pmap step (~40–60 s) because the carry returned by Epoch 1 now has mesh 'i'.
  • Every-eval recompile with donate_argnums: If a single-device evaluator uses @functools.partial(jax.jit, donate_argnums=(0,)) on eval_state while consuming unreplicated parameters from training_state, any leaf whose sharding flips between _device_put_sharded and 'i' across calls causes the evaluator to recompile on every single evaluation (~30 s × 100 evals = 50 min wasted).
  • Fix: Normalize initial replicated state through a 1-step pmap(lambda s: s, axis_name='i') identity (or construct all arrays on a single explicit Mesh), strip weak types, and commit unreplicated evaluator inputs via unreplicate_to_single_device (§1.2) before calling any donate_argnums evaluator.

Catch accidental host round-trips with the transfer guard

jax.transfer_guard [C] takes allow / log / disallow / log_explicit / disallow_explicit, plus directional variants (jax_transfer_guard_host_to_device, _device_to_device, _device_to_host). Two properties make it the right tool here:

  • "disallow" blocks only implicit transfers — a stray float(loss), print(array) or np.asarray(x) inside the step — while still allowing explicit device_put/device_get, so checkpointing keeps working. Use disallow_explicit only for the innermost loop.
  • The error names the destination sharding, which is exactly what you want when diagnosing placement [M]: INVALID_ARGUMENT: Disallowed host-to-device transfer: aval=…, dst_sharding=SingleDeviceSharding(…)

Caveat [C]: the context manager is thread-local and "a newly spawned thread will use the global option" — a guard set in the training thread does not cover a background prefetch thread. Set the global flag if you mean the whole process.

jax.debug.print is not free on several devices

Under pjit/sharded jit, "a global synchronization occurs that will materialize values on a single device"; under pmap the prints "might be reordered" and ordered=True raises [C]. A debug print left in a sharded step costs a device-wide sync every step. (A plain Python print under jit is free precisely because it only runs at trace time.)

Profiling

jax.profiler.trace("/tmp/prof") around a block_until_ready()'d call — the doc is explicit that without it async dispatch ends the trace before the work runs [C]. For a job already running, jax.profiler.start_server(9999) plus python -m jax.collect_profile <port> <ms> (or XProf's capture-by-address) beats restarting it. The TensorBoard plugin is now xprof; HLO Op Stats and the trace viewer are where all-reduce/all-gather time actually shows up. Negative finding [C]: the JAX profiling page contains no multi-process guidance at all — single-host multi-GPU is covered (each local device is its own row), multi-process is not, and you fall back to per-process capture.

Knobs upstream actually recommends

From GPU performance tips [C]:

  • Single-host multi-GPU NCCL tuning: NCCL_LL128_BUFFSIZE=-2, NCCL_LL_BUFFSIZE=-2, NCCL_PROTO=SIMPLE,LL,LL128 — intra-node only, "don't seem useful for multi-host communication yet".
  • --xla_gpu_enable_latency_hiding_scheduler (default off) to overlap collectives with compute. It changes the live-range structure, so re-check peak memory when you flip it.
  • Collective combiner thresholds default to 256 bytes and should be raised "high enough so as to combine at least a Transformer Layer's weight": --xla_gpu_all_gather_combine_threshold_bytes, --xla_gpu_reduce_scatter_combine_threshold_bytes, --xla_gpu_all_reduce_combine_threshold_bytes.
  • Multi-process: one process per GPU, not one per node.

multihost_utils is for multi-process, not multi-device

jax.experimental.multihost_utils (sync_global_devices, process_allgather, broadcast_one_to_all, assert_equal, host_local_array_to_global_array) is "Utilities for synchronizing and communication across multiple hosts" [C]. On a single-host box with several GPUs jax.process_count() == 1, every device is local, and none of it is needed — reaching for it there is a common cargo-cult. When you are multi-process, assert_equal and sync_global_devices are the cheap way to catch the mismatched-collective class of hang (processes disagreeing about what to call, or in what order) before it becomes a rendezvous stall.


5. Operational habits that pay for themselves

  • Never mutate a venv while runs are live. Replacing a package unlinks shared objects the running processes have mmap'd; they die silently, staggered, with no diagnostic. Clone the venv, mutate the clone, launch new runs against it.
  • Never kill by pattern. pgrep -f <name> matches your own command line, and on a shared box a name match can take out someone else's job. Resolve explicit PIDs (ps -eo pid,args), then signal those.
  • Stagger launches. Two simultaneous XLA compilations of a large program can push load high enough that something gets killed with no traceback. Start one, let it compile, then the next.
  • Budget the cold start. Shuffle-buffer fill plus multi-device compilation can take 15–25 minutes before the first step [F]. Judging a launch at 20 minutes has twice produced a false "it hangs".
  • Some warnings are permanent noise. VMM cuMemCreate with FABRIC+POSIX_FD … CUDA_ERROR_NOT_PERMITTED appears on every multi-GPU start under container capability limits and falls back automatically [F]. Confirm collectives work (psum + a large all-reduce) once, then stop reading it. (Some donated buffers were not usable is not in this category — see §4.)
  • Grep for the mechanism, not the string. Command-buffer formation is a thunk-level pass: optimized HLO contains command_buffer zero times even when buffers are being built [F]. Use TF_CPP_VMODULE=command_buffer_conversion_pass=3.
  • Checkpoint atomically. Stage every member to .tmp and rename, metadata last, so "metadata exists" implies a complete set. A non-atomic writer with keep_last=1 has a window (measured at 23–43 s on 12 GB checkpoints [F]) in which a death leaves zero resumable checkpoints.

Quick reference

SymptomLikely causeSection
Alive, 0% util, no metrics, holds GPUscollective hang after a replica died§2
Acquire clique / rendezvous.cc / Expected N threads to joinsame§2
Hung job survives a crash and keeps its GPUsPJRT thread never joins at shutdown§2.2
Received incompatible devices for jitted computationone operand pinned, one still replicated§1.3
Huge OOM at checkpoint/validation boundaryunreplication residency ate the margin§1
requested by op with a blank namenot attributable — read the code, don't rerun§1.7
Error blames a jit'd fn that has no such opasync attribution at the next sync point§1.7
Request size varies by model/loss configit is a computation's workspace, not a copy§1.7
Smaller MEM_FRACTION makes it fail soonermargin problem, not a leak in that request§1.7
Ported off pmap and the numbers changedx[0] on a replicated (not stacked) array§1.4
AttributeError: device_put_replicatedremoved in jax 0.10 — build an explicit mesh§0
Unknown flag in XLA_FLAGS at first device usethe flag name does not exist§2.1
Check failed: tsl::Flags::Parse … usage: --foo=BARthe flag exists; the VALUE is malformed§2.1
Warning says "unstuck … false-positive"straggler, not a hang — raise the warn timeout§2.1
ShardingTypeError … out sharding could not be resolvedx[0] on an Explicit-mode mesh§1.4
NCCL_TIMEOUT etc. had no effectPyTorch variables, absent from NCCL§2.1
N-GPU arm less sample-efficient than 1-GPUa step-counted schedule was not divided§3
Startup: batch not divisible by device counteval bucket, usually§3
Runs die silently just after a package changevenv mutated under live processes§5
Some donated buffers were not usable on a sharded jitdonation matched on shard shape, not global shape§4
A sharded step got slow after adding a debug printjax.debug.print forces a global sync§4

Pre-flight checklist before launching on N devices

  • Every unreplication goes through one helper that device_puts to a device — no bare x[0]
  • That helper guards non-array and 0-d leaves
  • A test asserts single-device placement and total resident bytes across a save cycle
  • steps_per_epoch, LR horizon, warmup and freeze/unfreeze schedules all divided by N
  • Every batch bucket, including eval, is divisible by N
  • Timeouts armed for the rendezvous you can actually hit, set before jax is imported and echoed in the log (--xla_gpu_nccl_termination_timeout_seconds, --xla_gpu_execution_terminate_timeout)
  • Every new XLA_FLAGS entry smoke-tested against a one-line program that touches a device
  • RLIMIT_CORE dropped to 0; faulthandler on SIGUSR1
  • Heartbeat watchdog with debounce + startup grace + fast-failure cap; entrypoint os._exits
  • Checkpoints written atomically, metadata last
  • The speedup you plan to quote is decomposed into batch effect vs sharding tax
  • Critical-batch estimate says another device can help at all

Related skills

  • jax-memory-and-retracing — the single-device half of the memory story: retrace leaks, host-RAM OOM, allocator diagnosis. Read it first if the OOM is not at a checkpoint/validation boundary.
  • jax-jit-and-fusion — jit boundaries, donate_argnums, and honest warm/block_until_ready benchmarking, which §3's speedup decomposition depends on.
  • jax-config — static/dynamic config separation, so the schedule constants in §3 have one source of truth to divide.

References

  • references/collective-hang-toolkit.md — the full XLA timeout-flag table with defaults and verbatim help strings, the flag-naming traps, real vs PyTorch NCCL variables, NCCL RAS, jax.distributed failure semantics, and how to inspect a job that is wedged right now.
  • references/scaling-and-batch-size.md — critical-batch-size and LR-scaling literature with exact equations, the caveats that invalidate a naive fit, and recent work on how CBS actually scales.

Pose DataJoint Query Reference

CRITICAL: NEVER Modify Database Entries

DO NOT update, delete, or alter any DataJoint database entries. This includes:

  • update1(), delete(), drop() on any table
  • Modifying settings lookup tables (KinematicReconstructionSettingsLookup, ProbabilisticReconstructionSettingsLookup, KineticReconstructionSettingsLookup, KeypointSet, etc.)
  • Altering any computed table entries

Database entries are shared state used by the entire lab. Changing a settings entry changes it for everyone and invalidates prior results computed with those settings.

Overview

The biomechanics pipeline has two parallel systems:

  1. Multi-Camera (MMC) - Lab-based, multiple synchronized cameras, uses participant_id (string)
  2. Monocular (PBL) - Phone-based portable recordings, uses subject_id (integer)

Both produce kinematic outputs (qpos, joints, sites) but have different table hierarchies.

Quick Reference: Package Imports

# Shared: Video and 2D/3D pose estimation
from pose_pipeline.pipeline import Video, VideoInfo, TopDownPerson, LiftingPerson

# === MULTI-CAMERA (MMC) ===
from multi_camera.datajoint.sessions import Session, Recording, Subject
from multi_camera.datajoint.multi_camera_dj import (
    MultiCameraRecording, SingleCameraVideo, PersonKeypointReconstruction
)
from body_models.datajoint.kinematic_dj import KinematicReconstruction
from body_models.datajoint.dataset import fetch_keypoints  # For synchronized KR+keypoint fetch

# === MONOCULAR (PBL) ===
from portable_biomechanics_sessions.emgimu_session import (
    Subject as PBLSubject,      # Note: different from MMC Subject!
    Session as PBLSession,      # Note: different from MMC Session!
    FirebaseSession
)
from body_models.datajoint.monocular_dj import MonocularReconstruction

Key Spaces (CRITICAL - Different Per Pipeline!)

Multi-Camera Key Space

# Session: participant_id is STRING
session_key = {'participant_id': '104', 'session_date': date(2023, 7, 21)}

# Video: video_project + filename
video_key = {'video_project': 'CLINIC_GAIT', 'filename': 'trial_001.27.mp4'}

Monocular Key Space

# Subject/Session: subject_id is INTEGER, project is part of key
subject_key = {'subject_id': 301, 'project': 'HLL'}

# Session adds timestamp
session_key = {'subject_id': 301, 'project': 'HLL',
               'session_start_time': datetime(2024, 1, 15, 10, 30, 0)}

# AppVideo links to Video table
app_video_key = {**session_key, 'app_start_time': ...,
                 'video_project': 'HLL', 'filename': '0301_gait.mp4'}

DataJoint Operators

OperatorMeaningExample
&Restrict (filter)Video & 'video_project="HLL"'
*Join tablesSession * Recording * MultiCameraRecording
-Set differenceVideo - TopDownPerson (videos without poses)
.proj()Select attributesTable.proj('field1', 'field2')

Fetching Data

# fetch1() - Exactly ONE row (raises error if 0 or >1)
timestamps, qpos = (Table & key).fetch1('timestamps', 'qpos')

# fetch() - Multiple rows as arrays
all_keys = (Table & restriction).fetch('KEY')  # List of dicts
values = (Table & key).fetch('field_name')     # Numpy array

# fetch(as_dict=True) - Multiple rows as list of dicts
records = (Table & key).fetch(as_dict=True)

Multi-Camera (MMC) Queries

Count Videos in MMC Project

from pose_pipeline.pipeline import Video
from multi_camera.datajoint.multi_camera_dj import MultiCameraRecording, SingleCameraVideo

# Videos linked to multi-camera recordings
count = len(Video & SingleCameraVideo & (MultiCameraRecording & 'video_project="CLINIC_GAIT"'))

Count Sessions/Participants in MMC

from multi_camera.datajoint.sessions import Session, Recording
from multi_camera.datajoint.multi_camera_dj import MultiCameraRecording
import numpy as np

# Sessions for a participant (participant_id is STRING!)
count = len(Session & {'participant_id': '104'})

# Unique participants in a project
participants = np.unique(
    (Session & (Recording & (MultiCameraRecording & 'video_project="CLINIC_GAIT"'))).fetch('participant_id')
)
print(f"Participants: {len(participants)}")

Get MMC Kinematic Reconstruction

from body_models.datajoint.kinematic_dj import KinematicReconstruction
from datetime import date

# CRITICAL: Always specify kinematic_reconstruction_settings_num!
key = {
    'participant_id': '102',
    'session_date': date(2023, 7, 21),
    'kinematic_reconstruction_settings_num': 137  # REQUIRED!
}

timestamps, qpos, joints, sites = (KinematicReconstruction.Trial & key).fetch1(
    'timestamps', 'qpos', 'joints', 'sites'
)
# qpos: (T, 41) joint angles in radians
# joints: (T, N_bodies, 3) body positions in meters
# sites: (T, N_sites, 3) marker positions in meters

Get 3D Triangulated Keypoints (MMC)

from multi_camera.datajoint.multi_camera_dj import PersonKeypointReconstruction

key = {
    'video_project': 'CLINIC_GAIT',
    'video_base_filename': 'trial_20231215_143022',
    'reconstruction_method': 0  # 0=Robust Triangulation
}
keypoints3d = (PersonKeypointReconstruction & key).fetch1('keypoints3d')
# Shape: (T, N_joints, 4) - [x, y, z, confidence], units: mm

Monocular (PBL) Queries

Count Videos in Monocular Project

from portable_biomechanics_sessions.emgimu_session import FirebaseSession

# AppVideo is a Part table of FirebaseSession
count = len(FirebaseSession.AppVideo & {'video_project': 'HLL'})
print(f"HLL monocular videos: {count}")

Count Subjects in Monocular Project

from portable_biomechanics_sessions.emgimu_session import FirebaseSession
import numpy as np

# Get unique subject_ids for a project
subject_ids = np.unique(
    (FirebaseSession.AppVideo & {'video_project': 'HLL'}).fetch('subject_id')
)
print(f"Subjects with HLL videos: {len(subject_ids)}")

Count Monocular Videos Processed with Kinematic Reconstruction

from portable_biomechanics_sessions.emgimu_session import FirebaseSession
from body_models.datajoint.monocular_dj import MonocularReconstruction

# Videos that have monocular reconstruction
processed = len(
    FirebaseSession.AppVideo
    & (MonocularReconstruction.Trial & {'video_project': 'HLL'})
)
print(f"HLL videos with monocular reconstruction: {processed}")

# Videos NOT yet processed
all_videos = FirebaseSession.AppVideo & {'video_project': 'HLL'}
unprocessed = len(all_videos - MonocularReconstruction.Trial)
print(f"HLL videos needing processing: {unprocessed}")

Get Monocular Kinematic Reconstruction

from body_models.datajoint.monocular_dj import MonocularReconstruction
from datetime import datetime

# Monocular uses subject_id (INTEGER) and project
key = {
    'subject_id': 301,
    'project': 'HLL',
    'session_start_time': datetime(2024, 1, 15, 10, 30, 0),
    'monocular_reconstruction_settings_num': 1  # Specify method
}

# Get all trials for this session
trial_keys = (MonocularReconstruction.Trial & key).fetch('KEY')

for trial_key in trial_keys:
    timestamps, qpos, joints, sites, rnc = (MonocularReconstruction.Trial & trial_key).fetch1(
        'timestamps', 'qpos', 'joints', 'sites', 'rnc'
    )
    # qpos: (T, 40) joint angles - monocular has 40 DOF (vs 41 for MMC)
    # rnc: (T, 3) camera rotation vector from phone attitude
    print(f"Video: {trial_key['filename']}, frames: {len(timestamps)}")

List Monocular Projects

from portable_biomechanics_sessions.emgimu_session import FirebaseSession
import numpy as np

projects = np.unique(FirebaseSession.AppVideo.fetch('video_project'))
print(f"Monocular projects: {projects}")

Fetching Synchronized Keypoints + Reconstruction (MMC)

When you need 2D keypoints aligned frame-by-frame with KR qpos, use fetch_keypoints:

from body_models.datajoint.dataset import fetch_keypoints as bm_fetch_keypoints
from body_models.datajoint.kinematic_dj import KinematicReconstruction

trial_key = {'participant_id': '104', 'session_date': date(2023, 7, 21),
             'recording_timestamps': '2023-07-21 14:06:37'}
full_key = {**trial_key, 'kinematic_reconstruction_settings_num': 137}

qpos = (KinematicReconstruction.Trial & full_key).fetch1('qpos')  # (T, 41)
timestamps, kp_raw = bm_fetch_keypoints(trial_key, only_detected=True)
# kp_raw: (C, T, 87, 3) — guaranteed qpos[i] matches kp_raw[:, i]

assert qpos.shape[0] == kp_raw.shape[1]  # ALWAYS verify

Do NOT fetch keypoints via TopDownPerson * VideoInfo and match timestamps — this produces silent frame offsets. See rae:fetching-synchronized-data for the full pattern including camera parameter reordering and contiguous segment selection.


Shared Queries (Work for Both Pipelines)

Get 2D Keypoints (Standalone, No KR Alignment)

For keypoints aligned with KR qpos, use the synchronized pattern above.

For standalone 2D analysis (no KR alignment needed):

from pose_pipeline.pipeline import TopDownPerson

key = {
    'video_project': 'HLL',  # Works for any project
    'filename': 'trial_001.mp4',
    'video_subject_id': 0,
    'top_down_method': 0  # 0=MMPose
}
keypoints = (TopDownPerson & key).fetch1('keypoints')  # Shape: (T, N_joints, 3)

Get 3D Lifted Keypoints

from pose_pipeline.pipeline import LiftingPerson

key = {**video_key, 'video_subject_id': 0, 'top_down_method': 0, 'lifting_method': 1}
keypoints_3d = (LiftingPerson & key).fetch1('keypoints_3d')  # Shape: (T, N_joints, 4)

Count All Videos by Project

from pose_pipeline.pipeline import Video
from collections import Counter

projects = Video.fetch('video_project')
for project, count in Counter(projects).items():
    print(f"{project}: {count} videos")

Table Relationships

Multi-Camera Hierarchy

Subject (participant_id)  ← STRING
    -> Session (participant_id, session_date)
        -> Recording -> MultiCameraRecording (video_project)
                            -> SingleCameraVideo -> Video
                            -> PersonKeypointReconstruction (3D triangulated)
        -> SessionCalibration.Grouping
            -> KinematicReconstruction (method 137)
                -> KinematicReconstruction.Trial (qpos, joints, sites)

Monocular Hierarchy

Subject (subject_id, project)  ← INTEGER + project
    -> Session (subject_id, project, session_start_time)
        -> FirebaseSession
            -> FirebaseSession.AppVideo -> Video
            -> FirebaseSession.PhoneAttitude (phone orientation)
            -> FirebaseSession.Gyro/Accel/Mag (IMU data)

MonocularReconstruction (subject_id, project, session_start_time, method)
    -> MonocularReconstruction.Trial (qpos, joints, sites, rnc)
        -> FirebaseSession.AppVideo (links video)

Common Mistakes

MistakeFix
MMC: {'subject_id': 104}Use {'participant_id': '104'} (string!)
PBL: {'participant_id': '301'}Use {'subject_id': 301} (integer!)
Keypoints2D tableUse TopDownPerson for 2D keypoints
Missing method for KinematicReconstructionAdd 'kinematic_reconstruction_settings_num': 137
Missing method for MonocularReconstructionAdd 'monocular_reconstruction_settings_num': 1
fetch(unique=True)Use np.unique(table.fetch('field'))
create_virtual_module()Direct import from modules
Mixing MMC Session with PBL SessionImport with alias: Session as PBLSession
Matching KR timestamps with VideoInfo timestampsUse fetch_keypoints(only_detected=True) for frame-aligned data
Assuming camera_params matches keypoint camera orderReorder from Calibration order to SingleCameraVideo alphabetical order

Method Numbers Reference

PipelineTableMethod FieldDefault
2D PoseTopDownPersontop_down_method0 (MMPose)
3D LiftingLiftingPersonlifting_method1 (VideoPose3D)
3D Triangulation (MMC)PersonKeypointReconstructionreconstruction_method0
Multi-Camera KinematicKinematicReconstructionkinematic_reconstruction_settings_num137
Monocular KinematicMonocularReconstructionmonocular_reconstruction_settings_num1

DataJoint Schema Names

Each repository stores tables in named MySQL schemas. Use these to connect directly or create virtual modules with dj.VirtualModule('alias', 'schema_name').

Schema NameRepositoryKey Tables
pose_pipelinePosePipelineVideo, VideoInfo, TopDownPerson, LiftingPerson
mocap_sessionsMultiCameraTrackingSubject, Session, Recording
multicamera_trackingMultiCameraTrackingMultiCameraRecording, SingleCameraVideo, PersonKeypointReconstruction, Calibration
multicamera_tracking_annotationMultiCameraTrackingVideoActivity (walking/standing labels)
project_body_modelsBodyModelsKinematicReconstruction, KinematicReconstruction.Trial, ProbabilisticReconstruction
project_monocular_testingBodyModelsMonocularReconstruction, MonocularReconstruction.Trial
project_body_models_gait_cyclesBodyModelsGaitTransformer, GaitTransformer.WalkingSegment, GaitTransformer.Steps, GaitTransformerMonocular
project_gdiBodyModelsGDICycles, GDICyclesMonocular, GDIJointsLookup
emgimu_sessionsPortableBiomechanicsSessionsSubject, Session, FirebaseSession, FirebaseSession.AppVideo
openpbl_session_annotationsPortableBiomechanicsSessionsVideoActivity (PBL), WalkingType

Connecting Without Code Access

import datajoint as dj

# Create virtual modules to access tables without installing the package
kinematic = dj.VirtualModule('kinematic', 'project_body_models')
gait = dj.VirtualModule('gait', 'project_body_models_gait_cycles')
sessions = dj.VirtualModule('sessions', 'mocap_sessions')
pose = dj.VirtualModule('pose', 'pose_pipeline')
mmc = dj.VirtualModule('mmc', 'multicamera_tracking')

# Then query as usual
keys = kinematic.KinematicReconstruction.fetch('KEY')

Gait-Specific Tables

For gait analysis (walking segments, step metrics, GDI), see the /gait-metrics skill which documents:

  • Walking segment detection and validation
  • Spatiotemporal gait metrics (step length, cadence, velocity)
  • Gait Deviation Index (GDI) computation
  • Joint ROM analysis per gait cycle/phase

Files to Explore

TaskFile
Video/TopDownPersonPosePipeline/pose_pipeline/pipeline.py
MMC Session/RecordingMultiCameraTracking/multi_camera/datajoint/sessions.py
MMC MultiCameraRecordingMultiCameraTracking/multi_camera/datajoint/multi_camera_dj.py
MMC KinematicReconstructionBodyModels/body_models/datajoint/kinematic_dj.py
PBL Subject/Session/FirebaseSessionPortableBiomechanicsSessions/portable_biomechanics_sessions/emgimu_session.py
PBL MonocularReconstructionBodyModels/body_models/datajoint/monocular_dj.py
Gait Cycles/Walking SegmentsBodyModels/body_models/datajoint/gait/gait_cycles_dj.py
GDI (Gait Deviation Index)BodyModels/body_models/datajoint/gait/gdi_dj.py
Gait Step/Phase MetricsBodyModels/body_models/datajoint/gait/gait_analysis.py
Gait Event DetectionBodyModels/body_models/biomechanics_mjx/gait/gait_transformer_cycles.py
Gait Metrics (standalone)BodyModels/body_models/biomechanics_mjx/gait/gait_metrics.py
VideoActivity (MMC walking labels)MultiCameraTracking/multi_camera/datajoint/annotation.py
VideoActivity (PBL walking labels)PortableBiomechanicsSessions/portable_biomechanics_sessions/session_annotations.py

RAE JAX Recompilation, Acceleration & Profiling Skill (rae-jax-profiling)

This skill consolidates the canonical Reproducible Agent Environment (RAE) standards from jax-jit-and-fusion, jax-memory-and-retracing, jax-multi-gpu, and jax-config — together with the general JAX acceleration principles established from StanfordASL/frax (Morton & Pavone 2026), XMimic (Bohlinger & Peters 2026), BodyModels, and BiomechanicalFollower (KinTwin / SuperMimic) — into a single operational checklist and verification protocol for /workspaces/eqx_pose.

Any newly implemented or compute-heavy JAX/Equinox algorithm must satisfy the rules below and pass both a single-compile assertion test (max_traces=1 + jax.monitoring backend compile listener) and a cold-vs-warm jax.block_until_ready() benchmark.


1. eqx.filter_jit & PyTree Dynamic vs. Static Separation

  1. Always decorate entry points with @eqx.filter_jit, never bare @jax.jit, when arguments include eqx.Module instances, callable PyTrees, or dataclass configs. eqx.filter_jit automatically partitions all JAX arrays (eqx.is_array) as dynamic traced leaves and treats non-array metadata as static.
  2. Pass scalar hyperparameters/weights that vary across calls as traced jnp.float32 / jnp.int32 arrays (or register config dataclasses via @register_config_dataclass where varying weights are Array fields). Python float, int, and bool values are hashed statically by eqx.filter_jit — varying them across trials or ablation toggles (e.g., use_sensors=True vs False) triggers a full XLA recompilation unless converted to traced jnp.float32 multipliers (0.0 vs 1.0).
  3. Never bake large frozen model weights as XLA constants inside closures. If a stage optimizer uses jax.jit over a flat parameter vector theta, split the frozen model via model_arrays, model_static = eqx.partition(model, eqx.is_array), pass model_arrays in the traced data dict, and reconstruct model = eqx.combine(data["model_arrays"], model_static) inside the traced function.
  4. Keep biomechanical/kinematic arrays in jnp.float32 throughout eqx_pose pipelines.

2. Bucket-Padding Variable Sequence Lengths $T$ with Masks

Because XLA compiles a distinct executable for every unique tensor shape, passing raw trial lengths $T \in {187, 243, 312, \dots}$ to a jitted solver or trajectory fitter causes an $O(\text{num_trials})$ compilation storm and host/GPU OOM.

  1. Snap $T$ to a small geometric bucket set (e.g., BUCKETS = (64, 128, 256, 512, 1024, 2048)):
    def bucket_length(n: int, buckets: tuple[int, ...] = (64, 128, 256, 512, 1024, 2048)) -> int:
        for b in buckets:
            if b >= n:
                return b
        return ((n + buckets[-1] - 1) // buckets[-1]) * buckets[-1]
    
  2. Edge-pad time/observation arrays to $T_{\text{bucket}}$ and attach a float32 validity mask mask of shape (T_bucket,) (1.0 for valid frames $0 \le t < T$, 0.0 for padded tail frames $T \le t < T_{\text{bucket}}$):
    • Use mode="edge" when padding timestamps and poses so finite differences / Nadaraya-Watson kernels / Jacobians remain well-conditioned on padded rows.
    • Replace jnp.mean(err) with masked reductions:
      def masked_mean(values: Array, mask: Array) -> Array:
          w = jnp.reshape(mask, (mask.shape[0],) + (1,) * (values.ndim - 1))
          denom = jnp.maximum(jnp.sum(mask) * (values.size // mask.shape[0]), 1.0)
          return jnp.sum(values * w) / denom
      
    • For finite-difference velocity ($T-1$) and acceleration ($T-2$) regularizers, construct edge masks vel_mask = mask[:-1] * mask[1:] and acc_mask = mask[:-2] * mask[1:-1] * mask[2:].
  3. Slice outputs back to [:T] at the host boundary after optimization or inference completes.

3. Eliminating Function-Local @jit Rebuilds & Closure-ID Cache Misses

Two closely related closure anti-patterns silently defeat JAX/Equinox compilation caching:

3a. Function-Local @eqx.filter_jit / @jax.jit Rebuilt on Every Call (Amortized Over an Inner Batch Loop)

Defining @eqx.filter_jit def fwd(...) or solve = jax.jit(jax.vmap(lambda ...)) inside an entry-point function or method (def decode(...), def place_in_camera(...)) to capture local variables (qp, k_jax) in a closure builds a brand-new function object on every call to the outer function:

# BAD: `@eqx.filter_jit` is defined inside `decode()` to close over `qp` and `k_jax`.
# Every call to `decode()` creates a fresh `fwd` function object and triggers a full ~9 s
# XLA compile on batch 0. Averaged over T=120 frames (15 batches of 8), the 9 s compile is
# amortized across the loop (`9000 ms / 120 + 4.5 ms ≈ 80 ms/frame`), masquerading as a
# plausible per-frame forward time (~18x slower than true steady state) and throwing away
# the compiled executable as soon as `decode()` returns!
def decode(head, images, K, boxes, valid, *, batch_size=8):
    qp, k_jax = head.query_points, jnp.asarray(K, jnp.float32)

    @eqx.filter_jit
    def fwd(model, ims, bxs, ok):
        return jax.vmap(lambda im, b, v: forward_one(model, qp, im, b[None], v[None], k_jax))(ims, bxs, ok)

    return [fwd(head.device_model, ...) for ...]


# GOOD: Hoisted to module scope with `qp` and `k_jax` passed as explicit traced arguments.
# Compiles once across the entire process lifetime (~4.5 ms/frame warm).
@eqx.filter_jit
def _decode_batch_jit(model, qp, k_jax, ims, bxs, ok):
    return jax.vmap(lambda im, b, v: forward_one(model, qp, im, b[None], v[None], k_jax))(ims, bxs, ok)

Why eqx.debug.assert_max_traces on a function-local definition fails silently: If you wrap fwd with eqx.debug.assert_max_traces(max_traces=1) inside decode(), a fresh guard object is created on every decode() call with its trace counter reset to 0. Within one decode() call, all 15 batches have shape (8, H, W, 3) so the counter reaches 1 and passes — even when 100 calls to decode() compile 100 separate XLA executables! Always hoist both @eqx.filter_jit and assert_max_traces to module scope.

3b. Passing Fresh Nested Closures to vmap / jit / Chunked Primitives

Creating a fresh nested Python closure on every call and passing it to a jitted or cached higher-order primitive misses any cache keyed on id(fn) or function identity:

# BAD: Creates a fresh function object `fk` with a new `id(fk)` on every call!
# Any cache keyed on `id(fn)` misses 100% of the time and leaks compiled executables.
def _landmarks_per_frame(state, fk_landmarks, frame_chunk):
    shp, scl, go, gt, bp = state.unpack()
    def fk(g, t, b):
        return fk_landmarks(g, t, b, shp, scl)
    return minibatch_vmap(fk, go, gt, bp, batch_size=frame_chunk)

Fixes:

  1. Always define @eqx.filter_jit / @jax.jit functions at module scope (or bind once in __init__), passing call-local arrays (qp, k_jax, shp, scl) as explicit traced arguments.
  2. Use callable eqx.Module PyTrees or eqx.Partial (or unroll closure cells into a PyTree before @eqx.filter_jit) so that captured arrays (shp, scl, model weights, index tables) are exposed as dynamic traced PyTree leaves while the underlying function code object is static and hashable.
  3. Never key a compilation cache on id(fn) without weakrefs or closure unwrapping. In Python, ephemeral closures either leak forever if stored in the cache or have their id() reused after garbage collection for a completely different closure.

4. Eliminating Host Syncs (float(loss)) Inside Training Loops via jax.lax.scan

Calling float(loss_val) or np.asarray(grad) inside a Python for step in range(num_steps): loop forces a device-to-host synchronization every step, destroying JAX's asynchronous dispatch (~100–500 $\mu$s dispatch overhead per step) and preventing cross-step XLA fusion.

# BAD: Python loop + per-step device-to-host sync `float(loss_val)`
for _ in range(config.num_steps):
    model, opt_state, loss_val, metrics = train_step(model, opt_state, data)
    loss_history.append(float(loss_val))

Fix — Fuse steps with jax.lax.scan inside @eqx.filter_jit:

@eqx.filter_jit
def _run_training_scan(
    model: Model,
    opt_state: optax.OptState,
    data: FitData,
    optimizer: optax.GradientTransformation,
    num_steps: int,
) -> tuple[Model, optax.OptState, Float[Array, "num_steps"], dict[str, Float[Array, ""]]]:
    params, static = eqx.partition(model, eqx.is_inexact_array)

    def _step(carry, _):
        curr_params, curr_opt = carry
        curr_model = eqx.combine(curr_params, static)
        (loss_val, metrics), grads = eqx.filter_value_and_grad(loss_fn, has_aux=True)(
            curr_model, data
        )
        updates, next_opt = optimizer.update(grads, curr_opt, curr_params)
        next_params = eqx.apply_updates(curr_params, updates)
        return (next_params, next_opt), (loss_val, metrics)

    (final_params, final_opt), (losses, metrics_hist) = jax.lax.scan(
        _step, (params, opt_state), None, length=num_steps
    )
    final_model = eqx.combine(final_params, static)
    last_metrics = jax.tree_util.tree_map(lambda x: x[-1], metrics_hist)
    return final_model, final_opt, losses, last_metrics

Then transfer losses to host once after the scan finishes: loss_history = np.asarray(losses, dtype=np.float64).tolist().


5. Single-Pass Nested jax.jvp & Precomputed Static Linear Operators

  1. Simultaneous $(x(t), \dot{x}(t), \ddot{x}(t))$ evaluation for scalar $t$: Using nested jax.jacfwd with a separate primal call evaluates the network 3 times per timestamp. Replace with a single nested jax.jvp along unit scalar tangent one = jnp.ones_like(t):
    def evaluate_single_pos_vel_acc(fn, t: Float[Array, ""]) -> tuple[Array, Array, Array]:
        one = jnp.ones_like(t)
        def val_and_vel(s: Float[Array, ""]) -> tuple[Array, Array]:
            return jax.jvp(fn, (s,), (one,))
        (val, vel), (_, acc) = jax.jvp(val_and_vel, (t,), (one,))
        return val, vel, acc
    
    This computes position, velocity, and acceleration in one single forward-over-forward pass.
  2. Precompute static Vandermonde / Savitzky-Golay pseudoinverses on CPU with @functools.lru_cache: Never call jnp.linalg.pinv (SVD) on a static-shape Vandermonde matrix inside a traced function. Compute the filter weights once in NumPy (@lru_cache) and convert to a small jnp.asarray(..., dtype=signal.dtype) kernel for jnp.convolve.

6. Verifying xla_compile_count == 1 & compiles_during_timed == 0

Every hot jitted function or training loop must be verified against unexpected retracing and function-local @jit rebuilds:

  1. In unit tests (eqx.debug.assert_max_traces at module scope):
    inner = eqx.debug.assert_max_traces(step_fn, max_traces=1)
    jitted_step = eqx.filter_jit(inner)
    
    (Never place assert_max_traces on a function defined inside another function or method — its trace counter resets to 0 on every outer call!)
  2. Process-wide XLA compile counter (_CompileCounter) for benchmarks and end-to-end pipelines:
    import logging
    import jax
    
    
    class _CompileCounter(logging.Handler):
        """Counts XLA compilations across the entire process (catches even function-local @jit rebuilds)."""
    
        n = 0
    
        def emit(self, record):
            msg = record.getMessage()
            if "Finished XLA compilation" in msg or "Compiling " in msg:
                _CompileCounter.n += 1
    
    
    def enable_compile_count():
        jax.config.update("jax_log_compiles", True)
        h = _CompileCounter()
        for name in ("jax", "jax._src.dispatch", "jax._src.interpreters.pxla", "jax._src.compiler"):
            lg = logging.getLogger(name)
            lg.addHandler(h)
            lg.setLevel(logging.WARNING)
            lg.propagate = False
    
  3. Interactive diagnostics (jax.explain_cache_misses & jax.log_compiles):
    with jax.explain_cache_misses():
        out1 = jitted_fn(x1)
        out2 = jitted_fn(x2)  # prints exact leaf/shape/id responsible if a cache miss occurs
    

7. Cold Compile vs. Warm block_until_ready() Profiling Protocol

Because JAX dispatch is asynchronous, wall-clock timings without jax.block_until_ready() only measure Python enqueue time, and warmup calls without a compile counter can silently hide function-local @jit rebuilds. Always benchmark on the same GPU and inside the same process:

import time
import jax

enable_compile_count()

# 1. Cold call (compilation + first execution)
t0 = time.perf_counter()
out_cold = jax.block_until_ready(jitted_fn(*args_1))
cold_ms = (time.perf_counter() - t0) * 1e3

# 2. Warm calls (steady-state execution; assert zero XLA compilations during timed window!)
c0 = _CompileCounter.n
warm_times_ms = []
for args_i in warm_inputs:
    t0 = time.perf_counter()
    _ = jax.block_until_ready(jitted_fn(*args_i))
    warm_times_ms.append((time.perf_counter() - t0) * 1e3)
compiles_during_timed = _CompileCounter.n - c0
assert compiles_during_timed == 0, f"Expected 0 warm compiles, got {compiles_during_timed}!"

Report:

  • Cold compile + execute latency (ms)
  • Warm execution latency (ms, median / IQR)
  • compiles_during_timed (0 required for warm timings) & total compilation count across $N$ trials / calls (1 vs. $N$)
  • Numerical parity (np.testing.assert_allclose / relative error) against the reference implementation.

8. Indicating Precision in JAX: jax.default_matmul_precision, DotAlgorithmPreset, and PyTree Policies (Replacing Legacy jmp)

JAX controls precision at two distinct levels, and neither requires DeepMind's archived jmp (0.0.4, last released 2022 for Haiku + float16 loss scaling):

8a. Level 1 — Matmul Accumulator & Tensor-Core Precision (jax.default_matmul_precision & DotAlgorithmPreset)

Even when arrays have dtype jnp.float32, NVIDIA GPUs can execute matmul / einsum / dot_general using different hardware instructions. Native JAX (0.10+) provides two built-in ways to specify this without changing array storage dtypes:

  1. Scoped context manager or per-op precision= (jax.default_matmul_precision):
    • with jax.default_matmul_precision("highest"): (or precision=jax.lax.Precision.HIGHEST on a single jnp.matmul / jnp.einsum): forces full 24-bit IEEE float32 FMA instead of 10-bit TF32. Use this for kinematic trees (ForwardKinematics), skinning (body_models/mhr/skin.py), and whitening/Cholesky paths.
    • with jax.default_matmul_precision("bfloat16"): (or "tensorfloat32"): allows float32 matmuls to round inputs to bf16 (or TF32) on tensor cores while inputs/outputs stay float32.
  2. Explicit jax.lax.DotAlgorithmPreset (jax >= 0.4.30 / 0.10.2):
    • You can pass an explicit <LHS>_<RHS>_<ACCUM> preset directly to precision= (e.g., precision="BF16_BF16_F32", "BF16_BF16_BF16", "TF32_TF32_F32", "F32_F32_F32", "F64_F64_F64").

8b. Level 2 — Array Storage & Activation Dtype Policy (param_dtype, compute_dtype, output_dtype)

To also halve activation bandwidth and guarantee that backward cotangent GEMMs ($\nabla_X = \Delta W^\top$ and $\nabla_W = X^\top \Delta$, which represent $2\times$ the FLOPs of forward) run with bf16 inputs and outputs on tensor cores, hidden activations themselves must be stored as jnp.bfloat16:

  • Why preferred_element_type=jnp.float32 is a trap on bf16 matmuls: NVIDIA tensor cores already accumulate bf16 × bf16 products in fp32 registers internally even when preferred_element_type=jnp.bfloat16. Setting preferred_element_type=jnp.float32 forces the layer output to float32, which makes the backward pass's incoming cotangent float32 and drops both backward GEMMs off the bf16 tensor cores (~2x slower).
  • Why per-layer jmp.Policy (body_models/implicit_representation.py) is suboptimal inside vmap: Calling self.jmp_policy.cast_to_compute((self.weight, self.bias, x)) inside Linear.__call__ when jax.vmap(implicit_traj)(timestamps) is mapped over $T$ frames casts the weights inside the per-timestamp vmap body (and required forking eqx.nn.Linear).
  • Preferred Equinox Pattern — PyTree cast_to_compute once outside vmap (or a 12-line native PrecisionPolicy dataclass):
    @dataclass(frozen=True)
    class PrecisionPolicy:
        param_dtype: jnp.dtype = jnp.float32
        compute_dtype: jnp.dtype = jnp.bfloat16
        output_dtype: jnp.dtype = jnp.float32
    
        def cast_to_compute(self, tree):
            return jax.tree.map(
                lambda x: x.astype(self.compute_dtype) if eqx.is_inexact_array(x) else x, tree
            )
    
        def cast_to_output(self, tree):
            return jax.tree.map(
                lambda x: x.astype(self.output_dtype) if eqx.is_inexact_array(x) else x, tree
            )
    
    Cast the eqx.Module PyTree once at the forward boundary (model_bf16 = policy.cast_to_compute(model_f32)), run standard eqx.nn.Linear / w @ x layers (where bf16 @ bf16 automatically uses preferred_element_type=jnp.bfloat16 with fp32 tensor-core accumulation), and cast the final output back via policy.cast_to_output(y). Reverse-mode AD automatically casts the bf16 weight gradients back to param_dtype (float32) at the boundary for optax.
  • Keep precision-critical operations in jnp.float32 (or float64 for Kalman/Gaussian covariances):
    • Master parameters and Optax/Adam moments (mu, nu).
    • Running observation normalizers (RunningMeanStd) and LayerNorm / RMSNorm variance reductions.
    • Autoregressive state integration (next_state = state_f32 + delta_bf16.astype(jnp.float32)), policy distribution heads (mean, log_std, log_prob), value heads, GAE advantages, and losses.
  • Gate with a single-batch fp32 vs. bf16 gradient equivalence test (test_fast_equiv.py):
    • Assert finite losses/gradients and cosine similarity $> 0.999$ between fp32 and bf16 parameter gradient vectors on an identical batch.

9. Analytic @jax.custom_jvp Geometric Jacobians Over Kinematic Trees (frax Principle)

Calling jax.jacfwd(residual_fn)(q) through a general multi-branch biomechanical tree (e.g., 64 bodies, 94 hinge DOFs, 80+ markers/sites) forces forward-mode autodiff to propagate a $94$-column tangent array through every body rotation and translation in the tree — even though each joint column $j$ only moves its descendant bodies. This causes 20–50 s XLA compile times and 2–10x slower warm LM/Gauss-Newton steps.

Rules for Fast Differentiable Kinematics & Inverse Kinematics (frax @morton2026frax & BodyModels):

  1. Wrap ForwardKinematics in @jax.custom_jvp using the closed-form geometric Jacobian and a static binary ancestor mask:
    • Precompute a static binary ancestor mask M_anc of shape (n_bodies, nv) (or (n_sites, nv)) at model load time, where M_anc[b, j] = 1.0 iff DOF $j$ is on the kinematic path from the root to body/site $b$.
    • In the forward primal pass, compute world body/site positions $\mathbf{p}_b$, world joint anchor positions $\mathbf{p}_j$, and world joint axes $\mathbf{a}_j$.
    • In the JVP rule, compute the exact geometric Jacobian in closed form without tracing through the tree recurrence: $$\frac{\partial \mathbf{p}b}{\partial q_j} = M{\text{anc}}[b, j] \begin{cases} \mathbf{a}_j \times (\mathbf{p}_b - \mathbf{p}_j) & \text{(revolute / hinge DOF } j\text{)} \ \mathbf{a}_j & \text{(prismatic / translation DOF } j\text{)} \end{cases}, \qquad \frac{\partial \boldsymbol{\omega}b}{\partial q_j} = M{\text{anc}}[b, j],\mathbf{a}_j^{\text{rot}}$$
    • Keep jax.jacfwd on the outer residual wrapper (camera_project, gauge weights, linear/polynomial joint couplings): because ForwardKinematics now has a @jax.custom_jvp rule, jacfwd(residual) automatically invokes the analytic geometric Jacobian for the tree and uses AD only for the thin outer residual assembly.
  2. Force precision=jax.lax.Precision.HIGHEST on small $3\times 3$ kinematic matmuls on GPU:
    • On NVIDIA GPUs, JAX defaults float32 matmuls to TF32 (10-bit mantissa), which introduces ~5e-4 to 1.4e-3 drift across chained $3\times 3$ body rotations and lever-arm cross products compared to mjx.kinematics / float64.
    • Always pass precision=jax.lax.Precision.HIGHEST to small $3\times 3$ rotation/tangent matmuls inside FK and its @jax.custom_jvp rule; this restores < 1e-6 GPU float32 parity at negligible runtime cost.
  3. Serial chains vs. shallow branched trees:
    • For long serial chains (manipulators, continuum spines), use jax.lax.associative_scan over $SE(3)$ homogeneous transforms (frax parallel scan) to reduce sequential depth from $O(N)$ to $O(\log_2 N)$.
    • For shallow branched human skeletons (depth $\le 14$, width $64$), a depth-layered vector gather (for d in range(1, max_depth + 1): updating all bodies at depth $d$ in parallel) avoids padding every body to max tree depth.

10. Catching pmap/pjit Recompiles via jax.monitoring & Fixing Sharding-Flip Bugs

Two multi-device compilation traps frequently waste 25–50 minutes per training run without appearing in jax_log_compiles:

  1. jax_log_compiles=True does NOT log pmap compilations in modern JAX (>= 0.4.30 / 0.10+):
    • To catch every jit, pjit, shard_map, and pmap backend compilation, register a listener on "/jax/core/compile/backend_compile_duration" in addition to the log handler:
      COMPILE_DURATIONS_S: list[float] = []
      jax.monitoring.register_event_duration_secs_listener(
          lambda event, duration_s: COMPILE_DURATIONS_S.append(duration_s)
          if event == "/jax/core/compile/backend_compile_duration"
          else None
      )
      
  2. The _device_put_sharded vs. pmap 'i'-Mesh Sharding-Flip Recompile Trap:
    • Legacy jax.device_put_replicated(x, devs) tags array leaves with NamedSharding(mesh={'_device_put_sharded': N}), whereas jax.pmap(..., axis_name='i') returns leaves tagged with NamedSharding(mesh={'i': N}).
    • Symptom A (Epoch-2 recompile): Epoch 1 compiles pmap for _device_put_sharded inputs, and Epoch 2 recompiles the entire pmap step (~40–60 s) because the carry returned by Epoch 1 now has mesh 'i'.
    • Symptom B (Every-eval recompile with donate_argnums): If an evaluator uses @functools.partial(jax.jit, donate_argnums=(0,)) on eval_state while consuming unreplicate(training_state.params), any leaf whose sharding flips between _device_put_sharded and 'i' across calls causes the evaluator to recompile on every single evaluation (~30 s $\times$ 100 evals = 50 min wasted).
    • Fix: Normalize initial replicated state through a 1-step pmap(lambda s: s, axis_name="i") identity (or construct all arrays directly on a single explicit jax.sharding.Mesh), strip weak types (jax.tree.map(lambda x: jnp.asarray(x, dtype=x.dtype) if hasattr(x, "dtype") else x, state)), and commit unreplicated single-device parameters via jax.device_put(leaf[0], jax.devices()[0]) before entering any donate_argnums evaluator.

11. Hoisting Loop-Invariant Work Outside lax.scan & Deferred 1-GEMM Weight Gradients in Unrolled BPTT

When optimizing recurrent rollouts (lax.scan over horizon $H=16\text{--}64$, such as SuperMimic world-model imagined rollouts or PPO unrolls):

  1. Hoist all state-independent gathers, encodings, and PRNG splits outside lax.scan:
    • Never slice reference windows via dynamic_slice_in_dim, split PRNG keys, or run an MLP encoder on reference/goal inputs inside the $H$-step scan body if those inputs do not depend on the evolving recurrent state.
    • Compute the full (H, B, ...) reference slice, encoded goal embeddings, and noise tensors once in vectorized (H, B, ...) form before lax.scan, and pass them as the scanned sequence xs.
  2. Deferred 1-GEMM-per-layer weight gradients for unrolled BPTT through shared MLPs (XMimic fast_nets.py):
    • When backpropagating through an $H$-step lax.scan that applies the same MLP weights $W$ at every step, standard reverse-mode AD computes $H$ separate per-step weight gradients $\nabla_{W}^{(t)} = x_t^\top \delta_t$ and accumulates them.
    • For compute-heavy unrolled horizons ($H \ge 16$), a custom VJP that propagates only activation cotangents $\delta_t$ through the sequential recurrence and then computes $\nabla_W = \text{reshape}(X, (H\cdot B, D_{\text{in}}))^\top \text{reshape}(\Delta, (H\cdot B, D_{\text{out}}))$ after the loop replaces $H$ small GEMMs with one high-occupancy tensor-core GEMM per layer.
  3. Eliminate redundant value-net forward passes in fixed-length rollouts:
    • If episode_length is fixed inside the scan and truncation is identically False on steps $0 \dots T-2$, next_value[t] equals value[t+1]. Evaluate the critic once per step plus one bootstrap call after the loop (T+1 calls total instead of 2T), cutting rollout critic compute in half.

12. Multi-Iteration lax.scan Program Granularity, Persistent Cache Hygiene & Sim-vs-Learner Crossover

  1. Multi-iteration lax.scan program granularity (K = 50–100 updates per JIT call) with on-device ring buffers:
    • Calling separate collect_step, replay_add, wm_step, and policy_step JIT functions from a Python for loop drains the GPU at every boundary.
    • Fuse trajectory collection, on-device ring-buffer insertion (jax.lax.dynamic_update_slice_in_dim along the time/slot axis), uniform window sampling, and learner updates into a single pure carry function scanned over $K \in [20, 100]$ iterations inside one @eqx.filter_jit(donate="all") program. Return stacked scalar metrics of shape (K,) and log them on the host only once per $K$-iteration chunk.
  2. Persistent XLA compilation cache hygiene (JAX_INCLUDE_FULL_TRACEBACKS_IN_LOCATIONS=false):
    • Configure the persistent XLA cache before importing jax in training and benchmark entrypoints:
      import os
      os.environ.setdefault("JAX_COMPILATION_CACHE_DIR", "/workspaces/eqx_pose/.jax_cache")
      os.environ.setdefault("JAX_PERSISTENT_CACHE_MIN_COMPILE_TIME_SECS", "0")
      os.environ.setdefault("JAX_PERSISTENT_CACHE_MIN_ENTRY_SIZE_BYTES", "-1")
      # Omitting full Python tracebacks from HLO op metadata roughly halves cache load/save time:
      os.environ.setdefault("JAX_INCLUDE_FULL_TRACEBACKS_IN_LOCATIONS", "false")
      
  3. Profile the Sim-vs-Learner scaling crossover before optimizing:
    • Batched GPU physics (MJX / MuJoCo-Warp) scales sub-linearly with num_envs until GPU SMs saturate (e.g., 4x more envs from 512 -> 2048 costs only 1.7x wall time), whereas large MLP learners (8×1024 value net over 64 minibatches) are GEMM-bound early and scale linearly (4.7x wall time for 4x envs).
    • At num_envs >= 2048, the learner often consumes 65–75% of total iteration wall time while physics is only 25–35%. Always time (A) env-step scan only vs. (B) full training step at the production batch/env count so you know whether the bottleneck is physics (mjx_impl="warp", njmax, nconmax, solver iterations) or the learner (bf16 GEMMs, critic size, num_updates_per_batch).

Overview

This skill creates a properly structured Python repository from scratch. It enforces the standards in enforce-guidelines/references/repo-structure.md.

Use when:

  • Creating a new Python project
  • Converting an unstructured project to proper layout
  • User says "new repo", "new project", "initialize", "scaffold"

Parameters

  • name (required): Project name (lowercase, hyphens allowed)
  • description (required): One-line project description
  • package_name (optional): Python package name (defaults to name with underscores)
  • author (optional): Author name (defaults to "James Cotton")
  • extras (optional): Additional optional-dependencies groups to include

Steps

1. Validate Inputs

  • You MUST verify project name is lowercase with hyphens only
  • You MUST derive package_name from project name (replace hyphens with underscores)
  • You MUST NOT proceed without a description

2. Create Directory Structure

mkdir -p src/{package_name}
mkdir -p tests
touch src/{package_name}/__init__.py
touch src/{package_name}/py.typed
touch tests/__init__.py
  • You MUST use src/ layout
  • You MUST create both src/ and tests/ directories
  • You MUST include py.typed marker for type hint support

3. Create pyproject.toml

Copy the canonical template from templates/pyproject.toml in the RAE plugin and customize:

  • Replace your-project-name with {name}
  • Replace your_package with {package_name}
  • Replace author name/email
  • Set the project description
  • Add any project-specific dependencies

The template is the single source of truth for ruff, pytest, and coverage config. Do not hardcode these values — always read from the template file.

  • You MUST set line-length = 120
  • You MUST put pytest and ruff in dev optional-dependencies
  • You MUST NOT put dev tools in main dependencies

4. Create .gitignore

# Python
__pycache__/
*.py[cod]
*.egg-info/
dist/
build/
.eggs/

# Virtual environments
.venv/
venv/
ENV/

# IDE
.idea/
.vscode/
*.swp

# Testing
.pytest_cache/
.coverage
htmlcov/

# RAE
scraps/
scratch/
.rae-version

# OS
.DS_Store
Thumbs.db

5. Create README.md

# {name}

{description}

## Installation

```bash
uv pip install -e ".[dev]"

Development

# Run tests
pytest

# Format code
ruff format .

# Lint code
ruff check .

### 6. Create Initial Test Files

Create `tests/conftest.py`:

```python
"""Shared test fixtures."""

Create tests/test_placeholder.py:

"""Placeholder test to verify pytest works."""


def test_placeholder() -> None:
    """Remove this test once real tests exist."""
    assert True
  • You MUST include type hints (-> None)
  • You MUST create conftest.py for shared fixtures

7. Initialize Git (if not already)

git init
git add .
git commit -m "feat: Initialize {name} with RAE structure"
  • You MUST NOT commit if already in a git repo with uncommitted changes
  • You SHOULD offer to commit but confirm with user first

8. Create CLAUDE.md

Copy or adapt templates/CLAUDE.md to create a CLAUDE.md with project-specific instructions, always including the DataJoint read-only rule (if applicable), the Fail-Fast Field Access rule, and the no-Claude-session-links rule:

## CRITICAL: NEVER Modify Database Entries

**DO NOT update, delete, or alter any DataJoint database entries.** This includes:
- `update1()`, `delete()`, `drop()` on any table
- Modifying settings lookup tables (KinematicReconstructionSettingsLookup, ProbabilisticReconstructionSettingsLookup, KineticReconstructionSettingsLookup, KeypointSet, etc.)
- Altering any computed table entries

Database entries are shared state used by the entire lab. Changing a settings entry changes it for everyone and invalidates prior results computed with those settings.

## Fail-Fast Field Access & Anti-Defensive-Programming

Never use defensive `getattr(obj, "field", None)` or `dict.get("field", None)` (nor broad `try/except Exception: pass`) when accessing expected struct, dataclass, Equinox module, or dictionary fields. Use direct named access (`person.mhr` / `person.mhr.joint_coords` or `data["mhr"]`) so schema mismatches fail fast with `AttributeError` or `KeyError`. (In `WheelchairPropulsion`, a defensive `getattr(person, "mhr", None)` / `getattr(person.mhr, "joints_3d", None)` check on a nonexistent attribute name silently evaluated `mhr_hand21` to `None` on every frame without raising an error, silently disabling the 20-DOF finger fit.) Always check for and reject this anti-pattern during code review and PR review.
  • You MUST include this section if datajoint is in the project's dependencies
  • You SHOULD include it by default for any biomechanics/motion-capture project

9. Verify Structure

ruff format .
ruff check .
pytest
  • You MUST run ruff format before completing
  • You MUST run ruff check with no errors
  • You MUST run pytest with all tests passing

10. (Optional) Add Devcontainer

If the user wants devcontainer support, ask which template to use:

Option 1: CPU-only (lightweight, faster startup)

  • Copy from templates/devcontainer-cpu/devcontainer.json
  • Uses mcr.microsoft.com/devcontainers/python:3.11
  • No GPU support, minimal dependencies
  • Best for: web apps, APIs, general Python development

Option 2: GPU-enabled (CUDA + cuDNN for ML/CV)

  • Copy from templates/devcontainer-gpu/devcontainer.json
  • Uses nvidia/cuda:12.6.0-cudnn-devel-ubuntu24.04
  • Includes: GPU access, OpenCV dependencies, git-lfs, Jupyter
  • Requires: --env-file .env in project root
  • Best for: machine learning, computer vision, biomechanics

Both templates:

  • Install Claude Code, pyright, RAE plugin, and full plugin suite via postCreateCommand
  • Mount ~/.claude for config persistence
  • Configure VSCode with ruff, Python testing, 120-char rulers
  • Include Node.js via "ghcr.io/devcontainers/features/node:1": {} (required for excalidraw rendering via npx)
  • No Dockerfile needed - everything via features and install script

Beads (bead-driven development): Ask the user whether they want beads enabled for this project. Beads is off by default. Only enable if the user explicitly opts in. If enabled, add the beads plugin to the devcontainer's postCreateCommand install list.

Note: Ensure ~/.claude exists on the host before starting: mkdir -p ~/.claude

GPU template also requires: Create a .env file in project root for secrets (gitignored)

Adding Optional Dependencies

If user requests specific libraries, add appropriate optional-dependencies:

OpenCV:

[project.optional-dependencies]
opencv = ["opencv-python>=4.0.0"]
opencv-headless = ["opencv-python-headless>=4.0.0"]
opencv-contrib = ["opencv-contrib-python>=4.0.0"]

PyTorch:

[project.optional-dependencies]
torch-cpu = ["torch>=2.0"]
torch-cuda = ["torch>=2.0"]  # User installs with CUDA separately

Jupyter:

[project.optional-dependencies]
notebooks = ["jupyter>=1.0", "ipykernel>=6.0"]

Headless 3D rendering (meshes / point clouds / video overlays):

[project.optional-dependencies]
viz = ["pyrender>=0.1.45", "trimesh>=4.0", "PyOpenGL>=3.1", "opencv-python-headless>=4.0.0"]

This needs the EGL/Vulkan system libraries and ICD manifests from the GPU devcontainer (libgl1, libegl1, libvulkan1, libosmesa6-dev, plus /usr/share/glvnd/egl_vendor.d/10_nvidia.json and /etc/vulkan/icd.d/nvidia_icd.json and -e NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics) — already in templates/devcontainer-gpu. Note that if a user restricts GPUs in runArgs with manual --device=/dev/nvidia* flags, they must also pass --device=/dev/nvidia-modeset and --device=/dev/dri for Vulkan/RTX rendering (e.g. Isaac Sim, SAPIEN). At runtime the project must set PYOPENGL_PLATFORM=egl (and MUJOCO_GL=egl) before importing any GL library and verify it didn't fall back to the llvmpipe CPU rasterizer. Point the user at the /efficient-rendering skill for the device-probing, context-reuse, camera-convention, and video-compositing patterns rather than re-deriving them here.

Examples

User: "/scaffold-repo my-analysis-tool A tool for analyzing motion capture data"

Agent:

  1. Creates directory structure
  2. Generates pyproject.toml with name="my-analysis-tool", package="my_analysis_tool"
  3. Creates .gitignore, README.md
  4. Creates placeholder test
  5. Runs ruff format, ruff check, pytest
  6. Reports success with next steps

User: "/scaffold-repo cv-processor Image processing pipeline --extras opencv-headless"

Agent:

  1. Creates standard structure
  2. Adds opencv-headless to optional-dependencies
  3. Completes verification

Semi-Formal Code Reasoning

This skill implements structured reasoning templates that force explicit evidence gathering before conclusions. Based on research showing 10+ percentage point accuracy gains over unstructured chain-of-thought on patch equivalence, fault localization, and code question answering tasks.

Reference: Ugare, Shubham, and Satish Chandra. 2026. "Agentic Code Reasoning." arXiv:2603.01896. Preprint, arXiv, March 4. https://doi.org/10.48550/arXiv.2603.01896.

Core Principle

Never conclude without tracing. The central failure mode in code reasoning is assuming behavior from names, signatures, or surface similarity. Semi-formal reasoning prevents this by requiring a certificate: explicit premises, concrete execution traces with file:line evidence, and a formal conclusion that follows from the traces. If you can't fill in the template, you haven't gathered enough evidence yet.

When to Use Which Template

TaskTemplateTrigger
Comparing two patches/diffsPatch Equivalence"are these equivalent", "same behavior", PR review with alternatives
Finding buggy code from test failuresFault Localization"find the bug", "why does this fail", "localize the fault"
Understanding code behaviorCode QA"what does this do", "how does X work", "trace through this"
General code reviewCode Review (adapted from Patch Equivalence)"review this PR", "is this change safe"

Pick the closest template. For ambiguous tasks, Code QA is the most general.


Template 1: Patch Equivalence

Use when comparing two code changes to determine if they produce the same behavior.

Structured Exploration Phase

Before filling the certificate, gather evidence systematically:

  1. Read both patches carefully. Identify every file and function touched.
  2. For each function called in the patches:
    • Search for its definition in the repository (do NOT assume standard library semantics — names can be shadowed by module-level or project-specific definitions).
    • Document: FUNCTION: name | LOCATION: file:line | ACTUAL BEHAVIOR: ...
  3. Read the test patch (if available). Identify what behaviors are tested.
  4. Check for side effects: imports, module-level definitions, class inheritance, decorators, middleware, signal handlers — anything that could alter execution semantics.

Certificate Template

DEFINITIONS:
  D1: Two patches are EQUIVALENT MODULO TESTS iff executing the
      repository test suite produces identical pass/fail outcomes.
  D2: The relevant tests are ONLY those in FAIL_TO_PASS and
      PASS_TO_PASS (the existing test suite).

PREMISES (state what each patch does — cite file:line):
  P1: Patch 1 modifies [file(s)] by [specific change, with line refs]
  P2: Patch 2 modifies [file(s)] by [specific change, with line refs]
  P3: The FAIL_TO_PASS tests check [specific behavior]
  P4: The PASS_TO_PASS tests check [specific behavior, if relevant]

FUNCTION RESOLUTION TABLE:
  | Function/Symbol | Assumed Meaning | Actual Definition (file:line) | Different? |
  |-----------------|-----------------|-------------------------------|------------|
  (Fill for EVERY function/method called in either patch)

ANALYSIS OF TEST BEHAVIOR:
  For each FAIL_TO_PASS test:
    Claim 1.1: With Patch 1, test [name] will [PASS/FAIL]
               because [concrete execution trace through code]
    Claim 1.2: With Patch 2, test [name] will [PASS/FAIL]
               because [concrete execution trace through code]
    Comparison: [SAME/DIFFERENT] outcome

  For each relevant PASS_TO_PASS test:
    Claim 2.1: With Patch 1, behavior is [description with trace]
    Claim 2.2: With Patch 2, behavior is [description with trace]
    Comparison: [SAME/DIFFERENT] outcome

EDGE CASES (only those exercised by actual tests):
  E1: [Edge case] → Patch 1: [behavior] | Patch 2: [behavior] | Same? [Y/N]

COUNTEREXAMPLE (if NOT EQUIVALENT):
  Test [name] will [PASS/FAIL] with Patch 1 because [traced reason]
  Test [name] will [FAIL/PASS] with Patch 2 because [traced reason]

  OR

NO COUNTEREXAMPLE EXISTS (if EQUIVALENT):
  All tests produce identical outcomes because [traced reason]

FORMAL CONCLUSION:
  By D1, since test outcomes are [IDENTICAL/DIFFERENT],
  patches are [EQUIVALENT / NOT EQUIVALENT].

ANSWER: [YES/NO]
CONFIDENCE: [HIGH/MEDIUM/LOW] — [brief justification]

Key Anti-Patterns to Avoid

  • Name assumption: Never assume format(), open(), print(), etc. are builtins. Always check for module-level shadowing.
  • Skipping tests: Trace EVERY test, not just the ones that look relevant.
  • Surface similarity: High textual similarity does NOT imply semantic equivalence. Patches can look identical but diverge on one edge case.
  • Premature conclusion: If you find yourself writing the ANSWER before completing the ANALYSIS section, stop and go back.

Template 2: Fault Localization

Use when finding buggy code given a failing test, without execution information.

Structured Exploration Phase

Explore in a hypothesis-driven loop:

HYPOTHESIS H[N]: [What you expect to find and why it may contain the bug]
EVIDENCE: [What from the test or previously read files supports this]
CONFIDENCE: [high/medium/low]

-> Read the file

OBSERVATIONS from [filename]:
  O[N]: [Key observation, with line numbers]

HYPOTHESIS UPDATE:
  H[M]: [CONFIRMED | REFUTED | REFINED] — [Explanation]

UNRESOLVED:
  - [What questions remain]

NEXT ACTION RATIONALE: [Why reading another file, or why enough evidence]

Certificate Template

## Phase 1: Test Semantics Analysis

PREMISE T1: The test calls [X.method(args)] and expects [behavior]
PREMISE T2: The test asserts [condition]
PREMISE T3: [Additional test constraints]

Key question: What is the EXPECTED vs OBSERVED behavior?

## Phase 2: Code Path Tracing

Trace from test entry point into production code:

  METHOD: [ClassName.methodName(params)]
  LOCATION: [file:line]
  BEHAVIOR: [what this method does — VERIFIED by reading source]
  RELEVANT: [why it matters to the test]

  -> [next call in chain]
  ...

## Phase 3: Divergence Analysis

For each traced path, identify where implementation diverges from expectations:

  CLAIM D1: At [file:line], [code] would produce [behavior]
            which contradicts PREMISE T[N] because [reason]
  CLAIM D2: ...

Each claim MUST reference a specific PREMISE and specific code location.

IMPORTANT — Distinguish ROOT CAUSE from CRASH SITE:
  - The crash site is where the error manifests (e.g., StackOverflowError)
  - The root cause is the code that creates the bad state
  - Always trace BACKWARD from crash site to root cause

## Phase 4: Ranked Predictions

  Rank 1 ([confidence]): [file:lines] — [root cause description]
    Supporting claims: D[N], D[M]
  Rank 2 ([confidence]): [file:lines] — [description]
    Supporting claims: D[N]
  ...up to Rank 5

Key Anti-Patterns to Avoid

  • Stopping at the crash site: The method that throws the exception is usually NOT the root cause. Trace backward to find what created the bad state.
  • Single-turn predictions: Don't predict from test code alone. Read source files.
  • Name-based guessing: Don't assume a method is buggy because its name is similar to the failing test. Trace the actual execution path.
  • Ignoring indirection: Bugs are often in configuration classes, factory methods, or helper utilities — not in the method directly called by the test.

Template 3: Code Question Answering

Use when answering questions about code behavior, semantics, or design.

Certificate Template

FUNCTION TRACE TABLE:
  | Function/Method | File:Line | Parameter Types | Return Type | Behavior (VERIFIED) |
  |-----------------|-----------|-----------------|-------------|---------------------|
  (Every function examined — MUST read source, not guess from name)

DATA FLOW ANALYSIS:
  Variable: [key variable name]
  - Created at: [file:line]
  - Modified at: [file:line(s), or 'NEVER MODIFIED']
  - Used at: [file:line(s)]
  (Repeat for each key variable relevant to the question)

SEMANTIC PROPERTIES:
  Property 1: [e.g., "HashMap is mutable", "this list is sorted"]
  - Evidence: [specific file:line]
  Property 2: ...

ALTERNATIVE HYPOTHESIS CHECK:
  If the opposite answer were true, what evidence would exist?
  - Searched for: [what you looked for]
  - Found: [what you found — cite file:line]
  - Conclusion: [REFUTED / SUPPORTED]

ANSWER: [Final answer with explicit evidence chain]
CONFIDENCE: [HIGH/MEDIUM/LOW]

Key Anti-Patterns to Avoid

  • Answering from function names: getUserName() might not return a user's name. Read the implementation.
  • Missing downstream handling: Even if you find an edge case, check whether downstream code already handles it before claiming it's a problem.
  • Incomplete trace chains: If your answer depends on 5 functions, trace all 5. A chain is only as strong as its weakest link.

Template 4: Code Review (Adapted)

Use for general PR/diff review when there's no second patch to compare against.

Certificate Template

CHANGE SUMMARY:
  C1: [file] line [N-M]: [what changed and why, based on PR description]
  C2: ...

FUNCTION RESOLUTION TABLE:
  | Function/Symbol | Called In Change? | Definition (file:line) | Semantics Verified? |
  |-----------------|-------------------|------------------------|---------------------|

BEHAVIORAL ANALYSIS:
  For each change C[N]:
    - Pre-change behavior: [traced from code]
    - Post-change behavior: [traced from code]
    - Regression risk: [NONE / LOW / MEDIUM / HIGH] — [specific reason]

INVARIANT CHECK:
  For each invariant or assumption the code relies on:
    I1: [invariant] — PRESERVED / BROKEN by change C[N] because [reason]
    I_FAILFAST (MANDATORY): Does the diff introduce any defensive `getattr(obj, "field", None)`,
        `d.get("field", None)`, or broad `try/except Exception: pass` on known/expected schemas?
        (Cite exact lines; reject in favor of direct `obj.field` / `d["field"]` access so schema
        mismatches raise `AttributeError` / `KeyError` rather than silently disabling downstream
        stages as in the `WheelchairPropulsion` `getattr(person, "mhr", None)` 20-DOF finger-fit bug.)

EDGE CASES:
  E1: [input/state] -> Pre: [behavior] | Post: [behavior] | Safe? [Y/N]

CONCLUSION:
  [APPROVE / REQUEST CHANGES / NEEDS DISCUSSION]
  Summary: [1-2 sentence summary of findings]
  Issues found: [list, or "none"]

General Guidance

Calibrating Depth

Not every analysis needs the full template. Use judgment:

  • Quick check (high-confidence, small diff, single file): Fill the template mentally, write a condensed version. ~5 minutes equivalent.
  • Standard analysis (moderate complexity, 2-5 files): Full template, moderate exploration. ~15-30 agent steps.
  • Deep analysis (subtle semantics, cross-module dependencies, framework-specific behavior): Full template with extensive exploration. ~30-50 agent steps.

If the user asks for a quick review, provide a condensed version but flag if you spot something that warrants deeper analysis.

When Semi-Formal Reasoning Helps Most

The accuracy gains are largest when:

  • Code spans multiple files with non-obvious dependencies
  • Functions or symbols might be shadowed or overridden
  • The difference between correct and incorrect hinges on a subtle semantic detail
  • Framework-specific behavior differs from standard library behavior
  • Multiple test cases need to be reasoned about independently

When It Helps Less

  • Single-file, self-contained code snippets
  • Pure syntax or formatting questions
  • Cases where the model already has strong baseline accuracy
  • When speed matters more than thoroughness (flag the tradeoff to the user)

Running Independent Probes

When you cannot execute repository code but need to verify language semantics (e.g., "what does Python's format() builtin do with this input?"), you may write and run small independent scripts that test general language behavior. This is legitimate static analysis — you are verifying your understanding of the language, not executing the repository's code.

Presenting Results

After completing the certificate:

  1. Lead with the answer — don't make the user read the whole trace first.
  2. Summarize key findings in 2-3 sentences.
  3. Offer the full certificate as supporting evidence if the user wants to audit your reasoning.
  4. Flag confidence level and explain what would change your mind.

~80% ASD-STE100 Technical Specifications & Explanations (ste100-spec)

Use this skill whenever writing, auditing, or updating technical specifications (docs/SPEC.md, docs/specs/*.md, or docs/papers/<slug>/SPEC.md) or explaining complex codebases, algorithms, and pull requests in ~80% ASD-STE100 (Simplified Technical English).

Background (The Karpathy ~80% ASD-STE100 Pattern): ASD-STE100 is the international aerospace maintenance specification standard designed to eliminate ambiguity in safety-critical engineering documentation. Strict 100% ASD-STE100 restricts vocabulary to a ~900-word dictionary, which forces awkward circumlocutions for modern machine learning, computer vision, and biomechanics concepts. At ~80% strictness (as highlighted by Andrej Karpathy), you retain exact domain-specific technical nouns and symbols (MHR, SequenceIK, Cholesky, ELBO, float32, multi_camera) while strictly enforcing ASD-STE100 grammar, sentence-length bounds, active voice, and one-idea-per-sentence structure. The result is dense, zero-fluff engineering prose that humans and LLMs can parse and verify at a glance.


1. Core ~80% ASD-STE100 Writing Rules

Every specification or STE-100 explanation MUST obey these six grammatical and structural constraints:

RuleConstraintWhy It Matters
1. Active Voice OnlyEvery sentence must name the explicit actor (<Actor> <verb> <object>). Passive voice ("is computed", "are filtered", "was optimized") is forbidden.Passive voice hides which module, function, or stage performs the operation.
2. Strict Sentence Length LimitsProcedural / pipeline steps: $\le 20$ words per sentence.
Descriptive / conceptual context: $\le 25$ words per sentence.
Short sentences prevent nested subordinate clauses and force explicit intermediate states.
3. One Idea Per SentenceEach sentence states one action, invariant, or causal relationship. Split compound sentences ("X does A while Y does B, which enables Z") into numbered sequential steps.Prevents hidden assumptions and makes every sentence independently falsifiable.
4. One Term Per Concept (Zero Synonym Drift)Pick one exact technical term for each concept and use it identically throughout the document. Never alternate synonyms for stylistic variety.Calling the same tensor pose, joint_angles, qpos, and kinematic_state across paragraphs causes implementation bugs.
5. Maximum 3-Word Noun ClustersBreak up stacked 4+ word noun chains ("multi-camera keypoint trajectory optimization weight" $\to$ "weight for multi-camera trajectory optimization").Long noun clusters obscure which word modifies which noun.
6. Explicit Shapes, Frames & Physical UnitsState array shapes ((T, J, 3)), data types (float32), coordinate frames (world, camera, pelvis-root), and physical units (mm, deg, rad, N/kg, Hz) inline. Strip all marketing adjectives ("seamless", "robust", "cutting-edge", "novel", "crucially").Eliminates unit/frame mismatch bugs and AI filler prose.

1.1 Before vs. After Examples (AI Slop vs. ~80% ASD-STE100)

Standard / AI-Generated Prose (Forbidden in Specs)~80% ASD-STE100 Specification (Required)
"Optimization of the biomechanical trajectory is seamlessly performed across multiple views by leveraging our novel implicit representation to minimize reprojection errors alongside temporal smoothness priors." (26 words, passive, vague actor, marketing fluff)1. ImplicitTrajectoryNet maps normalized timestamps $t \in [-1, 1]$ to joint angles $\mathbf{q}_t \in \mathbb{R}^{49}$ (rad). (16 words)
2. ForwardKinematics transforms $\mathbf{q}_t$ into 3D markers $\mathbf{X}_t \in \mathbb{R}^{M \times 3}$ (mm, world frame). (14 words)
3. SequenceIK minimizes the sum of robust 2D reprojection loss and joint velocity penalties using L-BFGS. (16 words)
"When handling missing detections, a confidence threshold is applied and low-quality keypoints are masked out before triangulation occurs." (18 words, passive twice, unspecified threshold/shape)1. filter_keypoints zeros 2D detections with confidence $c_{t,k} < 0.30$ in keypoints_2d (T, C, K, 3). (16 words)
2. triangulate_ransac requires at least two cameras with valid detections to estimate 3D point $\mathbf{x}_{t,k}$ (mm). (16 words)
"The handrim contact phase is robustly identified using kinematic heuristics and subsequently refined via an Extended Kalman Filter." (18 words, passive, hides exact signals)1. ContactClassifier predicts contact probability $p_t \in [0, 1]$ from 3D wrist-to-handrim distance (mm) and tangential speed (m/s). (18 words)
2. PhaseEKF updates continuous propulsion phase $\phi_t \in [0, 2\pi)$ (rad) at 60 Hz. (14 words)

2. Architectural Decision Framework: Single Unified Spec vs. Multi-Specification Suite

Before creating specification files in a repository or paper directory, decide whether the system belongs in one unified specification or a modular suite of specifications.

2.1 The 3-Question Split Test

Default to one unified specification (docs/SPEC.md or docs/papers/<slug>/SPEC.md). Split into multiple modular specifications (docs/specs/) if and only if at least one of the following three conditions holds:

  1. Independent Subsystem Contracts: Does the repository contain distinct subsystems with independent input/output contracts that can be executed and tested in isolation? (Example: (1) offline CT anatomical atlas construction, (2) multi-GPU flow-matching training, and (3) real-time single-camera clinical inference.)
  2. Independent Downstream Consumers: Do external repositories or users import or invoke one subsystem without running the others? (Example: BodyModels forward kinematics vs. MultiCameraTracking 2D-to-3D triangulation.)
  3. Complexity & Length Ceiling: Would a single end-to-end specification exceed 150 lines or 40 numbered procedural steps?

2.2 Decision Matrix

System ArchetypeLayout ChoiceFile PathsLine & Step Budget
Single End-to-End Pipeline or Focused Paper
(e.g., WheelchairPropulsion, ProstheticComponentry, single clinical validation or method paper)
Single Unified SpecRepo: docs/SPEC.md
Paper: docs/papers/<slug>/SPEC.md
$\le 120$ lines total
($\le 30$ procedural steps)
Multi-Subsystem Platform or Coupled Method Suite
(e.g., BodyModels + MhrFkOpt, multi-stage training + calibration + deployment repo)
Modular Spec Suite + Mandatory Top-Level Overviewdocs/specs/00_overview.md
docs/specs/01_<subsystem_a>.md
docs/specs/02_<subsystem_b>.md
docs/specs/03_<subsystem_c>.md
00_overview.md: $\le 35$ lines
Each 0N_<module>.md: $\le 100$ lines

MANDATORY RULE FOR MULTI-SPEC SUITES (docs/specs/00_overview.md): Never create a directory of numbered spec files without 00_overview.md. 00_overview.md must stay under 35 lines and contain:

  1. A 2-sentence system purpose statement.
  2. A dataflow table or ASCII diagram showing the exact tensor/file handoff between 01_..., 02_..., and 03_....
  3. Direct links to each module specification.

3. Canonical ~80% ASD-STE100 Templates

Starter templates are also available in references/templates.md.

3.1 Template A: Project / Repository Specification (docs/SPEC.md)

Use this structure to specify a repository's production pipeline in $\le 120$ lines:

# Technical Specification: <Repository or System Name>

> **Format:** ~80% ASD-STE100 (Active voice; $\le 20$ words per procedural step; $\le 25$ words per descriptive sentence; explicit shapes, frames, and units).
> **Code Anchor Commit:** `<git-sha>`

## 1. Purpose and Scope
- **System Actor:** `<package_name>` (`src/<package_name>/`)
- **Primary Function:** <1–2 active-voice sentences ($\le 25$ words each) stating what the system transforms from input to output.>
- **Out of Scope:** <1 sentence ($\le 25$ words) stating what upstream/downstream systems handle.>

## 2. Terminology and Coordinate Conventions
| Term / Symbol | Exact Definition | Shape & Type | Frame & Units |
| :--- | :--- | :--- | :--- |
| `<var_name>` | <Single-meaning definition> | `(T, J, 3)` `float32` | `world`, `mm` |

## 3. Input Contracts
1. `<Entrypoint>` accepts `<input_a>` (`<shape>`, `<dtype>`, `<units>`) from `<source>`.
2. `<Entrypoint>` validates that `<precondition>` holds; otherwise `<Entrypoint>` raises `ValueError`.

## 4. Sequential Pipeline Stages
*(Every step MUST use active voice and contain $\le 20$ words.)*

### Stage 1: <Stage Name> (`src/<pkg>/<module>.py:<func>`)
1. `<Actor>` loads `<input>` and normalizes `<field>` to `<range>` (`<units>`).
2. `<Actor>` computes `<intermediate>` (`<shape>`, `<dtype>`) using `<algorithm>`.

### Stage 2: <Stage Name> (`src/<pkg>/<module>.py:<func>`)
3. `<Actor>` projects `<intermediate>` into `<target_space>` (`<shape>`, `<units>`).
4. `<Actor>` minimizes `<objective>` over `<N>` iterations with step size `<lr>`.

## 5. Output Contracts and Invariants
- **Primary Output:** `<output_name>` (`<shape>`, `<dtype>`, `<units>`).
- **Hard Invariants:**
  1. `<Tensor>` contains zero `NaN` or `Inf` values across all valid frames.
  2. `<Physical constraint>` holds within tolerance `±<tol> <units>`.

## 6. Verification and Benchmark Gates
| Gate | Command | Pass Criterion |
| :--- | :--- | :--- |
| Unit & Parity Suite | `pytest tests/` | 100% pass, coverage $\ge 80\%$ |
| Primary Benchmark | `python scripts/<eval>.py` | `<Metric>` $\le <threshold> <units>` on `<cohort>` ($N = <count>$ participants) |

3.2 Template B: Paper Technical Specification (docs/papers/<paper_slug>/SPEC.md)

Write docs/papers/<paper_slug>/SPEC.md before drafting prose in methods.md and results.md, and keep it synchronized with the code and manuscript. While the manuscript (methods.md, results.md) uses flowing pre-2026 narrative paragraphs (per /isr-paper-writing), SPEC.md provides the crisp, unambiguous ~80% STE-100 engineering blueprint that prevents symbol drift, degree-of-freedom bugs, and code-to-paper mismatches.

# Paper Specification: <Paper Title> (`<paper_slug>`)

> **Format:** ~80% ASD-STE100 Engineering Blueprint for [`manuscript.md`](manuscript.md).
> **Code Provenance Commit:** `<git-sha>`

## 1. Core Problem and Thesis
1. **Measurement Gap:** <Active-voice sentence ($\le 25$ words) stating what clinical/biomechanical quantity existing systems fail to measure at scale.>
2. **Core Methodological Thesis:** <Active-voice sentence ($\le 25$ words) stating how our unified formulation solves this gap.>
3. **Statistical Unit of Analysis:** <State the exact independent unit, e.g., $N = 12$ participants ($df = 11$), never pooled trials or frames.>

## 2. Notation and Symbol Contract
*(Every symbol in `methods.md` and `results.md` MUST match this table without synonym drift.)*

| LaTeX Symbol | Code Variable (`file:line`) | Definition | Shape & Units |
| :--- | :--- | :--- | :--- |
| $\boldsymbol{\theta}_t$ | `qpos` (`src/model.py:L42`) | Generalized joint coordinates at frame $t$ | `(D,)` `rad` / `m` |

## 3. Unified Method Stages ($\le 20$ Words Per Step)
1. **Stage 1 (<Name>):** `<Component>` maps `<input>` to `<latent/state>` (`<!-- CODE: src/...:L10-L45 -->`).
2. **Stage 2 (<Name>):** `<Component>` predicts `<output>` by minimizing `<objective>` (`<!-- CODE: src/...:L50-L95 -->`).
3. **Inference Label-Blindness Invariant:** At test time, `<Model>` receives zero ground-truth clinical labels, forceplate signals, or per-trial bias corrections.

## 4. Primary Quantitative Claims and Evidence Anchors
| Claim ID | Manuscript Section | STE-100 Claim Statement ($\le 25$ words) | Metric & Value ($N$, $df$) | Reproduction Script & Output JSON |
| :--- | :--- | :--- | :--- | :--- |
| **C1** | `results.md` §3.1 | `<OurMethod>` reduces 3D joint error compared to `<Baseline>` on `<Cohort>`. | `<X.X ± Y.Y mm>` vs. `<A.A ± B.B mm>` ($N=12, p=0.002$) | `scripts/eval.py` $\to$ `outputs/eval.json` |

## 5. Ablations That Mattered
| Variant | What Changes Relative to Unified Method | Quantitative Effect |
| :--- | :--- | :--- |
| **Unified Method (Ours)** | Complete formulation (Stages 1–3). | `<Primary Metric>` |
| **Ablation 1 (`w/o <Component>`)** | `<Module>` removes `<term>`. | Error increases by `+<delta> <units>`. |

## 6. Explicit Scope Boundaries and Limitations
1. **Observability Limit:** `<System>` cannot observe `<unobservable quantity>` from `<modality>` alone.
2. **Cohort / Domain Limit:** Training data covers `<population>`; `<out-of-domain case>` requires `<adaptation>`.
3. **Compute Cost:** `<Hardware>` processes a `<duration>` trial in `<time>` (`<fps> Hz`).

3.3 Template C: Ad-Hoc ~80% STE-100 Explanation (Code, Bug, or PR Walkthrough)

When the user asks you to "explain how X works in STE-100", "summarize this PR in STE-100", or "give me a Karpathy STE-100 walkthrough", format your response directly using this structure:

### ~80% ASD-STE100 Walkthrough: <Topic / PR / Subsystem>

**Purpose:** <1 active-voice sentence ($\le 25$ words) stating what the code or change accomplishes.>

**Inputs:**
- `<arg_1>` (`<shape>`, `<dtype>`, `<units>`): <1-line description>

**Sequential Execution Steps ($\le 20$ words per sentence):**
1. [`<func_a>`](file:///abs/path/to/file.py#L10-L25) reads `<arg_1>` and validates `<condition>`.
2. [`<func_b>`](file:///abs/path/to/file.py#L30-L60) transforms `<state_1>` into `<state_2>` (`<shape>`, `<units>`).
3. [`<func_c>`](file:///abs/path/to/file.py#L65-L90) writes `<output>` to `<destination>`.

**Outputs & Invariants:**
- Returns `<output>` (`<shape>`, `<dtype>`, `<units>`) with invariant `<condition>`.

4. Evergreen Specification Maintenance Rule

Specifications in docs/SPEC.md, docs/specs/*.md, and docs/papers/<slug>/SPEC.md are evergreen engineering contracts, not historical changelogs:

  1. Same-Commit Synchronization Mandate: Whenever a code change or manuscript revision modifies input/output shapes, coordinate conventions, pipeline stages, loss terms, degrees of freedom, or headline benchmark numbers, you must update the corresponding STE-100 specification in the same commit / PR.
  2. Never Document Historical Evolution in SPEC.md: Do not write "In v1 we used X, and in v2 we switched to Y" inside a specification. Document only what the current production code does today. Leave change history to git log and the Pull Request description.
  3. Self-Audit Checklist Before Committing a Spec:
    • Does every procedural step use active voice (<Actor> <verb> <object>) and contain $\le 20$ words?
    • Does every descriptive sentence contain $\le 25$ words and express one idea?
    • Are all marketing adjectives ("robust", "seamless", "novel", "cutting-edge") removed?
    • Does every tensor/quantity specify its shape, dtype, coordinate frame, and physical units (mm, deg, rad, N/kg, Hz)?
    • Did you apply the 3-Question Split Test (single docs/SPEC.md $\le 120$ lines vs. modular docs/specs/00_overview.md + 01_<module>.md)?
    • Does every stage and quantitative claim link to an exact file:line or reproduction script?

Tufte Data Visualization

Apply Edward Tufte's principles whenever generating or reviewing code that renders data visually. This skill covers chart generation, not slide/presentation design.

Workflow

Follow these steps in order when creating any chart:

Step 1: Identify the message

Before writing code, determine:

  1. The key finding or trend the chart must make visible.
  2. The comparison context — a baseline, prior period, target, or peer group. A number without context is meaningless.
  3. The chart type that best fits the data structure (see Chart type guidance below).

Step 2: Apply universal rules

Review the rules below. Every rule is a default — deviate only when the user explicitly requests otherwise.

Step 3: Apply library-specific config

Use the Library quick reference table to find the essential overrides for the target library. For complete code examples and helper functions, read ONE rule file from rules/ matching the library.

Step 4: Validate

Run through the validation checklist at the bottom of this file before presenting the chart.


Universal rules

Rules 1–14 cover static principles; 15–19 extend them for screens; 20–22 address content and formatting.

1. Remove top and right borders

No chart should have top or right axis lines, borders, or spines. The bottom and left axes are sufficient. Top and right lines are pure chartjunk.

2. Direct labels, not legends

Label each data series directly — at the endpoint of a line, on or beside a bar, next to a cluster. Remove the <Legend> component entirely. If there is only one series, the chart title provides that context; no label is needed.

3. No gridlines by default

The default is zero gridlines. For static charts where users need to read precise values, add horizontal-only gridlines at very low opacity (0.08–0.12). For interactive charts, prefer a contextual crosshair on hover instead (see rule 15). Never add vertical gridlines.

4. Range-frame axes

Axis lines should span only the range of the data, not from zero to some arbitrary maximum. The axis starts at (or near) the minimum data value and ends at the maximum.

5. No 3D effects

No perspective, no depth, no shadows on chart elements. Two-dimensional data gets two-dimensional representation.

6. No pie charts unless explicitly requested

Default to a horizontal bar chart sorted by value. If the user explicitly asks for a pie chart: maximum 4 slices, 2D only, start at 12 o'clock, direct percentage labels on each slice.

7. Aspect ratio ~1.5:1

Charts should be approximately 50% wider than tall. Standard sizes: 600x400, 750x500, 900x600. Exception: sparklines and small multiples may be more compact.

8. Gray first, highlight selectively

The default data series color is medium gray (#666). Use a single accent color to highlight the most important series or data point. Never use more than 4 distinct colors. Choose the right palette type: categorical (4-color muted) for unordered groups, sequential (single-hue ramp) for ordered magnitude, diverging (two-hue from center) for deviation from a midpoint. See rules/typography-and-color.md for hex values.

9. Transparent background for exported figures (PDF / SVG / PNG)

Default to a transparent background (transparent=True / facecolor="none") for all exported static figures (PDF, SVG, PNG) so they embed seamlessly in LaTeX manuscripts, Markdown docs, and slides without a colored bounding box. Only use #fffff8 (light) or #151515 (dark) as an HTML page background when building a standalone web page whose entire canvas shares that color.

10. Serif fonts for data

Use serif fonts for data labels, annotations, and chart titles: "ET Book", "Palatino Linotype", Palatino, "Book Antiqua", Georgia, serif. Sans-serif (system-ui, sans-serif) is acceptable only for small axis tick labels (11-12px).

11. No dual y-axes

Two y-axes on one chart create false implied correlations. Use small multiples instead — two charts stacked vertically with shared x-axis.

12. Annotate the notable

If the data contains a peak, trough, inflection point, or event boundary, add a text annotation pointing to it directly on the chart. Place annotations in the nearest clear space — offset from the data point with a short leader line if needed. When multiple annotations compete for space, keep only the most important; move others to a footnote or tooltip.

13. Show comparison context

Include at least one reference element: a reference line (average, target, prior period), a shaded band, or a second series. A chart showing one line with no context fails the "Compared to what?" test.

14. Minimal tooltips

Tooltips should be plain text with the data value and label. No colored background, no border, no arrow pointer, no shadow.

15. Progressive disclosure over static density

Default to the Tufte-clean overview — high data-ink, minimal chrome. Layer details through hover, tap, and click (values, annotations, comparisons). Don't frontload everything onto a single static view. A contextual crosshair on hover replaces permanent gridlines.

16. Accessible by default

3:1 contrast ratio minimum for chart elements against their background; 4.5:1 for text in charts. Never use color as the sole differentiator — pair with shape, pattern, or direct label. Provide a text alternative for every chart (aria-label with key finding, or companion data table). Interactive charts must be keyboard-navigable.

17. Responsive, not just resized

Charts must have a responsive strategy — fluid (percentage width + viewBox), adaptive (breakpoint-based layout changes), or hybrid. At narrow viewports, change chart type or layout (horizontal bars for categories, reduced tick density, abbreviated labels), don't just shrink.

18. Animate to explain, not to decorate

Transitions for data changes (sorting, filtering, time progression) are good — they help the viewer track transformations. Gratuitous entrance animations, bouncing, and decorative motion are chartjunk. Duration: 200–500ms, ease-out. Always respect prefers-reduced-motion.

19. Dark mode as first-class citizen

Design both light and dark palettes intentionally. Never invert colors. Reduce saturation in dark mode (bright colors "vibrate" on dark backgrounds). Respect prefers-color-scheme. Use semantic color tokens (--tufte-bg, --tufte-text, --tufte-series-default) so charts adapt automatically.

20. Titles assert findings

The chart title states the key insight, not the axis description. "Revenue Surged 23% in Q3" not "Revenue by Quarter, 2024". The subtitle can provide context ("vs. prior year, USD millions"). If the data has no clear finding, the chart may not be needed (see rule 22).

21. Format numbers for humans

Abbreviate large numbers: $1.2M not $1,200,000. Use thousand separators for mid-range numbers (12,450 not 12450). Match decimal precision to significance (don't show $4.2391M when $4.2M suffices). Right-align numbers in tables. Use consistent units and state them once (in the axis label or title), not on every data point.

22. Don't chart what a sentence can say

If the data is 1–2 numbers, write a sentence with inline context ("Revenue was $4.2M, up 23% from Q2"). If the data is a simple ranking of 3–5 items, consider a table. Charts earn their space by revealing patterns, trends, or distributions that text and tables cannot. A chart of two bars is almost always worse than a sentence.


Library quick reference

The universal rules above are sufficient for most charts. For complete code examples and library-specific helpers, read the appropriate rule file from the rules/ directory in this skill's folder. Only read ONE rule file per task.

LibraryRule file to readEssential config (apply even without reading the file)
Rechartsrules/recharts.md<CartesianGrid stroke="none" />, remove <Legend />, <YAxis axisLine={false} tickLine={false} />, <Line dot={false} strokeWidth={1.5} />
EChartsrules/echarts.mdsplitLine: { show: false }, legend: { show: false }, grid: { show: false }, use endLabel on series
Chart.jsrules/chartjs.mdgrid: { display: false }, border: { display: false }, plugins.legend.display: false, use chartjs-plugin-datalabels
matplotlibrules/matplotlib.mdspines['top'].set_visible(False), spines['right'].set_visible(False), spines['bottom'].set_bounds(min, max), font.family: serif, pdf.fonttype: 42, ps.fonttype: 42, svg.fonttype: "none", mathtext.fontset: "stix"
Plotlyrules/plotly.mdshowgrid=False, showlegend=False, plot_bgcolor='#fffff8', zeroline=False
D3/SVG/HTMLrules/svg-html.md.domain { display: none }, no <rect> backgrounds, stroke-opacity: 0.1 for any gridlines

Chart type guidance

TypeKey settings
Line1.5–2px stroke, dot={false} unless <7 points (then r=2), direct label at rightmost point
BarPrefer horizontal for categories, sort by value descending, direct value labels, #7a7a7a default fill
ScatterGray dots #999 r=3, highlight key cluster/outlier with accent, regression line if meaningful (dashed, thin)
Time seriesLabel events on chart ("Recession", "Launch"), range-frame x-axis, YoY via opacity (current solid, prior 30%)
Small multiplesSame scale ALL panels, shared axis labels (x on bottom row, y on left column), no panel borders
Sparklines~80x20px, no axes/labels/gridlines, min/max dots r=1.5, embed inline in text or table cells
Data tablesNo zebra striping, whitespace + thin rules every 3–5 rows, right-align numbers, font-feature-settings: 'onum' 1
SlopegraphBefore/after categories, label both endpoints (value + name), gray default + highlight key slopes
AreaPrefer lines. If area: fillOpacity 0.03–0.08, no gradient, direct labels at endpoints
Stacked barAvoid — use small multiples instead. If forced: sort by total, direct labels per segment, max 4 segments
HeatmapSequential or diverging palette only, value labels in cells, companion data table for accessibility

For small multiples, sparklines, and slopegraph implementation patterns, see rules/small-multiples-sparklines.md.


Color quick reference

TokenLightDark
Background (exported PDF/SVG/PNG)transparent (none)transparent (none)
Background (standalone full-page web canvas only)#fffff8#151515
Text#111#ddd
Text secondary#666#999
Axis/rule#ccc#444
Grid (if used)#eee (8-12% opacity)#333
Default series#666#999
Highlight#e41a1c#fc8d62

Categorical (max 4): #4e79a7 steel blue · #f28e2b tangerine · #e15759 coral · #76b7b2 sage

Font stacks in rule 10. For full palettes (sequential, diverging), font loading, and old-style figures, see rules/typography-and-color.md.


Anti-pattern detection

When reviewing existing chart code, check for: legends (→ direct labels), pie charts (→ horizontal bars), 3D effects (→ flat 2D), dual y-axes (→ small multiples), heavy gridlines (→ remove or 0.1 opacity), rainbow palettes (→ gray + accent), gauge widgets (→ number + sparkline), gradient fills (→ solid color), rotated labels (→ flip axes or abbreviate), pure white/black backgrounds (→ #fffff8/#151515), hover-only information (→ tap/focus fallback), missing text alternatives (→ aria-label), color-only encoding (→ add shape/pattern).

For the full table with per-library detection patterns and one-liner fixes, see rules/anti-patterns.md.


Validation checklist

Before presenting any chart, verify:

  • No top or right borders/spines
  • No Legend component — series labeled directly on the chart
  • Gridlines removed or horizontal-only at opacity <= 0.12
  • Aspect ratio approximately 1.5:1
  • Background is transparent (transparent=True / none) for exported PDF/SVG/PNG figures (only use #fffff8 if the entire surrounding HTML page is #fffff8)
  • Serif font for data labels and titles
  • Default series color is gray (#666); color used only for emphasis
  • No 3D effects, no pie chart (unless explicitly requested)
  • Axis lines span only the data range (range-frame)
  • Notable data features annotated directly on chart
  • Comparison context present (reference line, band, or second series)
  • Tooltips are plain text with no decorative styling
  • Interactive elements have tap/click/focus alternatives (no hover-only)
  • Contrast ratios meet 3:1 (elements) / 4.5:1 (text) minimums
  • Chart has a text alternative (aria-label, description, or data table)
  • Animations respect prefers-reduced-motion
  • Charts render usably at 320px and 1440px+ widths
  • Title states the finding, not the axis description
  • Numbers are formatted for readability (abbreviations, separators, consistent precision)
  • A chart is warranted — the data couldn't be communicated as a sentence or table

Additional resources

Library rules (read ONE per task): rules/recharts.md, rules/echarts.md, rules/chartjs.md, rules/matplotlib.md, rules/plotly.md, rules/svg-html.md — complete code examples, helpers, and theme registrations.

Cross-cutting (read when specifically needed):

  • rules/interactive-and-accessible.md — progressive disclosure, WCAG, responsive, animation, dark mode
  • rules/typography-and-color.md — font loading, full palette tables, old-style figures
  • rules/anti-patterns.md — per-library detection heuristics and fixes
  • rules/small-multiples-sparklines.md — layout patterns for small multiples, sparklines, slopegraphs

Working examples in examples/ — one per library, plus an inline SVG sparkline.


Biomechanical Manuscript Figure Taxonomy & Exemplars (ISR Lab Standard)

When generating figures for scientific manuscripts (templates/myst-paper, ISR single-column & IEEE two-column builds), adhere to these mandatory rules and reference exemplars:

  1. Export .svg as the Primary Format for MyST Markdown Alongside .pdf & .png (transparent=True):
    • Always save programmatic Python/Matplotlib/vector figures as .svg (the primary format referenced in MyST .md files via ![](figures/fig_name.svg) so MyST Studio and HTML builds render crisp vector graphics natively) alongside .pdf and .png.
    • MyST (myst build --tex / myst build --pdf) automatically converts .svg -> .pdf via inkscape (or our zero-GUI rsvg-convert -f pdf CLI shim managed by scripts/ensure_inkscape_and_check_fonts.py).
    • Never use cream/off-yellow (#fbfaf5) backgrounds for manuscript figures (off-yellow only works in books matching that color). Use transparent=True on all fig.savefig(...) calls for .svg, .pdf, and .png manuscript figures so they blend seamlessly into both ISR Preprint and IEEE two-column pages.
  2. Prevent IEEE-Rejected Type 3 Fonts (pdf.fonttype = 42, ps.fonttype = 42, svg.fonttype = "none", mathtext.fontset = "stix"):
    • IEEE Xplore and PDF eXpress strictly reject PDFs containing Type 3 bitmap fonts or unembedded fonts (emb = no in pdffonts). Matplotlib's default pdf.fonttype = 3 and Computer Modern (cm) mathtext emit Type 3 fonts.
    • Always configure TUFTE_RC with "pdf.fonttype": 42, "ps.fonttype": 42, "svg.fonttype": "none", "mathtext.fontset": "stix", "text.usetex": False, and "font.serif": ["Times New Roman", "Liberation Serif", "DejaVu Serif", "Palatino"].
    • When svg.fonttype = "none" and mathtext.fontset = "stix" are used, scripts/ensure_inkscape_and_check_fonts.py registers Matplotlib's bundled STIXGeneral*.ttf and STIXNonUnicode*.ttf into ~/.local/share/fonts/matplotlib so inkscape/rsvg-convert embeds 100% TrueType (Type 42 / CID TrueType) outlines with zero Type 3 fonts.
  3. Keep Existing TikZ Diagrams Untouched as .tikz.tex / .pdf (Do Not Convert TikZ to .svg):
    • Empirical verification shows that converting standalone TikZ PDFs to SVG via pdftocairo -svg converts all TeX Type 1C fonts into raw <symbol><path/></symbol> outlines without <text> nodes or ToUnicode tables, stripping font metadata (0 fonts in pdffonts) and destroying PDF text searchability/selectability (pdftotext returns empty).
    • Therefore, keep existing standalone/inline TikZ diagrams untouched as .tikz.tex / .pdf for LaTeX builds, and use .svg as the primary format for all programmatic Python/Matplotlib/vector figures.
  4. Every Figure Panel Must State Its Claim in the 2-Line Header:
    • Line 1 (Bold Serif): A. Direct Empirical Claim or Mechanism (e.g., B. Bayesian Information Criterion (BIC) Parsimony).
    • Line 2 (Italic Muted #555555): Metric, units, sample size (N subjects / n trials), and statistical test (e.g., Lower ΔBIC = better parsimony (penalizes k · ln n)).
  5. Three Canonical Figure Archetypes (Flexible — Use When Aligned with the Paper's Main Points):
    • Archetype 1 — TikZ / Vector Method & Pathway Overview (fig1_cambr_architecture_pathways.png, bga_fig1_architecture_overview.png):
      • Prefer standalone TikZ (.tikz.tex $\to$ .pdf, keeping existing TikZ diagrams untouched as .tikz.tex/.pdf) or clean programmatic vector schematics saved as .svg + .pdf + .png with inline mathematical symbols ($\mathcal{L}_{\text{NLL}}$, $\boldsymbol{\theta}$, $\boldsymbol{\beta}$) and explicit visual separation between the Primary Best Pathway and Ablation Pathways. Verify zero text clipping or arrow overlap at 300 DPI before embedding.
    • Archetype 2 — Camera-Distortion-Corrected Qualitative Biomechanical Overlays + Time-Locked Waveforms (tremor_multiview_finger_overlay_keyframes.png, hop_test_triple_hop_skeleton_image_overlay_keyframes.png, fig_two_person_cse_mamma_contact.png, fig_fascia_flow_architecture_and_samples.png):
      • Project 3D meshes and skeletons using full radial/tangential camera distortion via multi_camera JAX projection (and/or undistorted video images).
      • Render MHR skin as a semi-transparent pearl envelope (alpha = 0.28–0.32) over rigid .stl bones colored by our Bilateral Bone Hue Standard: Left (_l) = Steel-Blue #7FA8D4, Right (_r) = Warm Coral-Red #D98880, Axial/Midline = Warm Ivory #E8E2D5, with CT-calibrated skull alignment (skull_offset_from_mhr).
      • Pair camera overlays with high-magnification insets (e.g., $2.6\times$ hand/finger or ribcage zoom) and synchronized bottom/side waveform panels (ensuring y-limits have $\ge 10%$ headroom so peaks never crop).
    • Archetype 3 — High-Density Tufte Quantitative & Occam's Razor Complexity Figures (fig_ssrp_occams_razor_ablation.png, fig_real_stroke_vs_nonstroke_ssrp.png, fig1_clinical_gaitlab_emg_grf.png):
      • Show individual subject trajectories or paired spaghetti lines (N subjects) alongside group medians/IQRs to prove zero pseudoreplication.
      • For model ablations, plot the Occam's Razor Complexity Ladder (in-sample vs. held-out CV performance + BIC/parsimony curve across model degrees of freedom $k$) with a direct callout box highlighting the simplest model that captures the effect.

Commands 1

Use the rae:bead-driven-development skill to orchestrate planning and execution with persistent beads tracking.

This command is for multi-task development workflows where you want:

  • Persistent task tracking across sessions (via beads)
  • Automatic skill orchestration (writing-plans, executing-plans, investigation)
  • Two-stage code review integration
  • Unified workspace organization in scratch/{date}-plan-{topic}/

Prerequisites: beads CLI installed (bd), beads plugin active, superpowers plugin active.

Sponsors

Details

Marketplace
peabody124/reproducible_agent_environment
Category
Community