Reproducible Agent Environment: 29 skills for coding standards enforcement, quality gates, repo scaffolding, repo auditing, golden/parity test harnesses, semi-formal code reasoning, ~80% ASD-STE100 technical specifications, GitHub activity review, JAX memory/retracing diagnosis, JAX JIT/fusion hygiene, efficient rendering, ISR MyST scientific paper writing & pre-2026 voice alignment, and domain-specific development (biomechanics, DataJoint, gait analysis, camera models)
- version
- 1.11.0
- GitHub stars
- 5
Install
/plugin install rae@reproducible_agent_environmentRun it inside Claude Code after adding the marketplace with /plugin marketplace add peabody124/reproducible_agent_environment.
Contents
Discovered from sourceSkills 27
Overview
Compare the current repository's configuration against the canonical RAE template
(templates/pyproject.toml) and report discrepancies. For major structural differences
(e.g., no src/ layout in an established repo), ask the user before proposing changes.
This skill does NOT automatically fix everything. It reports what it finds, fixes trivial gaps (missing ruff subsections, missing dev deps), and asks about anything that would be disruptive to change.
Steps
1. Read the Canonical Template
Read templates/pyproject.toml from the RAE plugin directory to get the current
source of truth. This template defines the expected configuration for:
- Build system (hatchling)
- Python version constraint (>=3.11)
- Dev dependencies (pytest, pytest-cov, ruff)
- Ruff config (line-length, lint rules, isort, format)
- Pytest config (testpaths, pythonpath, addopts, coverage)
- Coverage config (fail_under = 80)
2. Read the Current Repo
Read the current repository's:
pyproject.toml(required — if missing, this is a scaffold-repo situation, not an audit)CLAUDE.md(optional but recommended)- Directory structure (src/ vs flat layout)
3. Compare and Categorize
Compare each section and categorize discrepancies:
Auto-fixable (apply without asking):
- Missing
[tool.ruff.lint.isort]section - Missing
[tool.ruff.format]section - Missing
[tool.ruff]srckey - Missing ruff lint rules that are in the template
- Missing
[tool.pytest.ini_options]section - Missing
[tool.coverage.run]or[tool.coverage.report]sections
Ask the user first:
- No
src/layout (established repos may have good reasons) - Dev dependencies missing from
[project.optional-dependencies]when they're installed some other way (e.g., devcontainer, system-level) line-lengthdiffers from 120- Extra
ignorerules not in the template (may be project-specific, e.g., F722 for jaxtyping) - Missing CLAUDE.md (offer to create one)
- Build system differs from hatchling
Informational only (report but don't change):
- No
.github/CI workflows - No
.pre-commit-config.yaml(RAE does not require this) - Coverage threshold differs from 80 (may be intentionally lower for legacy repos)
4. Report Findings
Present a summary table:
## Audit Results: {repo_name}
### Aligned with RAE
- [x] ruff line-length = 120
- [x] Python >= 3.11
- ...
### Gaps Found
| Gap | Category | Action |
|-----|----------|--------|
| Missing [tool.ruff.format] | Auto-fix | Will add |
| Missing pytest config | Auto-fix | Will add |
| No src/ layout | Ask user | Uses body_models/ directly |
| Extra ignore F722 | Informational | Needed for jaxtyping |
### Recommended additions to CLAUDE.md
- Document F722 exception and why it's needed
- Ensure the **Fail-Fast Field Access & Anti-Defensive-Programming Rule** (no `getattr(obj, "field", None)` or `d.get("field", None)` on known schemas; cite the `getattr(person, "mhr", None)` silent 20-DOF finger-fit bug) is documented in `CLAUDE.md` / `AGENTS.md`
- ...
5. Apply Fixes
- Apply all auto-fixable changes to
pyproject.toml - For each "ask user" item, present the discrepancy and ask whether to fix or document as intentional
- Intentional deviations should be noted in the repo's CLAUDE.md under a
## RAE Deviationssection so future audits understand why
6. Verify
After applying fixes:
ruff format --check .
ruff check .
Report any issues from the verification step.
Important Notes
- The canonical source of truth is always
templates/pyproject.tomlin the RAE plugin - This skill does NOT enforce
src/layout changes on established repos — that's too disruptive - Extra
ignorerules are often legitimate (F722 for jaxtyping, etc.) — ask, don't remove - The goal is convergence over time, not instant compliance
Bead-Driven Development
Overview
This skill orchestrates existing skills with beads integration. It provides prompt refinements rather than reimplementing functionality:
- writing-plans → create plan + beads for major tasks
- executing-plans → use subagents + track via beads
- investigation → create blocking beads on failure, update plan
Announce at start: "I'm using the bead-driven-development skill to orchestrate planning and execution with persistent tracking."
Prerequisites
This skill requires beads and superpowers plugins. Install them first:
Install beads
# Install beads CLI
curl -fsSL https://raw.githubusercontent.com/steveyegge/beads/main/scripts/install.sh | bash
# Install uv (Python package manager)
curl -LsSf https://astral.sh/uv/install.sh | sh
# Add beads marketplace and install plugin
/plugin marketplace add steveyegge/beads
/plugin install beads
Install superpowers
/plugin marketplace add obra/superpowers-marketplace
/plugin install superpowers@superpowers-marketplace
Initialize beads in your repo
bd init
Required skills from superpowers:
- writing-plans
- executing-plans
- investigation (or from ~/.claude/skills/)
Unified Workspace Convention
All phases share one workspace: scratch/{YYYY-MM-DD}-plan-{topic}/
scratch/2026-01-18-plan-auth-system/
├── README.md ← Plan (Phase 1)
├── scripts/ ← Temp scripts (Phase 2)
└── debug/ ← Investigations (Phase 3)
└── bd-xxx/
Beads provide the commit history. Plan doesn't need to be committed separately.
Phase 1: Planning with Beads
Invoke writing-plans skill with these additions:
"Create plan in unified workspace following investigation pattern:
scratch/{YYYY-MM-DD}-plan-{topic}/README.mdExample:scratch/2026-01-18-plan-auth-system/README.mdAfter finalizing the plan, create a bead for each major task:
bd create 'Task 1: Component name' -t task bd create 'Task 2: Next component' -t task bd dep add <task-2-id> <task-1-id> --type blocksInclude bead IDs in the plan file for tracking.
Important: Add a section in the plan noting:
- Temp scripts go in:
scratch/{date}-plan-{topic}/scripts/- Debug investigations go in:
scratch/{date}-plan-{topic}/debug/- Beads provide commit history (plan doesn't need to be committed)
Note: TodoWrite will still be used for fine-grained in-session tracking."
Phase 2: Execution with Beads
Invoke executing-plans skill with these additions:
"Before starting each task:
- Run
bd readyto find next unblocked task- Mark bead in_progress:
bd update <id> --status in_progressUse TodoWrite as normal for fine-grained step tracking within the task.
Workspace: Put any temporary scripts, test outputs, or debugging artifacts in the plan's scratch directory:
scratch/{date}-plan-{topic}/scripts/After each successful task:
- Commit with bead ID:
git commit -m 'feat: description (bd-xxx)'- Add commit SHA to bead:
bd comment <id> 'Commit: <sha>'- Dispatch code-reviewer subagent (per subagent-driven-development two-stage review)
- If review passes:
bd close <id> --reason 'Implemented and reviewed'- Sync TodoWrite → bead status when marking task complete
Run
bd syncevery 2-3 completed tasks to persist to git.On failure → transition to Phase 3 (Failure Recovery)."
Phase 3: Failure Recovery with Beads
When any task fails, invoke investigation skill with these additions:
Step 1: Create blocking debug bead
bd create 'Debug: <issue description>' -t bug bd dep add <failed-task-id> <debug-bead-id> --type blocksStep 2: Investigation in plan's scratch directory Create investigation subfolder:
scratch/{date}-plan-{topic}/debug/{bead-id}/README.mdOr if separate investigation needed:
scratch/{date}-debug-{bead-id}/README.mdDocument findings, hypotheses, and tests in the investigation folder.
Step 3: Resolution Copy key findings summary to debug bead:
bd comment <debug-id> 'Resolution: <summary of fix>'If plan needs updates: edit
scratch/{date}-plan-{topic}/README.mdand note changes.If new tasks discovered:
bd create 'New task from investigation' -t task --discovered-from <debug-id>Step 4: Resume Close debug bead:
bd close <debug-id> --reason 'Resolved: <summary>'Original task becomes unblocked. Signal: "Plan updated, resume from Task N with
bd ready"
Hybrid Tracking Model
| Tool | Purpose | Scope |
|---|---|---|
| Beads | Major milestones/tasks | Cross-session persistence |
| TodoWrite | Fine-grained steps | In-session tracking |
Sync rule: When TodoWrite marks task complete → close corresponding bead.
Quick Reference
| Command | Purpose |
|---|---|
bd ready | Find next unblocked task |
bd update <id> --status in_progress | Mark task started |
bd close <id> --reason '...' | Mark task complete |
bd comment <id> 'message' | Add context to bead |
bd dep add <id> <blocker> --type blocks | Add dependency |
bd sync | Persist to git (every 2-3 tasks) |
bd show <id> | Get task context |
Workflow Summary
User: "execute this plan with beads"
↓
[bead-driven-development loads]
↓
Phase 1: Invoke writing-plans + beads prompt
→ Creates unified workspace
→ Plan in README.md
→ Creates beads for major tasks
→ Adds blocks dependencies
↓
Phase 2: Invoke executing-plans + beads prompt
→ Uses bd ready to pick next task
→ Updates bead status + TodoWrite
→ Commits with bead ID: (bd-xxx)
→ Two-stage code review
→ Close bead after review
→ bd sync every 2-3 tasks
↓
(On failure) Phase 3: Invoke investigation + beads prompt
→ Creates blocking debug bead
→ Investigates in workspace: debug/{bead-id}/
→ Copies findings to bead notes
→ Updates README.md when resolved
→ Resumes with bd ready
See Also
references/prompt-templates.md- Exact copy-paste prompts for each skillreferences/install-beads.md- Installation instructions when beads is not installed
Camera Model Reference
Overview
Our camera system is implemented in JAX and follows OpenCV conventions. There are two layers:
- Core library (
multi_camera.analysis.camera) - Low-level functions operating in millimeters - Camera adapter (
multiview_biomechanical_pose.geometry.camera_adapter) - High-level functions accepting/returning meters
Source files:
MultiCameraTracking/multi_camera/analysis/camera.py- Core camera functionsMultiviewBiomechanicalPose/multiview_biomechanical_pose/geometry/camera_adapter.py- Meter-safe wrappers
Camera Parameters Dict
All camera functions expect a single dict with parameters for ALL cameras:
camera_params = {
"mtx": array, # (num_cameras, 4) - [fx, fy, cx, cy] stored at 1/1000 scale
"rvec": array, # (num_cameras, 3) - Rodrigues rotation vectors (dimensionless)
"tvec": array, # (num_cameras, 3) - Translation vectors in METERS (stored at 1/1000 scale)
"dist": array, # (num_cameras, 5+) - Distortion coefficients [k1, k2, p1, p2, k3, ...]
}
The mtx and tvec fields are stored at 1/1000 scale (effectively meters). The library functions get_intrinsic() and get_extrinsic() multiply by 1000 internally to produce pixel-scale intrinsics and mm-scale extrinsics.
TFRecord format uses plural names (camera_intrinsics, camera_rvecs, camera_tvecs, camera_distortion) which the dataset pipeline converts to singular names automatically.
Unit Conventions (CRITICAL)
| Context | Units | Notes |
|---|---|---|
| 3D points throughout the system | meters | World coords, camera coords, joint positions |
camera_params["tvec"] storage | meters (1/1000 scale) | Scaled up by 1000 inside get_extrinsic() |
camera_params["mtx"] storage | 1/1000 of pixels | Scaled up by 1000 inside get_intrinsic() |
get_extrinsic() returns | 4x4 matrix in mm | Translation component is in millimeters |
get_intrinsic() returns | 3x3 K matrix in pixels | fx, fy, cx, cy in pixel units |
project_distortion() input | 3D points in mm | You MUST multiply meters by 1000 |
project_distortion() output | 2D points in pixels | |
triangulate_point() input | 2D points in pixels | With optional confidence channel |
triangulate_point() output | 3D points in mm | Divide by 1000 to get meters |
camera_adapter functions | meters in/out | Handle mm conversion internally |
| 2D keypoints | pixels | Original image resolution |
The Conversion Rule
When calling core library functions directly, always convert:
# Meters -> mm before projection
points_2d = cam_lib.project_distortion(camera_params, cam_idx, points_3d_m * 1000.0)
# mm -> meters after triangulation
points_3d_m = triangulate_point(camera_params, points_2d) / 1000.0
When using camera_adapter functions, no conversion is needed - they handle it internally.
OpenCV Extrinsic Convention
Our camera model follows the OpenCV world-to-camera convention:
P_camera = E @ P_world
get_extrinsic()returns a 4x4 SE(3) matrix that transforms points FROM world coordinates TO camera coordinates- This is NOT the camera position in world space
- The camera position in world space would be
-R^T @ t(orE_inv[:3, 3]) - Rotation is stored as Rodrigues vectors, converted via
SO3.exp(rvec) - Uses
jaxlieSE3/SO3 for all rigid body transformations
# get_extrinsic returns world-to-camera transform (units: mm)
E = get_extrinsic(camera_params, i) # 4x4 matrix
# To get camera position in world (in mm):
camera_pos_mm = jnp.linalg.inv(E)[:3, 3]
# To get camera position in meters:
camera_pos_m = camera_pos_mm / 1000.0
COLMAP Conversion Warning
COLMAP stores camera-to-world transforms. When converting from COLMAP:
R_w2c = R_c2w.T
t_w2c = -R_w2c @ t_colmap
Core Library Functions (multi_camera.analysis.camera)
All core functions operate in millimeters. Import as:
from multi_camera.analysis import camera as cam_lib
from multi_camera.analysis.camera import get_intrinsic, get_extrinsic
get_intrinsic(camera_params, i) -> 3x3 K matrix
Returns the intrinsic matrix with focal lengths and principal point in pixel units:
K = get_intrinsic(camera_params, 0)
# [[fx, 0, cx],
# [ 0, fy, cy],
# [ 0, 0, 1]]
get_extrinsic(camera_params, i) -> 4x4 SE(3) matrix
Returns the world-to-camera transformation matrix in millimeters:
E = get_extrinsic(camera_params, 0)
# 4x4 homogeneous transform [R|t; 0 0 0 1]
# t is in MILLIMETERS
# This is NOT the camera position - it's a world-to-camera transform
get_projection(camera_params, i) -> 3x4 projection matrix
Returns K @ E[:3] - the full 3x4 projection matrix:
P = get_projection(camera_params, 0) # 3x4
project_distortion(camera_params, i, points) -> 2D pixels
Projects 3D points to 2D image coordinates with radial distortion:
# CRITICAL: points must be in MILLIMETERS
points_2d = cam_lib.project_distortion(camera_params, cam_idx, points_3d_m * 1000.0)
Currently uses simplified radial model (k1 only). Handles behind-camera points by shrinking them toward image center.
undistort_points(points, K, dist, num_iters=5) -> undistorted 2D
Iteratively removes lens distortion from 2D image points. Supports full OpenCV model (k1-k6, p1-p2, s1-s4, tau_x, tau_y):
undistorted = cam_lib.undistort_points(
points_2d[None, ...], # (1, N, 2) - needs batch dim
K[None, ...], # (1, 3, 3)
dist_coeffs[None, :], # (1, n_coeffs)
)[0] # remove batch dim
distort_3d(camera_params, i, points) -> distorted 3D in camera coords
Computes equivalent 3D camera-frame points that would project to the same 2D coordinates under an ideal pinhole camera. Used for rendering 3D models (e.g., SMPL) on distorted images:
# points in mm, returns camera-frame coords in mm
vertices_distorted = cam_lib.distort_3d(camera_params, i, vertices_mm)
Uses full radial+tangential model (k1, k2, k3, p1, p2).
triangulate_point(camera_params, points2d, return_confidence=False) -> 3D mm
DLT triangulation from multiple 2D observations. Handles undistortion internally:
# points2d: (num_cameras, T, J, 3) with [u, v, confidence]
# Returns: (T, J, 3) or (T, J, 4) with confidence, in MILLIMETERS
points_3d_mm = cam_lib.triangulate_point(camera_params, points2d, return_confidence=True)
points_3d_m = points_3d_mm[..., :3] / 1000.0
robust_triangulate_points(camera_params, points2d, sigma=150, threshold=0.5) -> 3D mm
Robust triangulation using geometric median of pairwise triangulations (Roy et al. 2022). Automatically weights cameras by agreement:
points_3d, camera_weights = cam_lib.robust_triangulate_points(
camera_params, points2d, return_weights=True
)
# points_3d: (T, J, 4) in mm
# camera_weights: (num_cameras, T, J)
reprojection_error(camera_params, points2d, points3d) -> residual in pixels
Computes difference between observed and projected 2D points:
# points3d must be in mm (same space as projection)
residual = cam_lib.reprojection_error(camera_params, points2d, points3d_mm)
Camera Adapter Functions (Meter-Safe API)
These functions accept and return meters. Prefer these over direct core library calls when possible.
from multiview_biomechanical_pose.geometry.camera_adapter import (
transform_points_world_to_camera,
transform_points_camera_to_world,
transform_points_between_cameras,
project_camera_coordinates,
unproject_from_image_with_distortion,
unproject_and_transform_to_world,
transform_pose_se3_world_to_camera,
transform_pose_se3_camera_to_world,
compute_relative_extrinsics,
)
Point Transformations
# World -> Camera (meters in, meters out)
pts_cam = transform_points_world_to_camera(pts_world_m, camera_params, cam_idx)
# Camera -> World (meters in, meters out)
pts_world = transform_points_camera_to_world(pts_cam_m, camera_params, cam_idx)
# Camera A -> Camera B (meters in, meters out)
pts_target = transform_points_between_cameras(pts_source_m, camera_params, source_idx, target_idx)
Projection from Camera Coordinates
# Projects points already in camera frame to 2D (meters in, pixels out)
pts_2d = project_camera_coordinates(pts_cam_m, camera_params, cam_idx)
Uses identity extrinsics internally to avoid double-transformation.
Unprojection
# 2D pixels + depth -> 3D camera coords (pixels+meters in, meters out)
pts_cam = unproject_from_image_with_distortion(pts_2d, depth_m, camera_params, cam_idx)
# 2D pixels + depth -> 3D world coords (pixels+meters in, meters out)
pt_world = unproject_and_transform_to_world(image_point, depth_m, camera_params, cam_idx)
SE(3) Pose Transformations
# Transform full pose (position + quaternion) between frames
pos_cam, quat_cam = transform_pose_se3_world_to_camera(pos_world_m, quat_wxyz, camera_params, cam_idx)
pos_world, quat_world = transform_pose_se3_camera_to_world(pos_cam_m, quat_wxyz, camera_params, cam_idx)
Quaternions use wxyz format (jaxlie convention).
Common Workflows
Project 3D Skeleton to Image
from multi_camera.analysis import camera as cam_lib
# joints_3d: (T, J, 3) in meters
# Project to camera i
joints_2d = cam_lib.project_distortion(camera_params, i, joints_3d * 1000.0)
# joints_2d: (T, J, 2) in pixels
Triangulate from Multi-View 2D Keypoints
# keypoints_2d: (num_cameras, T, J, 3) with [u, v, confidence]
points_3d = cam_lib.triangulate_point(camera_params, keypoints_2d, return_confidence=True)
# points_3d: (T, J, 4) in mm - convert to meters:
points_3d_m = points_3d[..., :3] / 1000.0
confidence = points_3d[..., 3]
Render SMPL Mesh on Distorted Image
from multi_camera.analysis.camera import get_intrinsic, distort_3d
# vertices: (N, 3) in meters
K = np.array(get_intrinsic(camera_params, i))
vertices_distorted_mm = np.array(distort_3d(camera_params, i, vertices * 1000.0))
vertices_distorted_m = vertices_distorted_mm / 1000.0
# Render with ideal pinhole (K, identity R/T) since distortion is baked into 3D
Compute Reprojection Error
from multi_camera.analysis.fit_quality import reprojection_quality
# keypoints3d: (T, J, 3) in mm
# keypoints2d: (N_cams, T, J, 3) with [u, v, confidence]
metrics = reprojection_quality(keypoints3d_mm, camera_params, keypoints2d)
Convert Between Aniposelib and Our Format
# Aniposelib -> our format (mm -> meter storage)
from aniposelib.cameras import CameraGroup
cgroup = CameraGroup.from_names(cam_names)
cam_dicts = cgroup.get_dicts()
camera_params = {
"mtx": np.array([[c["matrix"][0][0], c["matrix"][1][1],
c["matrix"][0][2], c["matrix"][1][2]]
for c in cam_dicts]) / 1000.0,
"dist": np.array([c["distortions"] for c in cam_dicts]),
"rvec": np.array([c["rotation"] for c in cam_dicts]),
"tvec": np.array([c["translation"] for c in cam_dicts]) / 1000.0,
}
# Our format -> Aniposelib (meter storage -> mm)
from aniposelib.cameras import Camera, CameraGroup
cameras = []
for i, name in enumerate(cam_names):
K = np.array([[params["mtx"][i,0]*1000, 0, params["mtx"][i,2]*1000],
[0, params["mtx"][i,1]*1000, params["mtx"][i,3]*1000],
[0, 0, 1]])
cam = Camera(name=name, matrix=K, dist=params["dist"][i],
rvec=params["rvec"][i], tvec=params["tvec"][i]*1000)
cameras.append(cam)
cgroup = CameraGroup(cameras)
Distortion Model
Our cameras use the OpenCV radial-tangential distortion model:
- Projection (
project_distortion): Simplified model using k1 only (tangential terms commented out) - Unprojection (
undistort_points): Full model with k1-k6, p1-p2, s1-s4, tau - 3D distortion (
distort_3d): Full model with k1, k2, k3, p1, p2
Distortion coefficients are stored in OpenCV order: [k1, k2, p1, p2, k3, ...]
For EgoHumans data: images are pre-undistorted and distortion is set to [0, 0, 0, 0, 0].
Anti-Patterns (NEVER DO THESE)
1. Never manually access camera parameter arrays
# WRONG - bypasses scaling and formatting
fx = camera_params["mtx"][0, 0]
R = cv2.Rodrigues(camera_params["rvec"][0])[0]
t = camera_params["tvec"][0]
# CORRECT - use library functions
K = get_intrinsic(camera_params, 0) # Properly scaled 3x3
E = get_extrinsic(camera_params, 0) # Properly scaled 4x4 SE(3)
2. Never create single-camera parameter dicts
# WRONG - breaks the API contract
cam0_params = {
"mtx": camera_params["mtx"][0:1],
"rvec": camera_params["rvec"][0:1],
"tvec": camera_params["tvec"][0:1],
"dist": camera_params["dist"][0:1],
}
# CORRECT - pass full dict with camera index
result = cam_lib.project_distortion(camera_params, 0, points)
3. Never forget the mm conversion for core library functions
# WRONG - passing meters to mm-expecting function
projected = cam_lib.project_distortion(camera_params, i, points_3d_m)
# CORRECT
projected = cam_lib.project_distortion(camera_params, i, points_3d_m * 1000.0)
4. Never reimplement geometric transforms
# WRONG - manual projection ignoring distortion
x_2d = fx * (x_3d / z_3d) + cx
y_2d = fy * (y_3d / z_3d) + cy
# CORRECT - use library with distortion
from multiview_biomechanical_pose.geometry.camera_adapter import project_camera_coordinates
pts_2d = project_camera_coordinates(pts_cam_m, camera_params, cam_idx)
5. Never manually unproject without distortion correction
# WRONG - ignores lens distortion
x_3d = (x_2d - cx) * depth / fx
# CORRECT
from multiview_biomechanical_pose.geometry.camera_adapter import unproject_from_image_with_distortion
pts_3d = unproject_from_image_with_distortion(pts_2d, depth_m, camera_params, cam_idx)
6. Never confuse extrinsic matrix with camera position
# WRONG - treating tvec as camera position in world
camera_position = camera_params["tvec"][i]
# CORRECT - extrinsic is world-to-camera, invert to get camera position
E = get_extrinsic(camera_params, i)
camera_pos_mm = jnp.linalg.inv(E)[:3, 3]
camera_pos_m = camera_pos_mm / 1000.0
Quick Reference: Import Paths
# Core camera library
from multi_camera.analysis import camera as cam_lib
from multi_camera.analysis.camera import (
get_intrinsic, # camera_params, i -> 3x3 K (pixels)
get_extrinsic, # camera_params, i -> 4x4 E (mm, world-to-camera)
get_projection, # camera_params, i -> 3x4 P (mm)
project_distortion, # camera_params, i, pts_mm -> 2D pixels
undistort_points, # pts_px, K, dist -> undistorted_px
distort_3d, # camera_params, i, pts_mm -> distorted 3D mm
triangulate_point, # camera_params, pts_2d -> 3D mm
robust_triangulate_points, # camera_params, pts_2d -> 3D mm + weights
reprojection_error, # camera_params, pts_2d, pts_3d_mm -> residual px
)
# High-level meter-safe adapter
from multiview_biomechanical_pose.geometry.camera_adapter import (
transform_points_world_to_camera, # pts_m -> pts_m (camera frame)
transform_points_camera_to_world, # pts_m -> pts_m (world frame)
transform_points_between_cameras, # pts_m -> pts_m (target camera)
project_camera_coordinates, # pts_cam_m -> 2D pixels
unproject_from_image_with_distortion, # 2D px + depth_m -> 3D cam_m
unproject_and_transform_to_world, # 2D px + depth_m -> 3D world_m
transform_pose_se3_world_to_camera, # pos_m, quat -> pos_m, quat
transform_pose_se3_camera_to_world, # pos_m, quat -> pos_m, quat
compute_relative_extrinsics, # -> 4x4 relative transform (mm)
)
# Fit quality metrics
from multi_camera.analysis.fit_quality import reprojection_quality
Common Mistakes
| Mistake | Fix |
|---|---|
Passing meters to project_distortion | Multiply by 1000 first |
Treating get_extrinsic() translation as camera position | It's world-to-camera; invert for camera position |
Accessing camera_params["mtx"] raw values | Use get_intrinsic() which applies 1000x scaling |
| Creating per-camera param dicts | Pass full dict + camera index |
| Ignoring distortion in projection/unprojection | Always use distortion-aware functions |
Assuming triangulate_point returns meters | It returns mm - divide by 1000 |
Forgetting batch dims for undistort_points | Needs (batch, N, 2) input shape |
| Using COLMAP extrinsics directly | COLMAP is camera-to-world; ours is world-to-camera |
Overview
When you discover a better pattern, configuration, or guideline during work, this skill guides you through proposing it upstream to the Reproducible Agent Environment repository.
Parameters
- improvement_type (optional): "guideline", "skill", "config", or "template"
- description (required): Brief description of the improvement
Steps
1. Evaluate Scope
Determine if the improvement is universal or project-specific.
Constraints:
- You MUST verify the improvement works in the current project
- You MUST check if a similar pattern already exists upstream
- You SHOULD consider if the improvement applies broadly or only to specific domains
Decision:
- IF universal → Proceed to step 2
- IF project-specific → Add to local overrides with comment explaining why, then STOP
2. Document the Improvement
Prepare a clear description of the change.
Constraints:
- You MUST include before/after examples
- You MUST explain the rationale (why is this better?)
- You SHOULD reference any research or sources that informed the improvement
- You MUST specify which file(s) would be modified
3. Create Upstream PR
Constraints:
- You MUST create a branch with descriptive name (e.g.,
improve/ruff-config,guideline/async-patterns) - You MUST write a clear commit message explaining the improvement
- You SHOULD include test cases or examples if applicable
- You MUST open PR with the documented rationale
4. Update Local
After PR is merged:
- Update the RAE plugin to pull the improvement
- Verify it works correctly in your project
See references/propose-upstream.md for the detailed upstream contribution workflow.
Examples
User: "I found a better ruff configuration for catching common issues" Agent: Evaluates if it's universal (yes), documents the change with before/after examples, creates PR to update templates/pyproject.toml.
User: "/config-improvement guideline 'Add guidance on async/await patterns'" Agent: Checks if async guidance exists (no), drafts new section for python-standards.md, creates PR with examples.
Overview
Read and internalize the coding guidelines relevant to the current task. Guidelines are
bundled as reference files in the enforce-guidelines skill at
enforce-guidelines/references/.
Parameters
- task_type (optional): "python", "git", "refactor", "debug", "review", or "all"
Steps
1. Determine Relevant Guidelines
Based on task type, identify which guidelines to review from enforce-guidelines/references/:
| Task Type | Guidelines to Read |
|---|---|
python | coding-standards.md, python-standards.md |
git | git-workflow.md, pre-commit-checklist.md |
refactor | coding-standards.md, anti-patterns.md |
debug | coding-standards.md (TDD section) |
review | anti-patterns.md, coding-standards.md |
new-repo | repo-structure.md, python-standards.md |
all | All guidelines |
2. Summarize Key Points
Extract the 3-5 most relevant rules for the current task.
- You MUST highlight any MUST/MUST NOT constraints
- You SHOULD note any task-specific gotchas
3. Apply During Work
Keep these guidelines in mind while executing the task. Reference them when making decisions.
- You MUST cite the relevant guideline when it influences a decision
- You SHOULD flag if you discover a case not covered by guidelines
Examples
User: "I'm about to refactor the auth module" Agent: Reads coding-standards.md and anti-patterns.md, summarizes DRY principles and slop patterns to avoid, proceeds with refactor while citing guidelines.
User: "/consult-guidelines python" Agent: Reviews coding-standards.md and python-standards.md, extracts key rules about ruff, type hints, and path management.
DataJoint Biomechanics Schema Reference
CRITICAL: NEVER Modify Database Entries
DO NOT update, delete, or alter any DataJoint database entries. This includes:
update1(),delete(),drop()on any table- Modifying settings lookup tables (KinematicReconstructionSettingsLookup, ProbabilisticReconstructionSettingsLookup, KineticReconstructionSettingsLookup, KeypointSet, etc.)
- Altering any computed table entries
Database entries are shared state used by the entire lab. Changing a settings entry changes it for everyone and invalidates prior results computed with those settings.
Overview
The biomechanics pipeline uses DataJoint across four repositories with two complementary data collection approaches:
- Multi-Camera (MMC) - Lab-based with calibrated camera rigs →
MultiCameraTracking+BodyModels/kinematic_dj.py - Portable/Monocular (PBL) - Smartphone videos with optional IMU →
PortableBiomechanicsSessions+BodyModels/monocular_dj.py
Both produce the same output format: qpos, qvel, joints, sites from MuJoCo body models.
DataJoint Query Basics
Operators
| Operator | Meaning | Example |
|---|---|---|
& | Restrict (filter) | (Table & {'field': value}) |
* | Join tables | (Table1 * Table2) |
- | Set difference | (Table1 - Table2) finds rows NOT in Table2 |
.proj() | Select attributes | Table.proj('field1', 'field2') |
Fetch Patterns
# fetch1() - Exactly one row, returns tuple or dict
timestamps, qpos = (KinematicReconstruction.Trial & key).fetch1('timestamps', 'qpos')
# fetch() - Multiple rows, returns arrays
all_keys = (Session & filter).fetch('KEY')
keypoints = (TopDownPerson & key).fetch('keypoints')
# With ordering (critical for multi-camera consistency)
kp, names = (Table & key).fetch('keypoints', 'camera_name', order_by='camera_name')
Filtering by Project and Participant
video_project Field
video_project is a categorical field (varchar(50)) that identifies the study/cohort. It lives in MultiCameraRecording and Video tables.
Common projects:
| Project | Description |
|---|---|
GAIT_CONTROLS | Healthy control gait data |
PROSTHETIC_GAIT | Prosthetic gait subjects |
CM_GAIT | Cervical myelopathy gait |
CLINIC_GAIT | Clinical gait assessment |
PEDIATRIC_GAIT | Pediatric subjects |
ASB2024 | ASB conference dataset |
Query Patterns by Project
# List all unique projects
projects = np.unique((MultiCameraRecording).fetch("video_project"))
# Find all participants in a project
participants = (Session & (Recording & (MultiCameraRecording & 'video_project="GAIT_CONTROLS"'))).fetch("participant_id")
# Filter trials by project
trials = KinematicReconstruction.Trial & (Recording & (MultiCameraRecording & f"video_project='{project}'"))
# Multiple projects with IN clause
filt = 'video_project in ("CLINIC_GAIT", "GAIT_CONTROLS", "PROSTHETIC_GAIT")'
sessions = Session & (Recording * MultiCameraRecording & filt)
Participant Naming Conventions
| Project Type | Format | Examples |
|---|---|---|
| ASB studies | ASB_### | ASB_001, ASB_022 |
| Numeric IDs | integers | 72, 504, 127 |
| Special codes | mapped | TF01, TF02, TF47 |
Session → Video Path (via video_project)
Session (participant_id, session_date)
→ Recording → MultiCameraRecording (video_project)
→ SingleCameraVideo → Video (video_project, filename)
Key Output Fields Reference
| Field | Table | Shape | Description |
|---|---|---|---|
| qpos | KinematicReconstruction.Trial | (T, 41) | Joint angles in generalized coordinates (radians) |
| qvel | KinematicReconstruction.Trial | (T, 41) | Joint velocities (rad/s) |
| joints | KinematicReconstruction.Trial | (T, N_joints, 3) | 3D joint positions (mm) |
| sites | KinematicReconstruction.Trial | (T, N_sites, 3) | 3D anatomical markers (mm) |
| keypoints | TopDownPerson | (T, N_kpts, 3) | 2D keypoints [x, y, conf] |
| keypoints3d | PersonKeypointReconstruction | (T, N_kpts, 4) | 3D keypoints [x, y, z, conf] (mm) |
| keypoints_3d | LiftingPerson | (T, N_kpts, 4) | Lifted 3D from monocular [x, y, z, conf] |
| timestamps | VideoInfo | list[datetime] | Absolute frame times |
| delta_time | VideoInfo | (T,) | Seconds from first frame |
| fps | VideoInfo | float | Frames per second |
Common Access Patterns
# Get kinematic reconstruction (ALWAYS specify method!)
qpos, sites = (KinematicReconstruction.Trial & key &
{'kinematic_reconstruction_settings_num': 137}).fetch1('qpos', 'sites')
# Get 3D triangulated keypoints
kp3d = (PersonKeypointReconstruction & key).fetch1('keypoints3d')
For 2D keypoints synchronized with KR qpos, see the Temporal Synchronization section below. Do NOT use (TopDownPerson * VideoInfo & key).fetch1('timestamps', 'keypoints') — this produces misaligned frames.
Temporal Synchronization (CRITICAL)
KR qpos and 2D keypoints have independently managed timestamps. Naive matching silently corrupts frame alignment.
Correct approach: Use fetch_keypoints(only_detected=True) from BodyModels:
from body_models.datajoint.dataset import fetch_keypoints as bm_fetch_keypoints
# Returns keypoints in 1:1 frame correspondence with KR qpos
timestamps, keypoints_raw = bm_fetch_keypoints(trial_key, only_detected=True)
# keypoints_raw: (C, T, 87, 3) — (x, y, confidence)
# ALWAYS verify correspondence
assert qpos.shape[0] == keypoints_raw.shape[1]
Why direct TopDownPerson/VideoInfo fetch is broken: KR timestamps are zeroed relative to first detection; VideoInfo timestamps are zeroed relative to video start. When first detection occurs at frame N>0, matching on timestamp magnitude pairs the wrong frames (N-frame offset).
Camera parameter reordering: KinematicReconstruction.updated_calibration uses Calibration table order; keypoints use alphabetical SingleCameraVideo order. These differ — reorder explicitly before projection.
See rae:fetching-synchronized-data for the complete pattern with camera reordering and contiguous segment selection.
Two Data Collection Approaches
1. Multi-Camera (MMC) → KinematicReconstruction
Key space: (participant_id, session_date) bridged to (recording_timestamps, camera_config_hash)
Session → Recording → MultiCameraRecording → SingleCameraVideo → Video
↘ SessionCalibration.Grouping → Calibration
↓
Video → TopDownPerson → PersonKeypointReconstruction (3D triangulation)
↓
KinematicReconstruction.Trial (qpos, qvel, joints, sites)
Bridge Tables:
Recording- Links Session → MultiCameraRecordingSingleCameraVideo- Links MultiCameraRecording → VideoSessionCalibration.Grouping- Links Session → Calibration (required for KinematicReconstruction)
Use when: Lab-based data, multiple synchronized cameras, highest accuracy needed.
2. Portable/Monocular (PBL) → MonocularReconstruction
Key space: (subject_id, session_start_time) via Firebase
Subject → Session → Session.FirebaseSessionInfo
↓
FirebaseSession (Computed)
↓
┌─────────────────┼─────────────────┐
AppVideo Attitude/Gyro PhoneAttitude
(→ Video) (IMU sensors) (phone quaternion)
↓
TopDownPerson → LiftingPerson (3D lifting)
↓
MonocularReconstruction.Trial (qpos, qvel, joints, sites, rnc)
Additional output: rnc (T, 3) - camera rotation vector from phone attitude
Use when: Smartphone videos, remote/home monitoring, wearable IMU/EMG data.
Comparison
| Aspect | Multi-Camera | Monocular |
|---|---|---|
| Input | Multiple synced cameras | Single smartphone camera |
| 3D Method | Triangulation | 2D→3D lifting |
| Accuracy | ~10-15mm, ~2-5° | ~15-20mm, ~5-10° |
| qpos DOF | 41 | 40 |
| Extra data | updated_calibration | rnc (phone rotation), IMU/EMG |
| Schema | kinematic_dj.py | monocular_dj.py |
Method Numbers by Pipeline
CRITICAL: Always specify the method number in queries to avoid duplicates.
| Pipeline | Table | Method Field | Default Method |
|---|---|---|---|
| Multi-Camera | KinematicReconstruction | kinematic_reconstruction_settings_num | 137 |
| Monocular | MonocularReconstruction | monocular_reconstruction_settings_num | 1 |
Multi-Camera: Method 137
# CORRECT - Always specify method for KinematicReconstruction
key = {'participant_id': '102', 'session_date': date(2023, 7, 21),
'kinematic_reconstruction_settings_num': 137} # CRITICAL!
# WRONG - Returns BOTH 130 and 137 for dual-processed participants
key = {'participant_id': '102', 'session_date': date(2023, 7, 21)}
Dual-processed participants (have both 130 and 137): ASB_022, ASB_007, ASB_008, ASB_020, ASB_041, ASB_071, ASB_081, ASB_083, ASB_085, ASB_095, ASB_125, ASB_130
Method 137 config: humanoid_torque_rl.xml, floor leveling enabled, 40k iterations
Monocular: Method 1
# For MonocularReconstruction queries
key = {'subject_id': 123, 'session_start_time': ...,
'monocular_reconstruction_settings_num': 1}
Bridging Algorithm
Bottom-up MeTRAbs detection inserted into top-down tables:
MeTRAbs (580 keypoints) → BottomUpBridging → BottomUpBridgingPerson
↓ filter_skeleton("bml_movi_87") → indices 264-350
TopDownPerson (method="Bridging_bml_movi_87")
↓ re-fetch from BottomUpBridging (not TopDownPerson!)
LiftingPerson (method="Bridging_bml_movi_87")
Why "weird": LiftingPerson normally uses TopDownPerson, but for bridging it re-fetches directly from BottomUpBridging.
Quick Reference: Key Files
| Task | File |
|---|---|
| Video/VideoInfo/TopDownPerson | PosePipeline/pose_pipeline/pipeline.py |
| Bridging algorithm | PosePipeline/pose_pipeline/wrappers/bridging.py |
| MultiCameraRecording/SingleCameraVideo | MultiCameraTracking/multi_camera/datajoint/multi_camera_dj.py |
| Session/Recording (MMC) | MultiCameraTracking/multi_camera/datajoint/sessions.py |
| SessionCalibration | MultiCameraTracking/multi_camera/datajoint/session_calibrations.py |
| PersonKeypointReconstruction | MultiCameraTracking/multi_camera/datajoint/multi_camera_dj.py |
| KinematicReconstruction | BodyModels/body_models/datajoint/kinematic_dj.py |
| MonocularReconstruction | BodyModels/body_models/datajoint/monocular_dj.py |
| FirebaseSession/AppVideo (PBL) | PortableBiomechanicsSessions/portable_biomechanics_sessions/emgimu_session.py |
| PBL↔MMC linkage | PortableBiomechanicsSessions/portable_biomechanics_sessions/mmc_linkage.py |
| DataJoint principles | PipelineOrchestrator/docs/datajoint_principles.md |
Common Mistakes
- Missing method number - Queries without
kinematic_reconstruction_settings_num=137return duplicates - Wrong key space - Using
(video_project, filename)when you need(participant_id, session_date) - Assuming LiftingPerson uses TopDownPerson - For bridging methods, it re-fetches from BottomUpBridging
- Missing SessionCalibration.Grouping - KinematicReconstruction requires this bridge
- Forgetting order_by - Multi-camera queries need
order_by='camera_name'for consistent ordering - Confusing MonocularReconstruction vs KinematicReconstruction - Monocular uses FirebaseSession.AppVideo, Kinematic uses SessionCalibration.Recordings
- Not filtering by video_project - Queries across all projects mix different populations; always filter to your target cohort
- Matching KR timestamps with VideoInfo timestamps - Use
fetch_keypoints(only_detected=True)instead; see Temporal Synchronization section - Assuming camera_params order matches keypoint order - Calibration order != SingleCameraVideo alphabetical order; reorder explicitly
- Zeroing timestamps manually (
ts - ts[0]) - Destroys physical time correspondence between data sources
Review the diff against main and remove all AI-generated slop introduced in the outstanding code changes, most recent commit, or branch based on user instructions.
Refer to the anti-patterns guide in enforce-guidelines/references/anti-patterns.md
for the full list of slop patterns to catch.
Common slop to remove:
- Extra comments that a human wouldn't add or that are inconsistent with the rest of the file
- Defensive
getattr(obj, "field", None)ord.get("field", None)on known/expected schemas: Replace with direct fail-fast field/key access (obj.fieldord["field"]). Never allowgetattr(..., None)to silently evaluate a misspelled attribute toNone(as in thegetattr(person, "mhr", None)bug that silently disabled the 20-DOF finger fit on every frame). - Extra defensive checks or try/catch blocks that are abnormal for that area of the codebase (especially if called by trusted / validated codepaths)
- Casts to
Anyto get around type issues - Placeholder patterns (TODOs without context, "Phase 1" comments, commented-out code)
- Over-engineering (abstractions for single-use code, unnecessary factories)
- Any other style that is inconsistent with the file
Report at the end with only a 1-3 sentence summary of what you changed.
Going further: prevent over-engineering at write-time
deslop is a corrective pass — it scrubs slop out of changes already made. To stop
over-engineering before it is written, the third-party ponytail skills
(https://github.com/DietrichGebert/ponytail, MIT) are complementary:
ponytail— a write-time mode that enforces minimal/YAGNI code (stdlib and native features before custom code, no unrequested abstractions).ponytail-review/ponytail-audit— report-only over-engineering passes on a diff or the whole repo. Narrower than deslop (complexity/LOC only, no type-evasion or DB-mutation checks) and they list findings rather than applying them.
To adopt: either copy the pertinent skills/<name>/SKILL.md folders from that repo
into your skills directory (no hooks needed — the skills work standalone), or
install the full plugin from the repo for the activation hooks and /ponytail
mode switching as well.
Efficient Rendering
Performance and correctness patterns for rendering meshes, point clouds, and overlays — offline batches and differentiable in-the-loop rendering alike. The two recurring failure modes are (1) silently falling back to a CPU software GL rasterizer (10–50× slower) and (2) renders that don't line up with the real camera because a convention was dropped. Most of this skill is about avoiding those two.
This skill is about how to render fast and correctly. For the camera math
itself (projection, triangulation, unit conventions), see the camera-model
skill — this one assumes you already have intrinsics K, an extrinsic
world→camera transform, and distortion coefficients.
The golden rules
- Pin a real GPU for EGL before importing any GL library, and verify you
didn't land on
llvmpipe. This is the single biggest win. - Create the renderer/context once per resolution and reuse it across every frame at that size. Context creation is ~60 ms (Mesa/EGL path); per-frame work should be single-digit ms.
- Pick one distortion strategy and apply it consistently — either undistort the image and render with a pinhole, or keep the distorted image and bake distortion into the 3D vertices. Never half of each.
- Keep one coordinate convention end to end (OpenCV: +X right, +Y down, +Z forward). Convert to GL exactly once, at the camera node.
- Composite only the dirty region (mesh bounding box), not the whole frame.
- Precompute everything pose-independent once (skinning weights, normals topology, projection matrices, undistort remaps, camera frusta/bounds).
- For JAX rasterizers, chunk pixels and
jax.checkpointso peak memory ischunk × primitives, notH·W × primitives.
1. Headless GPU rendering with EGL
Offscreen rendering (no X server) goes through EGL. The trap: if the NVIDIA EGL
vendor library isn't selected, GL silently falls back to Mesa's llvmpipe CPU
rasterizer and everything still "works" — just 10–50× slower.
Set the platform before importing OpenGL/pyrender/anything that pulls in GL:
import os
os.environ.setdefault("PYOPENGL_PLATFORM", "egl") # MUST precede any GL import
# only now:
import pyrender
MuJoCo uses its own backend switch. When rendering MuJoCo geometry (e.g.
mujoco.Renderer), MuJoCo reads MUJOCO_GL, not PYOPENGL_PLATFORM. Set
MUJOCO_GL=egl (the unset/glfw default needs a display and dies headless;
osmesa is the CPU path). Set both before any import so a process that mixes a
MuJoCo renderer and a pyrender overlay routes both through hardware EGL:
os.environ.setdefault("MUJOCO_GL", "egl")
os.environ.setdefault("PYOPENGL_PLATFORM", "egl")
Setting the env var alone does not pin a GPU — see the next paragraph. The
llvmpipe verification below is just as necessary for the MuJoCo path.
Pin a GPU — but do not assume the EGL device index equals the CUDA index.
EGL device order does not match nvidia-smi / CUDA_VISIBLE_DEVICES order,
and there is no reliable EGL→card mapping. So don't set EGL_DEVICE_ID to a
CUDA index and trust it — that's the classic way to land on the wrong card or on
llvmpipe. Instead enumerate EGL devices, build a throwaway context on each,
read GL_RENDERER, and pick a hardware GPU (rank by free memory from
nvidia-smi if you want the emptiest one):
from OpenGL import EGL
devices = EGL.eglQueryDevicesEXT(...) # enumerate
for d in devices:
# create a context on `d`, read GL_RENDERER, keep the first non-llvmpipe one
...
os.environ["EGL_DEVICE_ID"] = str(chosen_egl_index) # only after probing/verifying
# If GLVND can't find the NVIDIA ICD, point it at the vendor JSON:
# os.environ["__EGL_VENDOR_LIBRARY_FILENAMES"] = "/path/to/10_nvidia.json"
If you must honor a pre-set EGL_DEVICE_ID, still validate it resolves to real
hardware (the GL_RENDERER assert below) rather than trusting the number.
NVIDIA Devcontainer Setup (nvidia/cuda:*-ubuntu24.04 gotchas):
NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics:nvidia/cudabase images default tocompute,utility. Unless"-e", "NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics"is indevcontainer.jsonrunArgs(andcontainerEnv),nvidia-container-clionly mounts CUDA compute libraries and omitslibEGL_nvidia.so.0,libGLX_nvidia.so.0, andlibnvidia-rtcore.so.- Missing
10_nvidia.jsonandnvidia_icd.json:nvidia/cudaUbuntu 24.04 images do not include GLVND or Vulkan JSON manifests, andapt-get install libgl1 libegl1 libvulkan1only registers Mesa (50_mesa.jsonandlvp_icd.json). Ensure/usr/share/glvnd/egl_vendor.d/10_nvidia.json("library_path": "libEGL_nvidia.so.0") and/etc/vulkan/icd.d/nvidia_icd.json("library_path": "libGLX_nvidia.so.0") exist (created intemplates/devcontainer-gpu/devcontainer.jsononCreateCommand), or set__EGL_VENDOR_LIBRARY_FILENAMES/VK_ICD_FILENAMESat runtime. - Manual
--device=/dev/nvidia*filtering and Vulkan/RTX: Ifdevcontainer.jsonrunArgsmanually lists--device=/dev/nvidia0,--device=/dev/nvidiactl,--device=/dev/nvidia-uvm(e.g. to skip unhealthy GPUs), you must also include--device=/dev/nvidia-modesetand--device=/dev/dri. Without them, CUDA and EGL pbuffers work, but Vulkan (vkCreateDevice, Isaac Simomni.rtx, SAPIEN, ManiSkill) segfaults or fails withNo device could be created.
Always verify you got the GPU, not the CPU fallback. Do this once at startup and fail loudly:
r = pyrender.OffscreenRenderer(W, H)
import OpenGL.GL as gl
renderer_str = gl.glGetString(gl.GL_RENDERER).decode()
assert "llvmpipe" not in renderer_str.lower(), f"CPU GL fallback: {renderer_str}"
Typical impact at ~2K resolution / ~18k-vertex mesh: llvmpipe ≈ 800 ms/frame,
NVIDIA EGL ≈ 15 ms/frame.
One process, one context. EGL contexts cannot be shared across processes.
If you parallelize across processes, each worker needs its own context (and
must not have CUDA/JAX forked into it — use multiprocessing.get_context("spawn"),
never fork, once CUDA is initialized). For most workloads a single process
with a reused context beats multiprocessing.
2. Reuse the GL context and scene
Allocate the OffscreenRenderer (plus Scene, camera node, lights) once per
viewport size and cache it. Re-pose the geometry each frame; never reallocate
the renderer.
_RENDER_CACHE: dict[tuple[int, int], dict] = {}
def get_context(h: int, w: int) -> dict:
key = (int(h), int(w))
if key not in _RENDER_CACHE:
renderer = pyrender.OffscreenRenderer(viewport_width=w, viewport_height=h)
scene = pyrender.Scene(bg_color=[0, 0, 0, 0]) # transparent for compositing
_RENDER_CACHE[key] = {"renderer": renderer, "scene": scene}
return _RENDER_CACHE[key]
import atexit
atexit.register(lambda: [c["renderer"].delete() for c in _RENDER_CACHE.values()])
The cached Scene, camera, and lights persist; only the mesh churns. Add the
posed mesh, render, then remove its node in a finally so the next frame starts
clean without rebuilding the scene:
node = ctx["scene"].add(mesh)
try:
color, depth = ctx["renderer"].render(ctx["scene"], flags=pyrender.RenderFlags.RGBA)
finally:
ctx["scene"].remove_node(node)
Keep video captures open too — a VideoCapture held across frames reads
sequentially in O(1); reopening or seeking per frame is a large hidden cost.
3. Camera handling for rendering
Getting the render to land on the right pixels is mostly about three conversions: OpenCV→GL axis flip, intrinsics→projection, and a single decision about distortion.
3a. OpenCV → GL axis convention
OpenCV cameras look down +Z with +Y down; OpenGL cameras look down
−Z with +Y up. The camera pose (camera→world) for a GL renderer is the
inverse of your world→camera extrinsic, post-multiplied by a flip:
GL_FLIP = np.diag([1.0, -1.0, -1.0, 1.0]) # OpenCV -> OpenGL
cam_pose = np.linalg.inv(extrinsic_w2c) @ GL_FLIP # what pyrender wants
Apply this flip in exactly one place. If your overlay is upside-down or mirrored, this is almost always the culprit — don't compensate elsewhere.
3b. Intrinsics → projection
Use an intrinsics camera directly rather than hand-rolling a projection matrix; it handles the principal-point offset correctly:
camera = pyrender.IntrinsicsCamera(fx=fx, fy=fy, cx=cx, cy=cy,
znear=0.05, zfar=100.0)
Set znear/zfar to bracket your actual scene depth — too wide wastes z-buffer
precision (z-fighting), too tight clips geometry. For a self-contained object,
deriving the frustum from the vertex bounds once per sequence (not per frame)
both fixes clipping and avoids recomputation.
Watch your units against the frustum. A zfar that's fine in meters will
clip the whole subject to a blank frame if the mesh is in centimeters — keep
znear/zfar/depth_tolerance in the same unit as your geometry.
For a canonical "look at the object" view (no real camera), an orthographic
camera sized to the vertex extent is robust: set xmag = ymag = extent, stand
the camera off at center.z + extent·3, and zfar = extent·10 so the standoff
plus body depth always fits. A fixed small zfar here is the usual cause of a
blank render.
When projecting points yourself, use the OpenCV pinhole with a safe-z guard and pixel centers (the 0.5 offset matters for alignment with sampled images):
z = jnp.maximum(verts_cam[:, 2], 1e-3)
u = fx * verts_cam[:, 0] / z + cx
v = fy * verts_cam[:, 1] / z + cy
# pixel grid for sampling: xs + 0.5, ys + 0.5
3c. Distortion — pick ONE of two strategies
Real lenses have radial/tangential distortion. There are two correct ways to make a render line up with a distorted photo; mixing them produces a subtle, maddening misalignment near the image edges.
Strategy A — undistort the image (preferred when you control the footage).
Undistort the video frame once, then render and project with a plain pinhole.
Use the original K as the new camera matrix so the projection model is
unchanged (do not use getOptimalNewCameraMatrix, which shifts the image
plane and breaks alignment with your K):
map1, map2 = cv2.initUndistortRectifyMap(K, dist, np.eye(3), K, (w, h), cv2.CV_32FC1)
# precompute once; reuse for every frame:
frame_undist = cv2.remap(frame, map1, map2, cv2.INTER_LINEAR)
# now render + project with plain pinhole K, no distortion anywhere
Likewise undistort 2D keypoints before unprojecting/triangulating. This keeps all 3D math pinhole-simple and is the right call for differentiable rendering.
Strategy B — bake distortion into the vertices (when you must keep the original image). Transform the 3D vertices in camera space into the "pre-distorted" positions that, under an ideal pinhole, land where the real lens would put them. Then render with a plain pinhole onto the untouched distorted image:
verts_distorted_cam = distort_3d(camera_params, cam_idx, verts_cam_mm) # see camera-model
# render with pinhole K, identity extrinsic, over the original distorted frame
Use B when the deliverable must overlay the raw footage (e.g. you can't ship undistorted video) or distortion is strong. Use A everywhere else — it's simpler and differentiable. Whichever you choose, apply it to both the image and the geometry, never one without the other.
4. Mesh rendering
- Skip geometry validation when you trust your topology:
trimesh.Trimesh(v, f, process=False)avoids a surprisingly expensive cleanup pass per frame. - Batch all meshes into one scene before a single
render()so the shared z-buffer resolves inter-mesh occlusion correctly in one pass — don't render bodies separately and composite, or near/far occlusion will be wrong. - Materials:
MetallicRoughnessMaterial(metallicFactor=0, roughnessFactor≈0.7)with modest ambient (~0.35) and a camera-mounted directional light reads well for anatomy/body meshes. RenderRGBAso you get an alpha mask for free. - Per-vertex color / heatmaps: render with the
FLATflag to skip lighting so the raw vertex colors survive (error maps, region labels, confidence). - Translucent over opaque: for a skin shell over bones, render bones
OPAQUEand skinBLENDwithSKIP_CULL_FACESso the back of the semi-transparent shell isn't culled. - Backface culling for a single closed opaque shell: drop faces whose normal
points away from the camera (
n · view_dir ≥ 0) — fewer triangles, identical result. - Layered translucency, exactly: rasterize each layer opaquely (its own z-buffer fixes self-occlusion) and composite back-to-front in numpy (painter's order). More predictable than GL depth-sorted blending.
- Precompute skinning once. Nearest-vertex skin weights/indices depend only on the template geometry, not the pose — compute once, reuse every frame. Skinning is often the dominant per-frame cost otherwise.
5. Point clouds and points "inside" a mesh
Rendering markers/points that may be occluded by (or embedded within) a body
mesh needs real depth reasoning, not just cv2.circle at projected locations.
Cheap occlusion via a coarse z-buffer. Project all points, bucket them into
a coarse pixel grid, take the per-bucket nearest depth with segment_min, and
flag a point visible only if it's within a depth tolerance (~3 cm) of its
bucket's front. Avoids a full per-pixel depth compare and is fully vectorized:
bu = jnp.clip(jnp.floor(u * scale).astype(jnp.int32), 0, Wb - 1)
bv = jnp.clip(jnp.floor(v * scale).astype(jnp.int32), 0, Hb - 1)
bucket = bv * Wb + bu
bucket_min = jax.ops.segment_min(jnp.where(candidate, z, 1e6), bucket, num_segments=Hb*Wb)
visible = z <= bucket_min[bucket] + depth_tolerance
Differentiable point/soft rendering (SoftRas-style). Per pixel, take a
depth-aware softmax over nearby points/vertices:
w ∝ exp(-‖pix − proj‖² / 2σ²) · exp(−z / depth_temp), normalized with
logsumexp for stability. The alpha channel is the max spatial weight. This is
what lets a renderer pass gradients back to 3D positions.
Keep peak memory bounded with chunking + remat. The naive weight matrix is
(H·W, V) — multiple GiB. Tile pixels into chunks and recompute each chunk's
weights in the backward pass instead of storing them:
@jax.checkpoint
def render_chunk(pix_chunk): # (C, 2)
d2 = jnp.sum((pix_chunk[:, None] - proj[None])**2, -1) # (C, V)
logit = jnp.where(vis[None], -d2/(2*sigma**2) - z[None]/depth_temp, -1e30)
w = jnp.exp(logit - jax.scipy.special.logsumexp(logit, -1, keepdims=True))
return w @ vertex_attr # (C, attr)
img = jax.lax.map(render_chunk, pix.reshape(n_chunks, C, 2))
256² × 18k verts drops from ~4.5 GiB to a few hundred MiB. The @jax.checkpoint
is load-bearing: without it the enclosing scan/map stores every chunk's
intermediate weight matrix on the autodiff tape, which defeats the chunking.
Pick the chunk size to divide H·W evenly and mark H, W, chunk as static JIT
args (one compile per resolution).
Hard triangle rasterization, differentiable where it counts. When you need
crisp coverage: compute barycentrics for every face per pixel, z-buffer to pick
the nearest covering face (argmin over depth). Make the face id and coverage
stop_gradient (a hard mask shouldn't leak gradient), but let the barycentric
coordinates and depth flow gradients to the vertex positions. Same
chunk-and-checkpoint trick for memory.
Vertex normals, vectorized: accumulate (area-weighted) face normals onto
vertices with segment_sum over the flattened face index — no Python loop:
fn = jnp.cross(v1 - v0, v2 - v0) # (F,3), |fn| = 2·area
vn = jax.ops.segment_sum(jnp.repeat(fn, 3, 0), faces.reshape(-1), num_segments=V)
vn = vn / jnp.maximum(jnp.linalg.norm(vn, axis=-1, keepdims=True), 1e-8)
6. 2D overlays: keypoints, skeletons, trails
-
Project once in camera frame, rescale to the output resolution, then draw. Vectorize the projection across frames/cameras with
jax.vmaprather than a Python loop. -
Encode visibility in the marker, not just position: filled circle + white halo for visible, a dim hollow ring for occluded/low-confidence. Skip NaN / out-of-bounds points.
-
Comet-tail trails via accumulated alpha blends — width and alpha ramp from tail to head:
for seg in range(t0 + 1, t + 1): frac = (seg - t0) / max(t - t0, 1) cv2.line(overlay, p0, p1, color, thickness=max(1, int(1 + (w_max-1)*frac))) cv2.addWeighted(overlay, 0.15 + 0.85*frac, img, 1 - (0.15 + 0.85*frac), 0, img) -
Crisp small markers: upscale the frame 2–4×, draw markers at scaled coordinates, downscale the final video. Cheaper and sharper than antialiasing every primitive.
-
Smooth playback: interpolate tracks onto a denser time grid and repeat the nearest source frame, so markers glide while video steps.
7. Video overlay pipeline
-
Composite only the bounding box of the render's alpha mask. For sparse overlays this is most of the speedup:
ys, xs = np.where(mask) y0, y1, x0, x1 = ys.min(), ys.max()+1, xs.min(), xs.max()+1 a = (mask[y0:y1, x0:x1, None] * alpha) out[y0:y1, x0:x1] = (rgb[y0:y1, x0:x1] * a + frame[y0:y1, x0:x1] * (1 - a))Integer (uint16) intermediates with a
// 255divide avoid a full float32 round-trip of the whole frame. -
Prefer piping raw frames to
ffmpegovercv2.VideoWriterfor H.264: more reliable codecs, faster, and you control pixel format. Useyuv420p+-movflags +faststartfor browser-playable output.proc = subprocess.Popen([ffmpeg, "-f","rawvideo","-pix_fmt","bgr24", "-s",f"{w}x{h}","-r",str(fps),"-i","-","-an","-c:v","libx264", "-crf","18","-pix_fmt","yuv420p","-movflags","+faststart", out_path], stdin=subprocess.PIPE) # -an: no audio, -crf 18: visually lossless for f in frames_bgr: proc.stdin.write(np.ascontiguousarray(f).tobytes())If you must use
cv2.VideoWriter(mp4v), a best-effortffmpegre-encode to H.264 afterward makes it streamable.
8. JAX-specific performance
-
One JIT boundary for FK + mesh extraction. Wrap forward kinematics and world-space vertex extraction in a single
jax.jitso it all runs on device; crossing the host boundary between them costs ~100 ms of dispatch. -
Vectorize vertex extraction — gather per-vertex transforms and
einsuminstead of a Python loop. Vectorizing alone (numpy) takes a body mesh from ~275 ms to ~2 ms; the sameeinsumjitted on-device is sub-millisecond:world = xpos[v2g] + jnp.einsum("vij,vj->vi", xmat[v2g].reshape(-1,3,3), local_vert) -
vmapfor multi-instance, padding to a fixed count so shapes stay stable and you compile once. Carry a validity mask rather than slicing. -
Cache compiled functions by instance id when the static model object can't be a JIT argument, and
stop_gradientstatic models inside the jitted fn. -
Donate buffers in the training/render step (
eqx.filter_jit(donate=...)) to free large inputs immediately. -
Static-arg the resolution/chunk size; accept one compile per resolution.
See the jax-memory-and-retracing skill for diagnosing OOMs and unexpected recompiles.
Pre-flight checklist
-
PYOPENGL_PLATFORM=egl(andMUJOCO_GL=eglif rendering MuJoCo) set before any GL import. - Verified
GL_RENDERERis the NVIDIA GPU, notllvmpipe. - Renderer/context allocated once and reused; cleaned up at exit.
- Exactly one OpenCV→GL flip, at the camera node.
-
znear/zfarbracket the real scene depth (no z-fighting, no clipping). - One distortion strategy, applied to both image and geometry.
- Pose-independent work (skinning, normals topology, remaps, frusta, projection matrices) precomputed once.
- Compositing restricted to the mask bounding box.
- JAX rasterizer chunks pixels +
jax.checkpoint; statics are resolution/chunk. - Output encoded
yuv420p+ faststart if it needs to play in a browser.
Anti-patterns
| Anti-pattern | Fix |
|---|---|
Importing GL before setting PYOPENGL_PLATFORM | Set the env var first, at the very top |
Relying on PYOPENGL_PLATFORM for a MuJoCo renderer | MuJoCo reads MUJOCO_GL; set MUJOCO_GL=egl too |
Trusting it "works" without checking GL_RENDERER | Assert no llvmpipe at startup |
New OffscreenRenderer per frame | Cache by viewport size, reuse |
| Distorting vertices and undistorting the image | Pick one strategy, apply consistently |
getOptimalNewCameraMatrix as the undistort target | Keep the original K as the new matrix |
| Compensating an upside-down render downstream | Fix the single OpenCV→GL flip instead |
Materializing (H·W, primitives) in a JAX rasterizer | Chunk pixels + jax.checkpoint |
| Recomputing skinning/normals/frusta every frame | Precompute once per sequence |
| Compositing the whole frame for a small overlay | Restrict to the mask bbox |
cv2.VideoWriter mp4v for a deliverable | Pipe raw frames to ffmpeg H.264/yuv420p |
fork for render workers after CUDA init | multiprocessing.get_context("spawn") |
The Iron Law
IF A GUIDELINE APPLIES TO YOUR TASK, YOU DO NOT HAVE A CHOICE. YOU MUST FOLLOW IT.
This is not a suggestion. Guidelines are requirements. Violations require correction before work is considered complete.
When This Skill Activates
This skill activates automatically when:
- Writing Python code →
python-standards.md,coding-standards.md - Creating a new repository →
repo-structure.md - Making commits →
git-workflow.md - Reviewing or refactoring code →
anti-patterns.md - Any code changes →
coding-standards.md
You MUST check for applicable guidelines before starting any task. Even a 1% chance a guideline applies means you consult it.
Decision Flow
Before responding to ANY code-related request:
1. What type of task is this?
└─→ Identify: python, git, new-repo, refactor, debug, review
2. Which guidelines apply?
└─→ Map task type to required guidelines (see mapping below)
3. Read the applicable guidelines
└─→ Actually read them, don't assume you know the content
4. Extract MUST/MUST NOT constraints
└─→ These are non-negotiable requirements
5. Proceed with task, citing guidelines when they influence decisions
Where to Find Guidelines
Guidelines are bundled as reference files alongside this skill:
references/coding-standards.mdreferences/python-standards.mdreferences/repo-structure.mdreferences/git-workflow.mdreferences/anti-patterns.mdreferences/pre-commit-checklist.md
Guideline Mapping
| Task Type | Required Guidelines |
|---|---|
| Any Python code | python-standards.md, coding-standards.md |
| New repository | repo-structure.md, python-standards.md |
| Git operations | git-workflow.md |
| Refactoring | anti-patterns.md, coding-standards.md |
| Code review | anti-patterns.md, python-standards.md |
| Bug fixes | coding-standards.md (TDD section) |
| New features | coding-standards.md, python-standards.md |
Red Flags (Rationalization Detection)
These thoughts indicate you're trying to skip guidelines:
| Thought | Reality |
|---|---|
| "This is just a quick fix" | Quick fixes still follow guidelines |
| "I already know the standards" | Read them anyway, they may have changed |
| "This is too simple to need guidelines" | Simple code still needs type hints and formatting |
| "I'll fix it later" | Fix it now, there is no later |
| "The user didn't ask for TDD" | TDD is the default, not optional |
| "100 chars is close enough to 120" | Line length is 120, configure ruff correctly |
Enforcement Mechanism
Before Starting Work
- Identify task type from the mapping above
- Read applicable guidelines - actually open and read them
- Extract key constraints - list the MUST/MUST NOT rules
- Acknowledge constraints - state which rules apply to this task
During Work
- Cite guidelines when they influence a decision
- Check compliance before completing each step
- Run verification commands (ruff, pytest) as guidelines require
Before Completing Work
- Run all required checks:
ruff format . && ruff check . && pytest - Verify coverage: pytest runs with
--covby default, coverage must be ≥80% - Verify against checklist from applicable guidelines
- Confirm no violations remain
Constraints
- You MUST read guidelines before starting code tasks
- You MUST cite the specific guideline when it influences a decision
- You MUST NOT skip guidelines because the task seems simple
- You MUST NOT proceed if guideline compliance is unclear
- You MUST run
ruff formatandruff checkbefore completing Python work - You MUST run
pytestif tests exist - You SHOULD flag if you discover a case not covered by guidelines
Verification Checklist
Before marking any code task complete:
- Identified applicable guidelines
- Read the guidelines (not just remembered them)
- Listed MUST constraints that apply
- Followed all MUST constraints
- Avoided all MUST NOT patterns
- Ran
ruff formaton changed files - Ran
ruff checkwith no errors - Ran
pytestwith coverage ≥80% - No anti-patterns from
anti-patterns.md
Cannot check all boxes? Work is not complete.
Examples
User: "Add a helper function to parse dates"
Agent (correct):
This is Python code, so I need to consult
python-standards.mdandcoding-standards.md.Key constraints:
- MUST use type hints for function signatures
- MUST run ruff format/check
- SHOULD write test first (TDD)
Let me write a failing test first, then implement...
Agent (incorrect):
Sure, here's a quick function:
def parse_date(s): return datetime.strptime(s, "%Y-%m-%d")
The incorrect response skips guidelines consultation, misses type hints, and doesn't run verification.
Excalidraw Diagram Generation
Generate Excalidraw diagrams, render them to images, and visually verify the output.
Setup
excalirender (Bun-based standalone binary, no Node.js required):
curl -fsSL https://raw.githubusercontent.com/JonRC/excalirender/main/install.sh | sh
excalirender --version
Alternative — Docker (no install):
docker run --rm -v "$(pwd):/data" -w /data jonarc06/excalirender diagram.excalidraw -o output.png
Fallback — npx (requires Node.js 18+):
npx -y @excalidraw/excalidraw-brute-export-cli diagram.excalidraw output.png
Workflow
Step 1: Generate .excalidraw JSON
Write a valid .excalidraw file to scratch/ (or user-specified path). Follow the JSON Format Reference below exactly.
Step 2: Render to Image
excalirender scratch/diagram.excalidraw -o scratch/diagram.png
Useful flags:
-s 2— 2x scale (sharper)--dark— dark mode--transparent— transparent background-o output.svg— SVG output (use file extension)
Step 3: Visually Verify
Read the output image with the Read tool to inspect the rendered diagram. Check:
- All elements are visible and properly positioned
- Arrows connect to the correct shapes
- Text is readable and not overlapping
- Layout is balanced and clear
- Embedded panels show actual pixels, not a blank box — a
fileIdmismatch renders a silent white rectangle (exit 0, correct size, no image). Inspect the panel, not just the file.
For a composite scientific figure, also run the design rubric and re-render until it passes:
- A naive viewer can name the main message in ~5 seconds
- There is ONE clear reading path (L→R linear / cycle / parallel-column compare)
- ≤3 accent colors, used semantically
- Box edges and baselines align to a grid with uniform gutters
- All text is readable at the target export size
- No chartjunk (3D, shadows, gradients, stray ticks/legends) that doesn't serve the message
Step 4: Iterate if Needed
Fix any issues in the .excalidraw JSON and re-render. Common fixes:
- Adjust x/y positions to fix overlapping elements
- Increase width/height of shapes containing long text
- Fix arrow
pointscoordinates for proper routing - Add missing
boundElementsentries on shapes that arrows connect to
Excalidraw JSON Format Reference
Top-Level Structure
{
"type": "excalidraw",
"version": 2,
"source": "https://excalidraw.com",
"elements": [],
"appState": {
"viewBackgroundColor": "#ffffff",
"gridSize": null
},
"files": {}
}
Common Element Properties
Every element shares these base properties:
{
"id": "rect1",
"type": "rectangle",
"x": 100,
"y": 200,
"width": 200,
"height": 80,
"angle": 0,
"strokeColor": "#1e1e1e",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 1,
"opacity": 100,
"seed": 1234567,
"version": 1,
"versionNonce": 987654,
"isDeleted": false,
"groupIds": [],
"frameId": null,
"boundElements": null,
"updated": 1700000000000,
"link": null,
"locked": false,
"roundness": { "type": 3 }
}
Property Reference:
| Property | Type | Values / Notes |
|---|---|---|
id | string | Unique identifier. Use short strings like "box1", "arr1" |
type | string | "rectangle", "ellipse", "diamond", "text", "arrow", "line", "freedraw", "image", "frame" |
x, y | number | Top-left corner position in canvas pixels |
width, height | number | Dimensions in pixels |
angle | number | Rotation in radians (0 = no rotation) |
strokeColor | string | Hex color for border/stroke |
backgroundColor | string | "transparent" or hex color for fill |
fillStyle | string | "solid", "hachure", "cross-hatch", "zigzag" |
strokeWidth | number | 1 (thin), 2 (normal), 4 (thick) |
strokeStyle | string | "solid", "dashed", "dotted" |
roughness | number | 0 (architect/clean), 1 (artist/default), 2 (cartoonist/rough) |
opacity | number | 0-100 |
seed | number | Random int for deterministic rough.js rendering |
version | number | Start at 1 |
versionNonce | number | Random int |
isDeleted | boolean | Always false for new elements |
groupIds | array | Array of group ID strings. Elements sharing a groupId are grouped |
frameId | string|null | ID of containing frame element |
boundElements | array|null | Array of {"id": "...", "type": "arrow"} or {"id": "...", "type": "text"} |
roundness | object|null | {"type": 3} for rounded corners, null for sharp |
locked | boolean | false for editable |
Element Types
Rectangle, Ellipse, Diamond
Use the common properties above. Set type to "rectangle", "ellipse", or "diamond".
To make a shape a container for text, add the text element's ID to boundElements:
{
"id": "box1",
"type": "rectangle",
"x": 100, "y": 100, "width": 200, "height": 80,
"boundElements": [{"id": "text1", "type": "text"}],
"...common properties..."
}
Text
Additional properties beyond the common set:
{
"id": "text1",
"type": "text",
"text": "Hello World",
"fontSize": 20,
"fontFamily": 1,
"textAlign": "center",
"verticalAlign": "middle",
"containerId": "box1",
"originalText": "Hello World",
"autoResize": true,
"lineHeight": 1.25
}
| Property | Values |
|---|---|
fontSize | Number in pixels. Common: 16, 20, 28, 36 |
fontFamily | 1 = Excalifont (hand-drawn), 2 = Nunito (sans-serif), 3 = Comic Shanns (code), 4 = Liberation Sans (system) |
textAlign | "left", "center", "right" |
verticalAlign | "top", "middle" |
containerId | ID of parent shape, or null for standalone text |
originalText | Same as text (used for undo) |
autoResize | true to auto-fit container |
lineHeight | 1.25 (default) |
Text-in-Container binding: The text element has containerId: "box1" AND the shape has boundElements: [{"id": "text1", "type": "text"}]. Both sides must reference each other. When text is inside a container, set the text's x, y, width, height to match the container's inner area (Excalidraw recomputes these, but providing reasonable values avoids rendering artifacts).
Arrow
Additional properties beyond the common set:
{
"id": "arr1",
"type": "arrow",
"x": 300, "y": 140,
"width": 200, "height": 0,
"points": [[0, 0], [200, 0]],
"startBinding": {
"elementId": "box1",
"fixedPoint": [1, 0.5]
},
"endBinding": {
"elementId": "box2",
"fixedPoint": [0, 0.5]
},
"startArrowhead": null,
"endArrowhead": "arrow",
"elbowed": false
}
| Property | Values |
|---|---|
points | Array of [dx, dy] relative to element's x,y. First point MUST be [0, 0] |
startBinding | null or {"elementId": "...", "fixedPoint": [fx, fy]} |
endBinding | Same as startBinding |
fixedPoint | [0-1, 0-1] normalized position on target shape. [0, 0.5] = left center, [1, 0.5] = right center, [0.5, 0] = top center, [0.5, 1] = bottom center |
startArrowhead | null, "arrow", "bar", "dot", "triangle", "diamond" |
endArrowhead | Same options as startArrowhead |
elbowed | false for curved/straight, true for right-angle routing |
Arrow-Shape binding: The arrow has startBinding.elementId: "box1" AND box1 has boundElements: [{"id": "arr1", "type": "arrow"}]. Both sides must reference each other.
Arrow positioning: Set the arrow's x, y to the start point. Set width and height to the bounding box of all points. The points array is relative to x, y, so the first point is always [0, 0].
Line
Same as arrow but with type: "line". No startBinding/endBinding/arrowheads.
{
"type": "line",
"points": [[0, 0], [100, 50], [200, 0]],
"...common properties..."
}
Frame
Groups elements visually with a labeled border:
{
"id": "frame1",
"type": "frame",
"name": "Backend Services",
"x": 50, "y": 50, "width": 600, "height": 400,
"...common properties..."
}
Child elements set "frameId": "frame1" to belong to the frame.
Image (embedded raster panel)
Embeds a rendered PNG/JPG/SVG (a matplotlib chart, a MuJoCo/mesh render, a photo) as a panel. An embedded image is two coupled parts, and the one load-bearing rule is:
element.fileId=== thefiles-map key === that entry's.id. The element holds no pixels; the bytes live in thefilesentry.
If they don't match (or the files entry is missing), excalirender draws a silent white
box — exit 0, correct size, no pixels. So Step 3 must inspect the panel, not just that
the file rendered.
Image element (extras beyond the common base):
| Property | Values |
|---|---|
fileId | Hex string; the key into files. Use sha1(bytes).hexdigest() (40 chars) |
status | "saved" for generated files |
scale | [1, 1] default; [-1, 1] flips horizontally |
crop | null, or {x, y, width, height, naturalWidth, naturalHeight} for a sub-region |
width/height | display box — set to the image's natural aspect ratio or it stretches (excalirender fills the box, it does not preserve aspect) |
Files entry (top-level files map, keyed by fileId):
| Property | Values |
|---|---|
id | Same as the key and the element's fileId |
dataURL | Full "data:image/png;base64,..." string including the prefix |
mimeType | "image/png", "image/jpeg", or "image/svg+xml" |
created | Epoch ms (e.g. 1700000000000) |
Hand-authoring megabytes of base64 is impractical — generate the pair with this stdlib
helper. Keep PNGs in scratch/ and reference them by path; never paste base64 into chat.
import base64, hashlib
def embed_image(path, x, y, width, height, mime="image/png"):
"""Return (image_element, files_patch) for a rendered panel. fileId is the
sha1 of the bytes, so re-embedding an identical panel dedupes."""
data = open(path, "rb").read()
file_id = hashlib.sha1(data).hexdigest()
data_url = f"data:{mime};base64," + base64.b64encode(data).decode()
element = {
"id": file_id[:8], "type": "image", "x": x, "y": y,
"width": width, "height": height, "angle": 0, "strokeColor": "#1e1e1e",
"backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2,
"strokeStyle": "solid", "roughness": 0, "opacity": 100, "seed": 42,
"version": 1, "versionNonce": 1, "isDeleted": False, "groupIds": [],
"frameId": None, "boundElements": None, "updated": 1700000000000,
"link": None, "locked": False, "fileId": file_id, "status": "saved",
"scale": [1, 1], "crop": None,
}
files_patch = {file_id: {"mimeType": mime, "id": file_id,
"dataURL": data_url, "created": 1700000000000}}
return element, files_patch
Downsample large renders to display resolution before embedding — base64 inflates ~33% and
lands inline in the JSON (and again in SVG export), so a few multi-MB panels make an
unreadable, undiffable file. Keep .excalidraw files with embedded panels out of git.
Color Palette
Use the Excalidraw default palette (Open Color shades):
Stroke colors (shade 7-9, saturated):
| Color | Hex |
|---|---|
| Black | #1e1e1e |
| Red | #e03131 |
| Pink | #c2255c |
| Grape | #9c36b5 |
| Violet | #6741d9 |
| Indigo | #3b5bdb |
| Blue | #1971c2 |
| Cyan | #0c8599 |
| Teal | #099268 |
| Green | #2f9e44 |
| Yellow | #f08c00 |
| Orange | #e8590c |
Background fills (shade 1-3, pastel):
| Color | Hex |
|---|---|
| Red | #ffc9c9 |
| Blue | #a5d8ff |
| Green | #b2f2bb |
| Yellow | #ffec99 |
| Violet | #d0bfff |
| Cyan | #99e9f2 |
| Orange | #ffd8a8 |
| Gray | #e9ecef |
Grouping
To group elements, assign the same group ID string to their groupIds array:
{"id": "a", "groupIds": ["group1"], "..."},
{"id": "b", "groupIds": ["group1"], "..."}
Nested groups: "groupIds": ["innerGroup", "outerGroup"] (innermost first).
Diagram Patterns
Architecture Diagram (Boxes + Arrows)
Layout: horizontal flow, 250px spacing between box centers.
[Client] --> [API Gateway] --> [Service] --> [Database]
x=0 x=300 x=600 x=900
Each box: width=200, height=80. Arrows start at right edge of source (x+200, y+40), end at left edge of target (x, y+40).
Flowchart (Decision Diamonds)
Layout: vertical flow, 150px spacing between element centers.
[Start]
|
<Decision?>
/ \
[Yes] [No]
Diamonds: width=160, height=100. Use fixedPoint: [0.5, 1] for bottom exit, [0, 0.5] and [1, 0.5] for side exits.
Layered Architecture
Layout: vertical stack with frames.
+-- Presentation Layer --+
| [Component] [Component] |
+------------------------+
+-- Business Logic ------+
| [Service] [Service] |
+------------------------+
+-- Data Layer ----------+
| [Repository] [Cache] |
+------------------------+
Use frame elements for each layer, 50px vertical gap between frames.
Composite scientific figures (rendered panels + annotation)
This section applies only to composite/overview figures. A plain architecture diagram or flowchart ignores all of it.
A scientific overview figure is rendered raster panels (matplotlib/plotly charts, MuJoCo/mesh/point-cloud renders, photos) under a vector annotation layer (arrows, panel labels, callouts, title). Excalidraw is the programmatic layout + annotation layer; it does not render the data — panels are rendered elsewhere and embedded. If every panel is a matplotlib chart, just use matplotlib subfigures/GridSpec instead. Reach for excalidraw the moment panels are heterogeneous (charts + 3D renders + hand annotation).
Workflow
- State the message. One sentence: the figure's single take-home, and the medium (paper column / slide / poster). Every element serves it.
- Render each panel separately at final 1:1 size. Charts: share one
rcParamsstyle block across all panels (sans-serif, fixed linewidth, the figure's hex color cycler),savefig(dpi=300, transparent=True, bbox_inches="tight", pad_inches=0). 3D/mesh/MuJoCo: hand off to/efficient-rendering(RGBA + transparent bg + matte material, so it composites with no grey box); overlay-on-video result panels use/camera-modelfor intrinsics/extrinsics/units. Keep PNGs inscratch/. - Embed each panel with
embed_image(...). Image elements go first inelements[](lowest z-order) so annotations sit on top. - Lay out + annotate (the value-add): one
frameper panel on a shared grid (snap x/y to a ~20px module, uniform gutters, equal-size siblings, shared baselines); bold lowercasea/b/clabels at a fixed corner offset (+8,+8); arrows along one reading direction (L→R / cycle / parallel-column compare); a title that asserts the finding. - Render + verify:
excalirender fig.excalidraw -o fig.png -s 2, Read the PNG, run the Step-3 rubric, iterate. - Export: prefer SVG/PDF (annotations stay vector, only panels are raster —
publication-safe). Full-figure PNG (
-s 2+) for drafts/slides.
Defaults & palette
- Publication look:
roughness: 0,fillStyle: "solid",fontFamily: 2(or 4) on all text, body labels ≥16px canvas. (roughness: 1/ Excalifont is for casual sketches only.) - DPI × export scale: embed at the pixel size you want at export scale s; set element
width ≈ natural/s; export with matching-s. Otherwise source is crisp but output blurs. - Color: ≤3 accent hues; gray (
#1e1e1e/#e9ecef) for everything de-emphasized; one hue per concept, reused everywhere that concept appears — feed the same hex list to the matplotlib cycler and to render colors. Never red+green to distinguish two classes. - Multiview: panels comparing views/methods must share the camera/unit convention from
/camera-model— don't mix mm and m across panels.
Companion skills (suggest, never auto-install)
Excalidraw does layout/annotation, not data rendering or palette selection. When a figure needs more, suggest a vetted external skill and let the user install it:
| Need | Skill (license) |
|---|---|
| Tufte-styled minimal-ink chart panels | tufte-data-viz (caylent, MIT, read-only) |
| Colorblind-safe palette / "what colors?" | color-expert (meodai, CC-BY-4.0) |
| Choose & justify a chart from raw data | SciPilot figure skill (MIT, local Python) |
| Crisp editorial schematic, no hand-drawn look | diagram-design (MIT) |
| Text-diffable infra/architecture diagram | Mermaid / D2 skills |
Vetting bar: permissive license (MIT/CC-BY), real adoption, prefer read-only/no-exec. Avoid
AntV mcp-server-chart for unpublished data — by default it POSTs your chart spec to a
hosted server.
Best Practices
- IDs: Use descriptive short strings (
"box_api","arr_to_db","text_title") - Spacing: 200-300px horizontal gap between connected nodes, 120-150px vertical gap
- Arrow points: First point MUST be
[0, 0]. A horizontal arrow of length 200:points: [[0, 0], [200, 0]] - Seeds: Set
seedto any random integer (e.g.,42,999,1577544) for consistent rough.js rendering - Bidirectional bindings: ALWAYS set both the arrow's
startBinding/endBindingAND the shape'sboundElementsarray. Missing either side breaks the connection - Text sizing: For text inside containers, set the text element's
width/heightsmaller than the container (leave ~20px padding each side) - Roughness 0: Use for clean technical diagrams. Use
1for the classic hand-drawn look - File extension: Always use
.excalidraw— this is a plain JSON file - Output location: Write diagrams to
scratch/by default, or user-specified path
Fetching Synchronized Keypoints + Reconstruction Data
Overview
KinematicReconstruction qpos and 2D keypoints come from different DataJoint tables with independently managed timestamps. Naive timestamp matching silently produces frame offsets that corrupt downstream tasks (guidance, evaluation, metrics). The correct approach uses fetch_keypoints(only_detected=True) from BodyModels, which guarantees 1:1 frame correspondence with KR qpos by construction.
When to Use
- Fetching 2D keypoints that must align frame-by-frame with KR qpos
- Building inference/guidance pipelines that pair observations with reconstruction
- Debugging high reprojection error or misaligned 2D-3D data
- Writing any code that joins TopDownPerson keypoints with KinematicReconstruction
When NOT to use: Fetching keypoints for standalone 2D analysis (no KR alignment needed), or querying KR qpos without 2D observations.
The Correct Pattern
from body_models.datajoint.dataset import fetch_keypoints as bm_fetch_keypoints
from body_models.datajoint.kinematic_dj import KinematicReconstruction
trial_key = {
"participant_id": participant_id,
"session_date": session_date,
"recording_timestamps": recording_timestamps,
}
full_key = {**trial_key, "kinematic_reconstruction_settings_num": 137}
# 1. Fetch qpos from KR
qpos = (KinematicReconstruction.Trial & full_key).fetch1("qpos") # (T, 41)
# 2. Fetch keypoints using the SAME function that built the KR
timestamps, keypoints_raw = bm_fetch_keypoints(trial_key, only_detected=True)
# timestamps: (T,) video-start-relative seconds
# keypoints_raw: (C, T, 87, 3) — (x, y, confidence) per camera/frame/marker
# 3. CRITICAL: verify 1:1 correspondence
assert qpos.shape[0] == keypoints_raw.shape[1], (
f"Frame count mismatch: qpos={qpos.shape[0]} vs keypoints={keypoints_raw.shape[1]}"
)
# 4. Extract components
confidences = keypoints_raw[..., 2] # (C, T, 87)
keypoints_2d = keypoints_raw[..., :2] # (C, T, 87, 2)
Why this works: fetch_keypoints(only_detected=True) applies the exact same frame filtering as KeypointDataset used when building the KR. Both arrays share the same frame indices — qpos[i] corresponds to keypoints[:, i] by construction.
Camera Parameter Reordering
KinematicReconstruction.updated_calibration stores camera parameters in Calibration table order. Keypoints from fetch_keypoints are in alphabetical SingleCameraVideo order. These may differ.
from multi_camera.datajoint.calibrate_cameras import Calibration
from multi_camera.datajoint.multi_camera_dj import SingleCameraVideo
cal_camera_names = (Calibration & parent_key).fetch1("camera_names")
camera_rows = (SingleCameraVideo & trial_key).fetch(as_dict=True, order_by="camera_name")
kp_camera_names = [row["camera_name"] for row in camera_rows]
# Reindex camera_params to match keypoint camera order
cal_names_list = list(cal_camera_names)
reorder_idx = [cal_names_list.index(name) for name in kp_camera_names]
for pkey in ("mtx", "rvec", "tvec", "dist"):
val = np.asarray(camera_params[pkey])
if val.ndim > 1:
camera_params[pkey] = val[reorder_idx]
Without this, camera i in keypoints_2d gets projected with the wrong camera's intrinsics/extrinsics.
Contiguous Segment Selection
After alignment, find the longest window with continuous detections and no time gaps:
detected = np.sum(confidences, axis=(0, 2)) # (T,) — sum across cameras and joints
dt = np.median(np.diff(timestamps))
time_gaps = np.concatenate([np.diff(timestamps), [dt]]) > (1.5 * dt)
valid = np.logical_and(detected > 0, ~time_gaps)
# find_longest_segment returns (start, end) exclusive
start, end = find_longest_segment(valid)
timestamps = timestamps[start:end]
qpos = qpos[start:end]
keypoints_2d = keypoints_2d[:, start:end]
confidences = confidences[:, start:end]
Anti-Patterns
| Pattern | Why It Fails |
|---|---|
Fetching TopDownPerson keypoints + VideoInfo timestamps separately, then matching with isin_with_tolerance() | KR timestamps zeroed relative to first detection; VideoInfo zeroed relative to video start. When first detection is at frame N>0, KR[0] incorrectly matches video[0] instead of video[N]. |
Zeroing timestamps with ts - ts[0] before matching | Destroys the physical time correspondence between independently-sourced arrays. |
Using fetch_keypoints(only_detected=False) with KR qpos | Returns all video frames including those without detections. Frame count won't match KR qpos (which only covers detected frames). |
Assuming camera_params order matches keypoint order | Calibration order != SingleCameraVideo alphabetical order. Must reorder explicitly. |
Real-world impact: The timestamp-matching anti-pattern caused a 38-frame temporal offset on trial ASB_144 cam 8 (first detection at video frame 38), producing 900px reprojection error and unusable guidance.
Invariants
- Frame correspondence:
qpos.shape[0] == keypoints_raw.shape[1]— always assert this - Camera ordering: After reordering,
camera_paramsindeximatcheskeypoints_2d[i] - Timestamps are video-start-relative: Not zeroed by KR; usable directly for time-gap detection
only_detected=Trueis mandatory when aligning with KR qpos
Quick Reference
| Function | Source | Returns |
|---|---|---|
fetch_keypoints(key, only_detected=True) | body_models.datajoint.dataset | (timestamps, keypoints) — (T,) and (C, T, 87, 3) |
KinematicReconstruction.Trial.fetch1("qpos") | body_models.datajoint.kinematic_dj | (T, 41) MuJoCo state |
KinematicReconstruction.fetch1("updated_calibration") | same | Camera params dict (Calibration order) |
Calibration.fetch1("camera_names") | multi_camera.datajoint.calibrate_cameras | List of camera names in calibration order |
Complete reference implementation: BiomechanicalFlowModel/src/biomechanical_fm/datajoint/datajoint_fetcher.py:fetch_trial_data()
Related skills: rae:datajoint-biomechanics-schema for schema details, rae:camera-model for projection conventions.
GaitLab Dataset Reference
Overview
The gait_lab schema stores traditional clinical gait lab data from Vicon c3d files. This is a separate system from the markerless multicamera (MMC) pipeline. The same participant may have data in both systems, recorded during the same session (MAC_LAB participants).
Source: GaitLabDataset/gait_lab_dataset/
Schema name: gait_lab
Table Hierarchy
Subject (subject_hash)
-> Session (subject_hash, session_name)
-> Trial (subject_hash, session_name, trial_name)
| -> Mocap (point_names, points in mm, point_rate, timestamps)
| | -> GaitEvents (event, side, time)
| | -> GaitParameters (parameter, side, value)
| | -> MocapAnalysis (property, value)
| | -> MocapProjection (toffset, camera params, error, proj, frames)
| -> GaitTrialVideo -> pose_pipeline.Video
| -> Analog (analog_names, channels, timestamps, units)
-> PTEval (eval_html) -> PTDemographics (age, diagnosis)
-> Assessment (assessment)
-> Height (computed, meters)
-> Weight (computed, kg)
-> SessionConditions -> SessionCondition + ConditionTrial
-> MPParam (property, value, all_values)
-> SubjectTrainTest (train boolean)
Table Definitions
Core Tables
Subject - De-identified subject identifier
subject_hash: varchar(100) # Primary key
Session - Clinical session grouping
-> Subject
session_name: varchar(100)
Trial - Individual gait trial
-> Session
trial_name: varchar(100)
---
timestamp: timestamp # When the trial was recorded
description: varchar(1000) # Free-text (e.g., "Barefoot with cane", "Right AFO")
vid_present: boolean # Whether video was recorded
mocap_present: boolean # Whether c3d mocap data exists
Motion Capture Data
Mocap - 3D marker data from c3d files
-> Trial
---
point_names: longblob # List of marker names (e.g., ["SACR", "LASI", "RASI", ...])
points: longblob # Shape: [n_frames, n_markers, 5]
# Channels: [x, y, z, residual, visibility]
# Units: MILLIMETERS
# Visibility: 0 = valid, non-zero = occluded
point_rate: float # Sampling rate, typically 100 Hz
frames: longblob # Frame indices
timestamps: longblob # Computed as frames / point_rate (seconds)
Analog - Force plate and EMG data
-> Trial
---
analog_names: longblob # Channel labels (e.g., forceplate channels, EMG channels)
channels: longblob # Shape: [n_channels, n_samples] (transposed)
timestamps: longblob # Synced to mocap: mocap_start + arange(n_samples) / analog_rate
units: longblob # Physical units per channel (e.g., "N", "V")
Analog data is synchronized to mocap via a shared trigger pulse. Timestamps are computed relative to the mocap frame start time.
GaitEvents - Annotated gait events from Vicon processing
-> Mocap
event_num: smallint
---
event: varchar(100) # e.g., "The instant the heel strikes the ground"
side: enum('Left', 'Right')
time: float # Event time in seconds relative to mocap timestamps
GaitParameters - Pre-computed clinical gait parameters
-> Mocap
parameter: varchar(50) # e.g., stride length, cadence
side: enum('Left', 'Right')
---
value: float
Video Linkage
GaitTrialVideo - Links gait trials to the pose pipeline's Video table
-> Trial
-> Video # from pose_pipeline (video_project, filename)
Session Metadata
SessionConditions (Computed) - Parsed trial conditions from descriptions
-> Session
---
num_conditions: smallint # Number of distinct gait conditions tested
Part table SessionCondition:
-> SessionConditions
support: varchar(50) # "none", "cane", "walker", "gait_trainer", "handheld", "crutch"
left_foot: varchar(50) # "barefoot", "afo", "smo", "kafos", "brace", "ucb", "insert"
right_foot: varchar(50) # Same options as left_foot
shoe: varchar(50) # Whether shoes were worn
---
num_mocap: smallint # Trials with synchronized mocap
num_frontal: smallint # Frontal-view-only video trials
num_sagital: smallint # Sagittal-view-only video trials
Part table ConditionTrial:
-> GaitTrialVideo
-> SessionConditions.SessionCondition
---
view: smallint # 0=with mocap sync, 1=frontal only, 2=sagittal only
Height (Computed) - Session-level height
-> Session
---
height: float # Meters, from MocapAnalysis("HEIGHT") or MPParam("Height")
Weight (Computed) - Session-level weight
-> Session
---
weight: float # kg, from MocapAnalysis("BODYMASS") or MPParam("Bodymass")
MocapProjection: Video-Mocap Synchronization
The MocapProjection table synchronizes 3D mocap markers with 2D video keypoints, solving for both a temporal offset and camera projection parameters.
Source: gait_lab_dataset/mocap_sync.py
Table Definition
-> Mocap
-> GaitTrialVideo
-> TopDownPerson # 2D keypoints from pose detection
-> VideoInfo
---
toffset: float # Temporal offset (seconds) between mocap and video
fx, fy: float # Camera focal lengths (pixels)
cx, cy: float # Principal point (image center)
tx, ty, tz: float # 3D translation (mocap -> camera frame)
rx, ry, rz: float # Rotation (Rodrigues / exponential map, stored at 1/1000 scale)
error: float # Huber loss of final projection fit (lower = better)
proj: longblob # Projected mocap positions in 2D [n_frames, n_joints, 2]
frames: longblob # Video frame indices where projection was valid
Synchronization Algorithm
The algorithm jointly optimizes temporal offset and camera parameters to align 3D mocap markers to 2D pose detections:
-
Data extraction (
get_dual_data()):- Fetches mocap 3D markers (hip, knee, ankle via Plug-In Gait bone model)
- Fetches 2D keypoints from TopDownPerson (COCO format)
- Filters to frames with high confidence 2D detections (>0.2) and valid mocap markers (visibility==0)
- Swaps Y/Z axes to align mocap vertical with image vertical
- Clips to temporal overlap with 200ms buffer at each end
-
OpenCV initialization (
cv2_calibrate()):- Uses
cv2.calibrateCamera()with all points across time as one observation - Fixes principal point at image center, fixes all distortion coefficients
- Initial focal length: ~1200 pixels
- Must achieve Huber loss < 12 (quality gate)
- Uses
-
JAX/BFGS optimization (
optimize_projection()):- 9 parameters:
[tx, ty, tz, rx, ry, rz, fx, fy, toffset_raw] - Time offset bounded:
toffset = tanh(toffset_raw) * 0.2(max +/- 200ms) - Mocap is interpolated at
timestamps + toffsetfor each candidate offset - Loss: Huber loss between detected 2D keypoints and projected 3D markers
- Uses
jax.scipy.optimize.minimize(method="BFGS", maxiter=1000) - Projection uses
SO3.exp(rvec)for rotation (jaxlie)
- 9 parameters:
-
Storage: Optimized parameters stored in MocapProjection table. Note
rx, ry, rzare stored at 1/1000 scale (divided by 1000 before insert).
Marker Mapping (Mocap <-> COCO)
The synchronization maps Plug-In Gait bone markers to COCO keypoints:
| COCO Keypoint | Plug-In Gait Marker | Description |
|---|---|---|
| Right Hip | RFEP | Right Femur Proximal |
| Right Knee | RFEO | Right Femur Origin |
| Right Ankle | RTIO | Right Tibia Origin |
| Left Hip | LFEP | Left Femur Proximal |
| Left Knee | LFEO | Left Femur Origin |
| Left Ankle | LTIO | Left Tibia Origin |
Data Statistics
- ~8,568 successful MocapProjection rows computed
- ~9,405 rows with Mocap + GaitTrialVideo + TopDownPerson available
- ~1,000 potentially recoverable with improved alignment
- ~15,359 videos with mocap present total
MMC-GaitLab Cross-System Matching
The gait_lab_dataset.mmc module compares markerless (MMC) and marker-based (GaitLab) motion capture data for the same participants (MAC_LAB sessions).
Source: gait_lab_dataset/mmc/
Core Workflow
from gait_lab_dataset.mmc import (
get_gaitlab_session_for_participant,
get_mmc_pelvis_data,
get_gaitlab_pelvis_data,
compute_session_times,
find_optimal_matching,
format_matching_report,
)
participant_id = "mac002"
# 1. Look up GaitLab session for this MMC participant (matches by date)
session_name = get_gaitlab_session_for_participant(participant_id)
# 2. Fetch pelvis trajectories from both systems
mmc_data = get_mmc_pelvis_data(participant_id)
gl_data = get_gaitlab_pelvis_data(session_name)
# 3. Compute session-relative timestamps
_, mmc_data = compute_session_times(mmc_data, "recording_time")
_, gl_data = compute_session_times(gl_data, "trial_time")
# 4. Find optimal trial matching (handles clock drift, spatial calibration)
result = find_optimal_matching(mmc_data, gl_data)
# 5. Human-readable report
report = format_matching_report(result, mmc_data, gl_data)
print(report)
MatchingResult Fields
| Field | Type | Description |
|---|---|---|
matches | list[tuple[int, int, float]] | (mmc_idx, gl_idx, offset_seconds) per matched trial |
coverage | float | Fraction of GaitLab trials matched (0-1) |
r_squared | float | Combined R-squared across all matched trials |
offset_mean | float | Mean temporal offset in seconds |
offset_std | float | Temporal offset std dev in seconds |
cost_matrix | ndarray [n_mmc, n_gl] | 1 - R-squared for each trial pair |
r_squared_matrix | ndarray [n_mmc, n_gl] | Best R-squared for each pair |
unmatched_mmc | list[int] | Indices of unmatched MMC trials |
unmatched_gl | list[int] | Indices of unmatched GaitLab trials |
affine_A | ndarray [2,2] | Global affine transform |
affine_b | ndarray [2] | Affine offset |
Synchronization Quality Interpretation
| Metric | Threshold | Interpretation |
|---|---|---|
| R-squared > 0.95 | Excellent | High confidence in trial matching |
| R-squared 0.8-0.95 | Good | Matches likely correct |
| R-squared < 0.8 | Poor | Review manually |
| Offset std < 2s | Consistent | Clock alignment stable across trials |
| Offset std > 5s | Problematic | Clock drift or matching errors |
Joint Angle Comparison
from gait_lab_dataset.mmc import (
get_mmc_joint_angles,
get_gaitlab_joint_angles,
align_joint_angles,
compute_joint_r_squared,
compute_joint_rmse,
compute_axis_correlations,
)
mmc_angles = get_mmc_joint_angles(participant_id)
gl_angles = get_gaitlab_joint_angles(session_name)
_, mmc_angles = compute_session_times(mmc_angles, "recording_time")
_, gl_angles = compute_session_times(gl_angles, "trial_time")
# Compare matched trials
for mmc_idx, gl_idx, offset in result.matches:
times, mmc, gl = align_joint_angles(mmc_angles[mmc_idx], gl_angles[gl_idx], offset)
r2 = compute_joint_r_squared(mmc, gl) # Per-joint R-squared (handles sign flips)
rmse = compute_joint_rmse(mmc, gl) # Per-joint RMSE in degrees
Joint angle mappings:
- Sagittal plane joints:
hip_flexion,knee_angle,ankle_angle(bilateral) - GaitLab markers:
LHipAngles,RHipAngles,LKneeAngles,RKneeAngles,LAnkleAngles,RAnkleAngles - Each has 3 channels: sagittal (X), frontal (Y), transverse (Z) in degrees
Plug-In Gait Marker Names
Surface Markers (17)
| Marker | Full Name |
|---|---|
| SACR | Sacral |
| LASI / RASI | Left / Right Anterior Superior Iliac Spine |
| LPSI / RPSI | Left / Right Posterior Superior Iliac Spine |
| LTHI / RTHI | Left / Right Thigh |
| LKNE / RKNE | Left / Right Knee |
| LTIB / RTIB | Left / Right Tibia |
| LANK / RANK | Left / Right Ankle |
| LHEE / RHEE | Left / Right Heel |
| LTOE / RTOE | Left / Right Toe |
Bone Markers (Used for Anatomical Calibration)
Naming pattern: {SEGMENT}{SUFFIX} where suffix is O=Origin, P=Proximal, A=Anterior, L=Lateral.
Segments: PEL (Pelvis), RFE/LFE (Femur), RTI/LTI (Tibia), RFO/LFO (Foot), RTO/LTO (Toe)
Common Queries
Count sessions with c3d data
from gait_lab_dataset.dataset_dj import Trial, Session
sessions_with_mocap = Session & (Trial & 'mocap_present=1')
print(f"Sessions with c3d data: {len(sessions_with_mocap)}")
Find MAC_LAB sessions with corresponding c3d files
from multi_camera.datajoint.sessions import Session as MMCSession, Recording
from multi_camera.datajoint.multi_camera_dj import MultiCameraRecording
from gait_lab_dataset.dataset_dj import Session as GLSession, Trial, Mocap
# Get MAC_LAB MMC sessions
mmc_sessions = (MMCSession & (Recording & (MultiCameraRecording & "video_project='MAC_LAB'"))).fetch(as_dict=True)
# Get GaitLab sessions with mocap, compute dates from trial timestamps
gl_sessions = (GLSession & Mocap).fetch(as_dict=True)
gl_dates = {}
for gl in gl_sessions:
timestamps = (Trial & gl).fetch('timestamp')
if len(timestamps) > 0:
gl_dates[gl['session_name']] = min(timestamps).date()
# Match by date
matched = sum(1 for mmc in mmc_sessions
if any(mmc['session_date'] == d for d in gl_dates.values()))
Query MocapProjection synchronization error
from gait_lab_dataset.mocap_sync import MocapProjection
import numpy as np
errors = MocapProjection.fetch('error')
print(f"Entries: {len(errors)}, Median error: {np.median(errors):.4f}")
Get gait events for a trial
from gait_lab_dataset.dataset_dj import GaitEvents
events = (GaitEvents & key).fetch(as_dict=True)
for e in events:
print(f" {e['side']} {e['event']} at {e['time']:.3f}s")
Get synchronized training data
from gait_lab_dataset.fetch_training_dataset import get_synced_trial
# Returns aligned: [timestamps, keypoints3d, keypoints2d, mocap, mocap_weight, mocap_phase, mocap_phase_weight]
data = get_synced_trial(key)
Data Conventions
Units
| Data | Units |
|---|---|
Mocap marker positions (points) | millimeters |
| GaitLab joint angles | degrees |
| Height | meters |
| Weight | kg |
| Gait event times | seconds (relative to mocap start) |
MocapProjection toffset | seconds |
MocapProjection rx, ry, rz | Rodrigues vector / 1000 (stored scaled down) |
| Analog timestamps | seconds (synced to mocap clock) |
| MMC pelvis data | meters (from KinematicReconstruction) |
Coordinate Axes
| System | X | Y | Z |
|---|---|---|---|
| Mocap raw (Vicon) | Forward | Lateral | Vertical |
| After sync transform | Forward | Vertical | Lateral |
The synchronization code swaps Y/Z to align the mocap vertical axis with the image height axis: points[:, :, [0, 2, 1, 3, 4]]
Visibility Flags
- Mocap:
points[:, :, 4] == 0means visible/valid; non-zero means occluded - Video 2D keypoints:
keypoints[:, :, 2] > 0.2indicates high-confidence detection
Known Issues
| Issue | Details |
|---|---|
| Knee angle sign convention | GaitLab and MMC use opposite signs (~-0.9 correlation). compute_joint_r_squared handles this via linear regression. |
| Unit mismatch | GaitLab points in mm; MMC pelvis_xyz in meters. Data loading functions handle conversion. |
find_optimal_matching performance | Builds O(n*m) cost matrix with offset search. ~30s per participant. |
rx, ry, rz storage scaling | Stored at 1/1000 scale in MocapProjection (divided before insert). Multiply by 1000 to recover original optimization values. |
| Missing TopDownPerson | ~5,959 videos lack 2D pose detections, preventing MocapProjection computation. |
cv2_calibrate duplicate | mocap_sync.py contains two definitions of cv2_calibrate. The second (line 232) is the active one. |
Quick Reference: Imports
# Core schema tables
from gait_lab_dataset.dataset_dj import (
Subject, Session, Trial, Mocap, GaitTrialVideo,
GaitEvents, GaitParameters, Analog, Height, Weight,
SessionConditions, MocapAnalysis, MPParam, SubjectTrainTest,
ValidGaitEvents, PTEval, Assessment,
)
# Synchronization
from gait_lab_dataset.mocap_sync import MocapProjection
# MMC-GaitLab cross-system matching
from gait_lab_dataset.mmc import (
get_gaitlab_session_for_participant,
find_optimal_matching,
format_matching_report,
get_mmc_pelvis_data,
get_gaitlab_pelvis_data,
compute_session_times,
get_mmc_joint_angles,
get_gaitlab_joint_angles,
align_joint_angles,
compute_joint_r_squared,
compute_joint_rmse,
compute_axis_correlations,
)
from gait_lab_dataset.mmc.types import MatchingResult
# Training data
from gait_lab_dataset.fetch_training_dataset import get_synced_trial
# Associated tables from other systems
from pose_pipeline.pipeline import Video, VideoInfo, TopDownPerson
from multi_camera.datajoint.sessions import Session as MMCSession, Recording
from multi_camera.datajoint.multi_camera_dj import MultiCameraRecording
File Paths
| Component | File |
|---|---|
| Schema definition | gait_lab_dataset/dataset_dj.py |
| Video-mocap sync | gait_lab_dataset/mocap_sync.py |
| Gait event processing | gait_lab_dataset/mocap_processing.py |
| Training data fetch | gait_lab_dataset/fetch_training_dataset.py |
| Gait phase estimation | gait_lab_dataset/gait_phase_kalman.py |
| MMC data loading | gait_lab_dataset/mmc/data_loading.py |
| MMC trial matching | gait_lab_dataset/mmc/matching.py |
| MMC joint comparison | gait_lab_dataset/mmc/joint_comparison.py |
| MMC types | gait_lab_dataset/mmc/types.py |
Gait Metrics Reference
Overview
The gait analysis pipeline detects walking segments from kinematic reconstructions, extracts gait events (foot contacts), and computes spatiotemporal metrics, joint kinematics, and the Gait Deviation Index (GDI). It works with both multi-camera (MMC) and monocular (PBL) data.
Schemas: project_body_models_gait_cycles (walking detection, gait events) and project_gdi (GDI computation)
Source: BodyModels/body_models/datajoint/gait/ and BodyModels/body_models/biomechanics_mjx/gait/
Finding Valid Walking Segments
Walking segments are detected by GaitTransformer (MMC) and GaitTransformerMonocular (PBL). Both use a GaitTransformer neural network + Kalman filter pipeline to identify when a person is walking.
Detection Pipeline
- Keypoint extraction from sites (17 keypoints mapped to GastNet order)
- GaitTransformer inference produces 8-channel gait phase output
- Kalman filter smoothing refines phase estimates, produces error signal
- Walking probability =
prob_err * prob_speedwhere:prob_err = exp(-errors * 2)(Kalman filter prediction error)prob_speed= sigmoid of phase velocity * consistency with median
- Hysteresis thresholding (0.35/0.65) + median filter deglitching
- Minimum segment length filter (>100 frames)
Querying Walking Segments
from body_models.datajoint.gait.gait_cycles_dj import GaitTransformer, GaitTransformerMonocular
# Get the longest walking segment for a trial
key = {
'participant_id': '104', 'session_date': date(2023, 7, 21),
'kinematic_reconstruction_settings_num': 137,
'transformer_method_name': 'default',
}
# WalkingSegment part table has aggregate stats per segment
segments = (GaitTransformer.WalkingSegment & key).fetch(as_dict=True, order_by='num_frames DESC')
best_segment = segments[0] # Longest segment
# Aggregate stats available per segment:
# num_frames, cadence, velocity,
# stance_left, stance_right, ss_left, ss_right, dst,
# length_left, length_right, width_left, width_right,
# cycles_left, cycles_right (interpolated gait cycles, 100 timepoints each)
Filtering for Walking Trials
from multi_camera.datajoint.annotation import VideoActivity
# Only trials annotated as walking
walking_trials = (
GaitTransformer
& (VideoActivity & {'video_activity': 'Overground Walking'})
& {'kinematic_reconstruction_settings_num': 137}
)
# From a specific project
from multi_camera.datajoint.multi_camera_dj import MultiCameraRecording
walking_controls = walking_trials & (MultiCameraRecording & {'video_project': 'GAIT_CONTROLS'})
Monocular Walking Segments
from portable_biomechanics_sessions.session_annotations import VideoActivity as PBLVideoActivity
monocular_walking = (
GaitTransformerMonocular
& (PBLVideoActivity & {'video_activity': 'Overground Walking'})
& {'monocular_reconstruction_settings_num': -108}
)
Gait Events
GaitTransformer stores four event arrays as longblobs:
| Field | Description |
|---|---|
left_down | Left foot initial contact timestamps |
left_up | Left foot toe-off timestamps |
right_down | Right foot initial contact timestamps |
right_up | Right foot toe-off timestamps |
events = (GaitTransformer & key).fetch1()
left_down = events['left_down'] # numpy array of timestamps
right_down = events['right_down']
# Filter events to longest walking segment
timestamps = (KinematicReconstruction.Trial & key).fetch1('timestamps')
start, end = (GaitTransformer.WalkingSegment & key).fetch(
'segment_start', 'segment_end', order_by='num_frames DESC', limit=1
)
t0, te = timestamps[start[0]], timestamps[end[0]]
valid_left_down = [t for t in left_down if t0 <= t <= te]
Gait Cycle Definition
A gait cycle runs from one foot contact to the next same-foot contact:
- Stride = foot_down[i] to foot_down[i+1] (one full cycle)
- Stance phase = foot_down to foot_up (foot on ground)
- Swing phase = foot_up to next foot_down (foot in air)
- Step = contralateral foot_down to ipsilateral foot_down
Spatiotemporal Metrics
Step-by-Step Metrics (Steps Part Table)
GaitTransformer.Steps stores per-step metrics:
| Field | Description |
|---|---|
side | Left or Right |
step_time | Timestamp of the step |
velocity | Walking velocity (m/s) at this step |
cadence | Steps/min at this step |
ss_left | Left single support (%) |
ss_right | Right single support (%) |
dst | Double support time (%) |
length | Step length (m), NULL for monocular |
width | Step width (m), NULL for monocular |
steps = (GaitTransformer.Steps & key).fetch(as_dict=True)
Computing Detailed Metrics from Raw Data
For more detailed analysis beyond the stored tables, use gait_analysis.py:
from body_models.datajoint.gait.gait_analysis import compute_gait_analysis, summarize_gait_metrics
results = compute_gait_analysis(key)
# Returns dict with: timestamps, reoriented_qpos, reoriented_joints, events_raw, metrics
# metrics contains DataFrames:
# step_metrics: step_time, step_length per step
# gait_cycle_metrics: cycle_time, stride_length, walking_speed per cycle
# phase_metrics: stance%, swing% per cycle
# joint_rom: ROM per joint per cycle (radians)
# knee_swing_rom: knee ROM during swing only
# ankle_at_foot_down: ankle angle at each foot contact
summary = summarize_gait_metrics(results)
# Returns DataFrame with Mean/Std per side + asymmetry indices
Or use the standalone function (does not require GaitTransformer table):
from body_models.biomechanics_mjx.gait.gait_metrics import compute_gait_analysis
results = compute_gait_analysis(key, remove_first_cycle=True)
# metrics dict: step_metrics, phase_metrics, cycle_rom, phase_rom, ankle_at_foot_down
Metric Definitions
Step Metrics
| Metric | Definition | Units |
|---|---|---|
| Step length | Distance between opposite heels at foot contact | m or mm |
| Step width | Lateral distance between heels at foot contact | m or mm |
| Step time | Time between contralateral contacts | s |
| Stride length | Pelvis displacement over one gait cycle | m |
| Cycle time | Duration of one full gait cycle | s |
| Walking speed | Stride length / cycle time | m/s |
| Cadence | Steps per minute derived from phase velocity | steps/min |
Phase Metrics
| Metric | Definition | Normal Range |
|---|---|---|
| Stance phase | % of stride foot is on ground | ~60% |
| Swing phase | % of stride foot is in air | ~40% |
| Single support | % of stride only one foot on ground | ~40% |
| Double support | % of stride both feet on ground | ~20% |
Joint Angles Analyzed
| Joint | Index in qpos | Description |
|---|---|---|
hip_flexion_r/l | via ForwardKinematics | Hip flexion/extension |
hip_adduction_r/l | via ForwardKinematics | Hip adduction/abduction |
hip_rotation_r/l | via ForwardKinematics | Hip internal/external rotation |
knee_angle_r/l | via ForwardKinematics | Knee flexion/extension |
ankle_angle_r/l | via ForwardKinematics | Ankle dorsiflexion/plantarflexion |
Asymmetry Index
asymmetry = |left_mean - right_mean| / max(left_mean, right_mean) * 100
Computed for: step time, step length, cycle time, stride length, walking speed, stance phase, hip flexion ROM, knee angle ROM.
Gait Deviation Index (GDI)
GDI quantifies overall gait pathology by comparing joint angle patterns to a control group using PCA.
Schema: project_gdi
Tables
| Table | Description |
|---|---|
GDIJointsLookup | Defines which joints to use (method 0 = 13 joints: pelvis tilt/list/rotation + bilateral hip/knee/ankle) |
GDICyclesMethod | Links GaitTransformer trials to GDI computation (MMC) |
GDICycles | Stores parsed gait cycles per trial: parsed_cycles_right, parsed_cycles_left (Joint x 50 timepoints x Steps) |
GDICyclesMethodMonocular | Links GaitTransformerMonocular trials to GDI (PBL) |
GDICyclesMonocular | Same as GDICycles for monocular data |
Computing GDI
from body_models.biomechanics_mjx.gait.gdi import get_gdi_matrix_session, pca_gdi
# Get control group matrix (aggregate per session)
ctrl_keys = (GDICyclesMethod & {'video_project': 'GAIT_CONTROLS'}).fetch('KEY')
ctrl_matrix, ctrl_session_keys = get_gdi_matrix_session(ctrl_keys)
# Get treatment group matrix
trtmt_keys = (GDICyclesMethod & {'video_project': 'PROSTHETIC_GAIT'}).fetch('KEY')
trtmt_matrix, trtmt_session_keys = get_gdi_matrix_session(trtmt_keys)
# Compute GDI scores
gdi_ctrl, gdi_trtmt, ctrl_pca, trtmt_pca, n_components = pca_gdi(ctrl_matrix, trtmt_matrix)
# GDI ~ 100 = normal, lower = more pathological (each 10-point drop = 1 SD from normal)
GDI Interpretation
| GDI Score | Interpretation |
|---|---|
| >= 100 | Normal gait pattern |
| 90-100 | Mild deviation |
| 80-90 | Moderate deviation |
| < 80 | Severe deviation |
Walking Segment Validation
Stride Length Filter
gait_analysis.py filters cycles by minimum stride length (default 0.1m) using pelvis displacement. It also discards the first valid cycle per side to remove gait initiation artifacts.
# Internal logic in _filter_events_by_valid_cycles:
# 1. For each foot-down-to-foot-down cycle, compute pelvis displacement
# 2. Keep cycles where displacement >= 0.1m (walking forward)
# 3. Drop first valid cycle per side (initiation artifact)
Quality Filtering with MultiTrackingDetection
from multi_camera.datajoint.quality_metrics import MultiTrackingDetection
# Filter for trials with no tracking breaks
good_track = (MultiTrackingDetection & "top_down_method=12 and reconstruction_method=2") & "num_breaks=0"
clean_gait = GaitTransformer & good_track
GaitAnalytics (Per-Step/Cycle Detail)
Schema: idjuraskovic_gait_analytics
Source: BodyModels/body_models/datajoint/gait/gait_analytics_dj.py
Branch: gait_metrics_rebase (in development, not merged to main)
GaitAnalytics stores detailed per-step and per-cycle metrics in normalized part tables. Preferred over StepMetrics when available — provides individual step/cycle data rather than trial-level aggregates.
Table Structure
GaitAnalytics (master) → KinematicReconstruction.Trial, steps_count
| Part Table | Primary Data | Key Fields |
|---|---|---|
Steps | Per-step spatiotemporal | side, step_length, step_width, step_time, t_down, t_up |
PhaseMetrics | Per-cycle phase breakdown | side, phase (stance/swing/single_support/double_support), percent_of_stride, duration_s |
CycleROM | Joint ROM per gait cycle | side, joint, rom, min_angle, max_angle |
PhaseROM | Joint ROM per phase | side, joint, phase (stance/swing), rom |
AnkleAtFootDown | Ankle at contact | side, ankle_angle, foot_down_time |
Querying GaitAnalytics
from body_models.datajoint.gait.gait_analytics_dj import GaitAnalytics
key = {'participant_id': '104', 'session_date': date(2023, 7, 21),
'kinematic_reconstruction_settings_num': 137}
# Per-step data
steps = (GaitAnalytics.Steps & key).fetch(as_dict=True)
# Phase breakdown per cycle
phases = (GaitAnalytics.PhaseMetrics & key).fetch(as_dict=True)
# Joint ROM per cycle
rom = (GaitAnalytics.CycleROM & key).fetch(as_dict=True)
Utility Functions
from body_models.utils.gait_analytics_utils import (
fetch_all_gait_analytics, # All part tables as dict of DataFrames
compute_step_summary_statistics, # Median/std/min/max per side
compute_all_asymmetry_indices, # Signed: 100 * (R - L) / avg(L, R)
compute_normative_step_statistics, # Control group step stats
compute_normative_walking_speed, # Control group walking speed
highlight_outliers, # Styler coloring outside normative range
)
# Fetch everything for a trial
data = fetch_all_gait_analytics(key)
# Returns: {'steps': df, 'phases': df, 'cycle_rom': df, 'phase_rom': df, 'ankle_at_foot_down': df}
# Summary statistics
summary = compute_step_summary_statistics(data['steps'], aggregation='median')
# Asymmetry (signed: positive = right > left)
asym = compute_all_asymmetry_indices(data['steps'], data['phases'], aggregation='median')
Control group filter: participant_id LIKE '6%'
Quick Reference: Imports
# Walking detection tables
from body_models.datajoint.gait.gait_cycles_dj import (
GaitTransformer, GaitTransformerMonocular,
TransformerMethod, TransformerMethodMonocular,
TransformerMethodLookup,
)
# GDI tables
from body_models.datajoint.gait.gdi_dj import (
GDICycles, GDICyclesMonocular,
GDICyclesMethod, GDICyclesMethodMonocular,
GDIJointsLookup,
)
# Detailed gait analysis (DataJoint-based, uses GaitTransformer)
from body_models.datajoint.gait.gait_analysis import (
compute_gait_analysis, summarize_gait_metrics, plot_gait_analysis, StepMetrics,
)
# Standalone gait metrics (direct from KinematicReconstruction)
from body_models.biomechanics_mjx.gait.gait_metrics import (
compute_gait_analysis as compute_gait_analysis_standalone,
calculate_step_metrics,
)
# GDI computation
from body_models.biomechanics_mjx.gait.gdi import (
get_gdi_matrix_session, get_gdi_matrix_steps, pca_gdi,
)
# GaitAnalytics tables (branch: gait_metrics_rebase, not merged)
from body_models.datajoint.gait.gait_analytics_dj import GaitAnalytics
from body_models.utils.gait_analytics_utils import (
fetch_all_gait_analytics, compute_step_summary_statistics,
compute_all_asymmetry_indices, compute_normative_step_statistics,
compute_normative_walking_speed, highlight_outliers,
)
# Walking activity annotations
from multi_camera.datajoint.annotation import VideoActivity
from portable_biomechanics_sessions.session_annotations import VideoActivity as PBLVideoActivity
Common Mistakes
| Mistake | Fix |
|---|---|
Querying GaitTransformer without transformer_method_name | Add 'transformer_method_name': 'default' to key |
| Using all events instead of filtering to walking segment | Filter events to [t0, te] of best WalkingSegment |
Forgetting kinematic_reconstruction_settings_num=137 | Always specify method number |
| Computing metrics on gait initiation | Use remove_first_cycle=True or the built-in stride-length filter |
Mixing gait_metrics.py and gait_analysis.py | gait_metrics.py is standalone; gait_analysis.py uses GaitTransformer events |
| Monocular GaitTransformer uses different settings | Default monocular_reconstruction_settings_num=-108 for monocular |
GitHub Activity Review
Periodic cross-repository review of GitHub activity, organized thematically with
appropriate depth. Produces a structured report in scratch/.
Parameters
- Period: Default 2 weeks. User can override (e.g., "last month", "since March 1").
- GitHub user: Detect from
gh auth status. User can override. - Orgs: Detect from
gh api user/orgs. User can specify which are relevant.
Process
digraph activity_review {
rankdir=TB;
node [shape=box];
discover [label="Pass 1: Discover\nFind all active repos"];
triage [label="Pass 2: Triage\nCount commits, rank by activity"];
gather [label="Pass 3: Gather\nParallel subagents per repo cluster"];
synthesize [label="Pass 4: Synthesize\nThematic organization"];
deepen [label="Pass 5: Deepen\nFill gaps, read PROGRESS files"];
finalize [label="Pass 6: Finalize\nTeam report, cross-cutting themes"];
discover -> triage -> gather -> synthesize -> deepen -> finalize;
}
Pass 1: Discover Active Repos
# Get orgs
gh api user/orgs --jq '.[].login'
# For each relevant org, find repos pushed in period
gh api "orgs/ORG/repos?sort=pushed&per_page=30&type=all" \
--jq '.[] | select(.pushed_at > "SINCE_DATE") | .full_name'
# Also check personal repos
gh api "users/USER/repos?sort=pushed&per_page=20&type=owner" \
--jq '.[] | select(.pushed_at > "SINCE_DATE") | .full_name'
Pass 2: Triage by Commit Count
For each discovered repo, count commits by the user and by others:
gh api "repos/REPO/commits?since=SINCE&author=USER&per_page=100" --jq 'length'
gh api "repos/REPO/commits?since=SINCE&per_page=100" --jq 'length'
Rank repos into tiers:
- Heavy (20+ user commits): Full investigation with PROGRESS files
- Moderate (5-19): Commit log analysis with theme grouping
- Light (1-4): Brief summary
- Others only (0 user, >0 total): Team activity section
Pass 3: Gather Data (Parallel Subagents)
Dispatch subagents in parallel. Group small repos into a single agent. For each:
- README (first 80 lines):
gh api repos/REPO/readme --jq .content | base64 -d | head -80 - User's commits:
gh api "repos/REPO/commits?since=SINCE&author=USER&per_page=100" --jq '.[] | "\(.commit.committer.date) \(.sha[0:8]) \(.commit.message | split("\n")[0])"' - Others' commits: same query filtering
select(.author.login != "USER") - PRs:
gh pr list --repo REPO --state all --search "created:>SINCE" --limit 20 --json number,title,author,state,mergedAt - PROGRESS.md / CHANGELOG.md:
gh api repos/REPO/contents/PROGRESS.md --jq .content | base64 -d | head -200
IMPORTANT: Subagents often lack Bash permissions. Gather all raw data yourself in the main conversation, then use subagents only for analysis/writing if needed.
Pass 4: Synthesize Thematically
Do NOT organize the report repo-by-repo only. The first sections should be thematic:
- Read all gathered data
- Identify cross-cutting themes (shared techniques, common goals, connected projects)
- Look for mono-repos, submodules, and shared libraries that tie projects together
- Group commits by intellectual contribution, not just repository
Common themes to look for:
- Same technique applied across repos (e.g., contrastive learning, JIT compilation)
- Pipeline stages (data prep -> training -> evaluation -> deployment)
- Shared infrastructure or library development
- Factored/disentangled representations across different model types
- Validation against gold-standard comparators
Pass 5: Deepen (Critical -- Do Not Skip)
After the initial synthesis, actively look for gaps:
- PROGRESS.md files: These contain versioned experimental results with quantitative outcomes, ablation tables, and methodological decisions. They are far more informative than commit messages alone. Read them for any repo with 10+ commits.
- Submodule relationships: Check
.gitmodulesfor mono-repos. Trace how subprojects relate to each other (shared data pipelines, common evaluation frameworks). - Cross-repo patterns: If the same function/technique name appears in multiple repos (e.g., "phaser", "contrastive", "STP"), trace how it was developed in one and applied in others.
- Quantitative results: Pull specific numbers (error rates, speedups, SRM values, accuracy) -- these make the report concrete rather than vague.
- Negative results: Architecture ablations that failed, approaches that were rejected. These are often the most interesting findings.
Pass 6: Finalize
Write the report to scratch/YYYY-MM-DD-github-activity-review/README.md.
Report structure:
# Cross-Repository Activity Report: USERNAME
**Period:** DATE_RANGE
**Organization:** ORG_NAMES
## Executive Summary
3-5 bullet points on major thrusts with commit counts.
## Thematic Analysis
### Theme 1: [Descriptive Name]
Cross-cutting narrative tying together work across repos.
Include quantitative results, key decisions, connections.
### Theme 2: ...
## Per-Repository Details
### Repo Name (N commits by user, M by others)
**What it is:** 1-2 sentences from README.
**Key work:** Grouped by feature/theme, not chronological.
**Results:** Specific numbers from PROGRESS files.
## Team Activity
| Contributor | Repos | Activity Summary |
|---|---|---|
| name | repo list | What they did |
Common Mistakes
| Mistake | Fix |
|---|---|
| Organizing only by repo, not by theme | Lead with thematic sections; repos are reference |
| Ignoring PROGRESS.md files | These have the real results; always check heavy repos |
| Missing submodule relationships | Check .gitmodules; mono-repos hide connected work |
| Shallow commit-message-only analysis | Read PRs, PROGRESS files, and README for context |
| Skipping team member activity | Always include "others only" repos in team section |
| Not tracing cross-repo techniques | If same keyword appears in 2+ repos, it's a theme |
| Vague summaries without numbers | Pull specific metrics, error rates, speedups |
| Single-pass report | The deepen pass catches 30-50% of the interesting content |
Golden / Parity Test Harness
Overview
Integration and parity tests compare a port (or refactor) against a reference (PyTorch model, prior implementation, multi-view ground truth). The reference outputs are "golden" data. This skill codifies the preferred way to manage that golden data across this environment, so every repo does it the same way.
The one rule that matters:
Golden data is generated, not committed. A config-driven test harness regenerates it on demand;
tests/golden/is.gitignored. The only test inputs checked into git are tiny, public-domain fixtures (e.g. one image) or a config that points at a DataJoint trial.
Committing golden NPZs (or trial-derived tests/data/) is an anti-pattern by
default — it bloats the repo, goes stale silently, and hides which reference the
test actually depends on. Vendor goldens only when explicitly chosen to make
tests runnable offline, and document that choice (see "When committing is OK").
Why generated, not committed
- Golden NPZs are large (component-level captures, video E2E). They bloat clones.
- A committed golden is a frozen number with no provenance — when the reference model changes, the test passes against a stale golden and the regression hides.
- A config that names the trial (DataJoint subject/session) or the fixture image makes the test's input explicit and reproducible from source.
Preferred input sources (in order)
- DataJoint trial via config (ideal for E2E/integration). The config names a
standard test subject/session; the harness fetches synchronized video + ground
truth from the database. Nothing trial-derived is committed. See the
/pose-datajointand/fetching-synchronized-dataskills for query patterns. - Checked-in public-domain fixture (fine for component/unit tests). One small
image (COCO val or similar, ~200 KB) under
tests/fixtures/, committed. Use for deterministic component-level parity that must run without a database. - Vendored golden NPZ (last resort, must be documented). Only when the test must run with neither torch nor the database available (e.g. offline CI).
File layout
tests/
├── conftest.py # shared fixtures, flags, load-or-generate logic
├── fixtures/
│ └── test_image.png # public-domain input — CHECKED IN (~200 KB)
├── golden/ # .gitignored — auto-generated, NEVER committed
│ ├── component_golden.npz # per-component: encoder, decoder layers, heads
│ └── e2e_golden.npz # E2E outputs on the configured DataJoint trial
├── test_component_parity.py # uses checked-in fixture + component golden
└── test_integration.py # uses DataJoint trial + e2e golden (+ benchmark)
Add to .gitignore (every repo with parity tests):
tests/golden/
Keep a tests/golden/.gitkeep if you want the directory to exist on a fresh
clone, but never commit the .npz files themselves.
The load-or-generate fixture pattern
The harness generates from the reference on first run, caches the NPZ, and reuses
it on subsequent reference-free runs. No standalone generate_golden.py that must
be remembered and run by hand.
# tests/conftest.py
from pathlib import Path
import numpy as np
import pytest
GOLDEN_DIR = Path(__file__).parent / "golden"
FIXTURE_DIR = Path(__file__).parent / "fixtures"
def pytest_addoption(parser):
parser.addoption("--run-comparison", action="store_true", default=False,
help="Run reference-comparison tests (generates golden if missing)")
parser.addoption("--run-integration", action="store_true", default=False,
help="Run DataJoint integration tests")
parser.addoption("--regenerate-golden", action="store_true", default=False,
help="Force re-generation even when the golden NPZ exists")
@pytest.fixture(scope="module")
def component_golden(request, test_image):
"""Load cached golden, or generate it from the reference (needs torch)."""
path = GOLDEN_DIR / "component_golden.npz"
if path.exists() and not request.config.getoption("--regenerate-golden"):
return dict(np.load(path, allow_pickle=False))
pytest.importorskip("torch", reason="Golden missing — run with PyTorch to generate.")
outputs = _generate_component_golden(test_image) # project-specific
GOLDEN_DIR.mkdir(exist_ok=True)
np.savez_compressed(path, **outputs)
return outputs
The same shape works for e2e_golden, fed by an integration_video_frames
fixture that pulls the configured DataJoint trial.
Behavior matrix
| Golden exists | Reference (torch) available | --regenerate-golden | Result |
|---|---|---|---|
| Yes | — | No | Load from disk, run comparison |
| Yes | Yes | Yes | Regenerate, save, run comparison |
| Yes | No | Yes | Skip with clear message |
| No | Yes | — | Generate, save, run comparison |
| No | No | — | FAIL with "run with PyTorch installed to generate" |
Never let a missing golden silently skip — a parity test that quietly no-ops is worse than no test. Skip only when the reference is unavailable and the golden is cached; otherwise fail loudly.
Test marking and usage
@pytest.mark.comparison
class TestComponentParity:
def test_encoder(self, model, component_golden, test_image):
cos = cosine_similarity(model.encode(test_image), component_golden["encoder_features"])
assert cos > 0.9999
@pytest.mark.integration
class TestEndToEnd:
def test_e2e(self, model, e2e_golden, integration_video_frames):
out = model.predict(integration_video_frames[0])
for k in ("points", "conf", "camera_poses"):
np.testing.assert_allclose(out[k], e2e_golden[k], atol=0.01)
pytest tests/ --run-comparison # generates golden first time, caches after
pytest tests/ --run-comparison --regenerate-golden # after the reference model changes
pytest tests/ --run-comparison --run-integration -s # full suite incl. DataJoint trial + benchmark
When committing a golden IS acceptable
Only when all of these hold, and it is documented in the README:
- The test must run where neither the reference (torch) nor the database is available (e.g. an offline CI lane), AND
- A regeneration script/command exists and is referenced next to the data
(e.g. a
.gitignorecomment:# regenerate with scripts/capture_pt_stages.py), AND - The committed artifact is small. Large captures stay generated/
.gitignored even in this case.
If you vendor a golden, add a README note stating why it is committed and how to regenerate it. Silent vendoring is the anti-pattern, not vendoring per se.
Anti-patterns to flag and fix
git ls-filesshows committedtests/golden/*.npz→ should be generated + ignored.- Committed
tests/data/<subject>/…arrays derived from a DataJoint trial → replace with a config that names the trial and fetches at test time. - A parity test that
pytest.skips silently when the golden is missing. - A standalone
generate_golden.pythat must be run by hand and is easy to forget — fold generation into theconftest.pyload-or-generate fixture instead.
Guards that cannot fail
A parity suite is only worth its gates. Four shapes recur, and each produces a test that passes for the wrong reason — worse than no test, because nobody distrusts a green one. Prove every guard by MUTATION: break what it guards, watch it fail, revert.
- Asserting a value against the constant it was derived from.
assert jax.config.jax_default_matmul_precision == config.MATMUL_PRECISIONis an identity when the fixture set the flag from that constant — it holds for every value, including the wrong one. Pin the required VALUE to a literal in the test, and assert the constant against it, so the constant is what is guarded rather than its plumbing. - A guard conditioned on the work being incomplete.
assert unfinished_stages(),assert len(STUBS) > 0, "this hazard is still unwritable" — each goes inert or red the moment the port finishes, i.e. exactly when it starts mattering. Make the precondition the check's REACH, not the work remaining. - A set defined by subtraction, then asserted to partition. If
captured = all_keys - uncaptured, thencaptured | uncaptured == all_keysis true by construction and cannot fail. - A test whose subject never enters the assertion — e.g. a "the port does not sort" test that inspects only the reference array.
See also the jit-cache hazard in jax-memory-and-retracing: a mutation absorbed
by a cached trace makes any of the above look proven.
Provenance: what a stamp must cover
- Hash the GENERATOR SOURCE into the golden, not just the inputs, and re-check it on load. A golden whose recipe is unknown cannot be reproduced later. Never backfill a stamp after the fact — that converts an honest gap into a false claim.
- Every file the generator READS is a recipe source, including the config module it pulls constants from. A config edit that changes what a golden MEANS while its stamp still reads "current" is undetectable.
- A stale or missing stamp should FAIL LOUDLY, not silently regenerate mid-suite: an unattended rebuild voids every gate measured from the old numbers, and on a shared GPU it is an unannounced multi-GB job.
- Pin the capture DEVICE explicitly.
"cuda" if available else "cpu"is ambient: one capture in a shell without a visible card lands on CPU, and a CPU and a CUDA capture differ exactly in reduction order — so one wrong-device golden makes its siblings' floors describe a different computation. - Commit the generator before trusting its output. A generator fix left uncommitted while goldens are built against it leaves HEAD rejecting every golden.
Measure the floor on every backend the suite runs on
A tolerance derived on one backend is a tolerance for that backend. Measured on one port: the CPU arm of the same comparison was 3.5x wider than the GPU arms, so a GPU-only floor is too tight for a CPU run of the same suite. The same holds across input scales — an arithmetic floor that grows with sequence length makes a constant measured at one size a gate for that size only. Derive from an envelope over the configurations the suite actually runs, and have the suite report which backend produced its numbers.
Related skills
/scaffold-repo— creates thetests/layout and.gitignore(ensuretests/golden/)./pose-datajoint,/fetching-synchronized-data— fetch the configured trial for E2E goldens.- The workspace
eqx-portingskill (Step 6, parity tests) drives what to capture from the reference; this skill governs how the golden data is stored and regenerated.
Set up an investigation
Instructions
- Create a folder in
{REPO_ROOT}/scratch/with the format{YYYY-MM-DD}-{descriptive-name}:mkdir scratch/$(python -c "import datetime; print(datetime.date.today().isoformat())")-{descriptive-name} - Create a
README.mdin this folder with: task description, background context, task checklist. Update with findings as you progress. - Create scripts and data files as needed for empirical work.
- For complex investigations, split into sub-documents as patterns emerge.
Investigation Patterns
These are common patterns, not rigid categories. Most investigations blend multiple patterns.
Tracing - "trace from X to Y", "what touches X", "follow the wiring"
- Follow call stack or data flow from a focal component to its connections
- Can trace forward (X → where does it go?) or backward (what leads to X?)
- Useful for: assessing impact of changes, understanding coupling
System Architecture Archeology - "document how the system works", "archeology"
- Comprehensive documentation of an entire system or flow for reusable reference
- Start from entry points, trace through all layers, document relationships exhaustively
- For complex systems, consider numbered sub-documents (01-cli.md, 02-data.md, etc.)
Bug Investigation - "figure out why X happens", "this is broken"
- Reproduce → trace root cause → propose fix
- For cross-repo bugs, consider per-repo task breakdowns
Technical Exploration - "can we do X?", "is this possible?", "figure out how to"
- Feasibility testing with proof-of-concept scripts
- Document what works AND what doesn't
Design Research - "explore the API", "gather context", "design alternatives"
- Understand systems and constraints before building
- Compare alternatives, document trade-offs
- Include visual artifacts (mockups, screenshots) when relevant
- For iterative decisions, use numbered "Design Questions" (DQ1, DQ2...) to structure review
Best Practices
- Use subagents for parallel exploration to save context
- Write small scripts to explore APIs interactively
- Generate figures/diagrams and reference inline in markdown
- For web servers:
npx serve -p 8080 --cors --no-clipboard & - For screenshots: use Playwright MCP for web, Qt's grab() for GUI
- For external package API review: clone to
scratch/repos/for direct source access
Important: Scratch is Gitignored
The scratch/ directory is in .gitignore and will NOT be committed.
- NEVER delete anything from scratch - it doesn't need cleanup
- When distilling findings into PRs, include all relevant info inline
- Copy key findings, code, and data directly into PR descriptions
- PRs must be self-contained; don't reference scratch files
ISR Scientific Paper Writing, Pre-2026 Voice Alignment & RAE MyST Workflow (isr-paper-writing)
Use this skill whenever scaffolding, drafting, revising, critiquing, or auditing scientific manuscripts (isr single-column preprints and ieeetrans / conference submissions) across the Intelligent Sensing & Rehabilitation (ISR) research portfolio.
1. Grounding in R. James Cotton's Authentic Pre-2026 (Non-AI-Written) Voice
CRITICAL ANTI-ECHO-CHAMBER RULE: Never calibrate prose style, paragraph structure, or rhetorical transitions against 2026 manuscripts (
B3D,PBL,CAMBR,FASCIA,BiomechGPT, etc.), which were co-drafted or edited with AI agents. Aligning to AI-assisted papers creates a self-reinforcing echo chamber of bold-lead bullet lists, em-dash chains, and formulaic transitions. Always anchor voice alignment strictly to R. James Cotton's pre-2026 (2013–2025) human-written papers.
1.1 Querying the Pre-2026 Voice Corpus via Local Zotero API (http://localhost:23119/api)
Before drafting or editing any section, query the local Zotero HTTP API (available on the host and tunneled into every devcontainer at http://localhost:23119/api) and filter strictly to year <= 2025:
# Fetch R. James Cotton's pre-2026 (non-AI-written) publications from local Zotero
curl -s "http://localhost:23119/api/users/0/items?q=Cotton&limit=100" | python3 -c "
import json, sys, re
for it in json.load(sys.stdin):
d = it.get('data', {})
if d.get('itemType') in ('attachment', 'note'): continue
m = re.search(r'(20\d\d)', d.get('date', ''))
yr = int(m.group(1)) if m else 0
if 2013 <= yr <= 2025:
print(f\"{it['key']} ({yr}) [{d.get('publicationTitle') or d.get('proceedingsTitle') or 'Preprint'}]: {d.get('title')}\")
"
# Export BibTeX directly for citation keys
curl -s "http://localhost:23119/api/users/0/items?q=Cotton&format=bibtex"
Canonical Pre-2026 Human-Written Exemplars (2013–2025)
| Domain | Zotero Key (Year, Venue) | Title & Full-Text Reference Location (if in /workspaces/Notes/Manuscripts/ or /workspaces/Manuscripts/) |
|---|---|---|
Probabilistic Computation & Structured Statistics (2013–2021) | 23SPKLY6 (2013, PNAS)E22PCNTX (2015, PLoS Comp Biol)J5B3JRRF (2020, Nature Neurosci)VF2QWPFD (2021, Nature Comms) | Trial-to-trial, uncertainty-based adjustment of decision boundaries Improved Estimation and Interpretation of Correlations in Neural Circuits (sparse + low-rank covariance) A neural basis of probabilistic computation in visual cortex Revealing nonlinear neural decoding by analyzing choices |
Clinical Rehab, Outcomes & Precision Rehab (2020–2024) | SE8LVGCI (2020, IEEE BioRob)EUWZZ49R (2021, Sci Reports)GAMD4ML8 (2023–2024)8TGRW2I6 (2024, Sci Reports)RWKKTS5E (2024, JNER) | Smartphone Control for People with Tetraplegia by Decoding Wearable EMG Portable, open-source solutions for estimating wrist position during reaching in people with stroke A Causal Framework for Precision Rehabilitation ( 2023 - PrecisionRehab/Causal Framework for Precision Rehab.md — tag jner-submission-2025)Validation of portable in-clinic video-based gait analysis for prosthesis users ( MarkerlessMocapGaitRite)Transforming modeling in neurorehabilitation: clinical insights for personalized rehabilitation |
Markerless Mocap, Calibrated Pose & Differentiable Biomechanics (2020–2025) | 84ENQZZ6 (2022)L7F5RXTR (2022, Digit Biomark)6TWZYM4B (2022/2023)5MZ5V29K (2023)BYMFMV2C (2023, IEEE BHI)35LAPNK8 (2024)N2PK29BL (2024, IEEE BioRob)S3BSZYID (2024/2025, ICORR)PXQCMRJA (2025, EMBC) | PosePipe: Open-Source Human Pose Estimation Pipeline for Clinical Research Video-Based Pose Estimation for Gait Analysis in Stroke Survivors Optimizing MPJPE promotes miscalibration in multi-hypothesis human pose lifting Improved Trajectory Reconstruction for Markerless Pose Estimation Optimizing Trajectories and Inverse Kinematics for Biomechanical Analysis of Markerless Motion Capture Data ( MarkerlessMocapBiomechanics)Platypose: Calibrated Zero-Shot Multi-Hypothesis 3D Human Motion Estimation Advancing Monocular Video-Based Gait Analysis Using Motion Imitation with Physics-Based Simulation ( Biomechanical Follower)Differentiable Biomechanics Unlocks Opportunities for Markerless Motion Capture ( DifferentiableBiomechanics)Biomechanical Reconstruction with Confidence Intervals from Multiview Markerless Motion Capture ( BiomechanicalConfidenceIntervals) |
1.2 Five Stylistic Pillars of Pre-2026 James Cotton Prose (vs. AI-Written Slop)
Inspecting James's human-written manuscripts (2013–2025) reveals five concrete stylistic traits that distinguish his authentic voice from LLM-generated scientific prose:
-
Cohesive Narrative Paragraphs Over "Bold-Lead Bullet Lists":
- How James writes (pre-2026): Every section of the Introduction, Methods, Results, and Discussion is written in complete, flowing paragraphs where each paragraph develops one unified idea. Bullet lists appear only at the end of the Introduction for the 3–4 headline contributions (or when defining a formal taxonomy in a review paper).
- AI habit to strip: Never write Methods, Results, or Limitations as a sequence of
- **Bold Phrase:** sentence fragmentbullets or numbered pseudo-paragraphs. Convert bulleted body text back into connected narrative prose.
-
Plainspoken, Direct Scientific Motivation ("What We Truly Need"):
- How James writes (pre-2026): Before introducing any formalism, James states the practical clinical or measurement bottleneck in plain, unpretentious English:
- "However, this does not provide the information truly needed -- how accurate is a particular measurement with a novel camera configuration on the next current participant?" (
BiomechanicalConfidenceIntervals, 2025) - "These confidence intervals are much more useful than the quality metrics we previously used, such as the mean reprojection error. They directly reflect what we want - how accurately we are tracking joint angles and body segment positions at each moment of time." (
BiomechanicalConfidenceIntervals, 2025) - "We intentionally recruited a diverse rehabilitation population both to ensure the system was robustly validated for all types of patients, and because this is the population we intend to apply this final system to." (
MarkerlessMocapBiomechanics, 2023)
- "However, this does not provide the information truly needed -- how accurate is a particular measurement with a novel camera configuration on the next current participant?" (
- How James writes (pre-2026): Before introducing any formalism, James states the practical clinical or measurement bottleneck in plain, unpretentious English:
-
Pedagogical 3-Part Loss & Equation Introductions (Brief Intuition $\to$ Equation $\to$ Term Details):
- How James writes: Whenever introducing a loss function or optimization objective, follow a crisp 3-part rhythm:
- Brief intuitive explanation (1–2 sentences): State directly what the objective/loss achieves at a high level before displaying math (avoiding multi-paragraph storytelling preambles).
- Displayed equation: Present the clean equation, using
\underbrace{...}_{\text{...}}annotations where helpful so the reader can see the role of each term at a glance. - Details of the equation: Immediately after the equation, concisely explain the symbols and what each term does (moving low-level numerical/implementation minutiae to the Appendix).
- Example from pre-2026 prose:
- "The pose parameters predict the detected keypoint locations through two non-linear steps. First, the marker locations can be predicted in 3D space from the pose parameter using the forward kinematic model of the body, $\mathcal{M}_\beta: \boldsymbol{\theta} \rightarrow \mathbf{x}$... Next, these marker locations are projected into the image plane using the calibrated camera model, $\Pi_c: \mathbf{x} \rightarrow \hat{\mathbf{y}}^c$... Because $p(\mathbf{Y}, \mathbf{S})$ is a constant, we can maximize the Evidence Lower Bound (ELBO) to minimize the KL divergence:" (
BiomechanicalConfidenceIntervals, 2025)
- "The pose parameters predict the detected keypoint locations through two non-linear steps. First, the marker locations can be predicted in 3D space from the pose parameter using the forward kinematic model of the body, $\mathcal{M}_\beta: \boldsymbol{\theta} \rightarrow \mathbf{x}$... Next, these marker locations are projected into the image plane using the calibrated camera model, $\Pi_c: \mathbf{x} \rightarrow \hat{\mathbf{y}}^c$... Because $p(\mathbf{Y}, \mathbf{S})$ is a constant, we can maximize the Evidence Lower Bound (ELBO) to minimize the KL divergence:" (
- How James writes: Whenever introducing a loss function or optimization objective, follow a crisp 3-part rhythm:
-
Measured, Understated Empirical Claims (Zero Marketing Adjectives):
- How James writes (pre-2026): Quantitative improvements speak for themselves, framed with measured scientific restraint:
- "Our fairly low ECE scores suggest our confidence intervals are fairly well calibrated, meaning 95% of the time, the ground truth should fall within our 95% confidence intervals."
- "We also found some indications that it is less inclined to fall into local minima than our prior deterministic method..."
- "Fairly unsurprisingly, policies driven by keypoints from video produced greater tracking errors and errors for spatial gait parameters than policies driven by ground truth data."
- Banned AI hype vocabulary: Strip words like "paradigm-shifting", "unprecedented", "seamlessly", "crucially", "remarkably", "delves into", "testament to", "pivotal", "synergistic", "holistic", and "underscores".
- How James writes (pre-2026): Quantitative improvements speak for themselves, framed with measured scientific restraint:
-
Candid Operational Transparency on Negative Results, Compute Costs & Model Mismatches:
- How James writes (pre-2026): Rather than hiding what failed or writing generic AI boilerplate (
"Future work will explore larger datasets"), James candidly reports negative experimental branches, real GPU wall-clock costs, and biomechanical model simplifications:- "We also implemented short horizon actor-critic in Brax as an optimizer, given the claim it can produce more precise tracking for differentiable environments. However, we found that it was extremely slow to train and could not obtain good policies." (
Biomechanical Follower, 2024) - "However, our method also has limitations. Optimizing the probabilistic representation is several times slower than our non-probabilistic implicit representation; for example, 10-20 trials can take 1-2 hours on an A100. Our synthetic testing also revealed that certain modeling discrepancies, such as errors in the calibration, can produce overly confident results by attributing too much uncertainty to the likelihood function." (
BiomechanicalConfidenceIntervals, 2025) - "We focused on only sagittal plane joint angles as we anticipated the vast differences between our simplified humanoid model and the accurate biomechanical model used in the mocap data would be too significant of a model mismatch." (
Biomechanical Follower, 2024)
- "We also implemented short horizon actor-critic in Brax as an optimizer, given the claim it can produce more precise tracking for differentiable environments. However, we found that it was extremely slow to train and could not obtain good policies." (
- How James writes (pre-2026): Rather than hiding what failed or writing generic AI boilerplate (
Quick Reference: AI-Drafted Slop vs. Authentic Pre-2026 James Cotton Voice
| Dimension | AI-Drafted Slop (Rewrite Immediately) | Authentic Pre-2026 James Cotton Prose |
|---|---|---|
| Body Structure | Wall of - **Bold Header:** sentence bullets in Methods, Results, and Limitations | Cohesive narrative paragraphs; bullets reserved for the 3–4 Intro contributions |
| Punctuation | Heavy em-dash chains (— ... —) and rhetorical colon headers (Key Insight: ...) | Clean declarative sentences, natural subordinate clauses (although, while, because) |
| Math Presentation | Giant 7-term $\mathcal{L}_{\text{total}} = \sum \lambda_i \mathcal{L}_i$ dump + symbol list | Step-by-step inline prose walkthrough of the forward model $\to$ why naive inference fails $\to$ compact objective |
| Tone & Claims | "Crucially, our novel framework seamlessly achieves unprecedented accuracy..." | "This end-to-end approach produced lower reprojection errors than a traditional multistage pipeline..." |
| Limitations | Vague boilerplate ("Future work will expand to larger cohorts and real-time deployment") | Specific physical/compute candor ("10–20 trials take 1–2 hours on an A100; calibration errors can produce overconfident posteriors by inflating the learned likelihood width") |
1.3 Strict Ban on Internal Code, Config, & Lab-Notebook Jargon
Never leak internal codebase identifiers, run tags, checkpoint filenames, or config flags into manuscript prose, tables, or figure labels:
| Banned Internal Jargon | Clean Scientific Terminology |
|---|---|
O_v48, O_v48_s0, v4.7, fits_v47 | Pointwise Regression (FASCIA-Pointwise), CT registration archive |
whole_v47_c6p, c6b, flow_calib | Conditional Flow Matching (FASCIA-Flow) |
humanoid_simple_arm_20261021.xml, correspondence_20261026 | CT-calibrated anatomical skeleton, calibrated surface-to-skeleton correspondence model |
"CT skeleton" (when applied to non-CT subjects like amputees or gait patients) | Anatomically calibrated reference skeleton (never imply participants had CT scans when they did not!) |
dualhead_C_20261026, pseudoanchor_scratch_tgoff | Unified dual-head multi-view transformer, online pseudo-anchor supervision |
carry3sh, carry3shac, dvltqft | Carried Gaussian decoder with spatial-hash query selection, velocity-continuity regularization |
NLF v0.2.0 vs. NLF v0.3.2 | Report only one NLF baseline (the faithful port of the original NLF paper, called simply NLF) alongside MAMMA and CSE |
unified_v4_31b_grpo_perc_nf_v2, c0 + c1 * z, M5 b3d | GRPO reinforcement learning with verifiable kinematics rewards, affine calibration, monocular mesh baseline |
2. Canonical RAE MyST Paper Template (templates/myst-paper/)
Every ISR manuscript across all repositories must be authored in MyST Markdown (mystmd) using the canonical template in reproducible_agent_environment/templates/myst-paper/ (also mirrored in .agents/skills/isr-paper-writing/templates/myst-paper/). Never hand-roll standalone LaTeX .tex article files.
2.1 Scaffolding a New Paper in One Command
# From any repository or devcontainer with RAE installed:
/path/to/reproducible_agent_environment/templates/myst-paper/scripts/scaffold_myst_paper.sh \
docs/papers/<paper_slug> \
"Full Jargon-Free Paper Title" \
"<paper_slug>"
2.2 Canonical Directory Layout (docs/papers/<paper_slug>/)
docs/papers/<paper_slug>/
├── myst.yml # Project config, Eq. %s numbering, references.bib, toc
├── manuscript.md # Dual ISR + IEEE export frontmatter + modular {include} directives
├── abstract.md # Structured Background / Methods / Results / Conclusions abstract
├── introduction.md # 4-paragraph clinical/scientific hook -> gap -> unified method -> contributions
├── methods.md # Best unified method first, with <!-- CODE: file:line --> anchors
├── results.md # Primary benchmark -> physical/clinical validation -> ablations that mattered
├── discussion.md # Synthesis + forthcoming "Limitations and Future Directions" subsection
├── acknowledgments.md # Funding, cohort, and mandatory AI usage acknowledgment
├── appendix.md # Exhaustive per-bone/per-joint tables, derivations, sensitivity grids
├── references.bib # Populated via local Zotero API (http://localhost:23119/api)
├── BUILD.md # Git commit provenance table + exact reproduction commands
├── build_isr.sh # MyST -> XeLaTeX + Biber -> 2x XeLaTeX + unresolved-citation gate
├── build_ieee.sh # MyST -> pdflatex + Biber -> 2x pdflatex + unresolved-citation gate
├── scripts/
│ ├── generate_paper_figures.py # Deterministic Tufte serif figure generator
│ └── diff_commits_pdf.py # Word-level redline PDF diff between any 2 Git commits/tags/WORKTREE
└── templates/
├── isr/ # isrpaper.cls, isr-logo.pdf, template.tex, template.yml (single/two-col, orange/blue)
└── ieeetrans/ # IEEEtran.cls, template.tex, template.yml
2.3 Key Build & Provenance Rules in the RAE MyST Template
- Modular
{include}Architecture (manuscript.md):- Keep
manuscript.mdas a thin frontmatter +{include}spine (abstract.md,introduction.md,methods.md,results.md,discussion.md,acknowledgments.md,appendix.md). If a conference submission requires a separate supplementary PDF (manuscript_conf.md+supplementary.md), both include the exact same section.mdfiles so numbers and text can never drift between ISR and conference builds.
- Keep
- Preventing MyST
.bibCulling (cp references.bib main.bib):myst build --texwrites a culledmain.bib. Bothbuild_isr.shandbuild_ieee.shautomatically overwritemain.bibwithreferences.biband copytemplates/isr/{isrpaper.cls,isr-logo.pdf}into the working directory before runningxelatex+biber+xelatex$\times 2$, then grep the.logto hard-fail on any undefined citation (?).
- Code-to-Paper Provenance Anchors (
<!-- CODE: path:Lstart-Lend -->) &BUILD.md:- Every architectural claim, loss term, and evaluation metric in
methods.mdandresults.mdshould carry an HTML comment anchoring it to the exact source file and line range (<!-- CODE: src/pkg/model.py:L120-L155 -->), andBUILD.mdmust record the git commit SHA and exact CLI invocations that produced every table and figure.
- Every architectural claim, loss term, and evaluation metric in
3. Organizational Strategy: "Best Method First $\to$ Ablations That Mattered"
Never organize a paper as a chronological lab notebook ("First we tried X, then we fixed bug Y, then in v3 we tried Z..."). Structure every manuscript around the strongest unified method first, followed by targeted ablations and lesions that isolate why each component is necessary:
- Introduction (
1–1.25pages, 4–5 narrative paragraphs):- Paragraph 1 (Clinical / Scientific Need): What measurement or control capability does rehabilitation or movement science need, and why is it inaccessible at scale?
- Paragraph 2 (Why Existing Silos Fall Short): Contrast traditional laboratory optical mocap / CT imaging (accurate but restricted) against commodity computer vision pose/mesh estimators (scalable but biomechanically uncalibrated, lacking joint-axis fidelity, calibrated uncertainty, or physical consistency).
- Paragraph 3 (Core Insight & Unified Method): Walk through how our unified formulation bridges this gap in plain, concrete terms.
- Paragraph 4 (Contributions & Validation Summary):
3–4crisp bullets summarizing the method, the validation cohort, and the headline findings. - Strict Introduction Hygiene Rules (No Unintroduced Acronyms/Math & No Results Data Dumps):
- Spell Out Every Acronym on First Use: Never use an acronym (e.g.,
LOSOfor leave-one-subject-out,SynX,FSM,CMT) before defining it in full on first occurrence in the text. - Zero Unintroduced Mathematical Notation in the Introduction: Never drop un-introduced parameter symbols or subscripted math ($\uparrow\kappa_{\text{SOL,GAS}}^V$, $\downarrow\alpha_{\text{SOL,GAS}}^{\text{strength}}$, $\theta_{\text{reflex}}$) into the Introduction. Explain physiological and algorithmic mechanisms intuitively in plain prose in the Introduction, and introduce formal parameter vectors ($\theta \in \mathbb{R}^P$) sequentially in the Methods where the circuit/model is defined.
- No Quantitative "Data Dumps" in the Introduction: You may preview the study design and state the high-level conclusions in the Introduction, but never dump laundry lists of correlation coefficients ($r=0.682\text{--}0.804$), $p$-values, or unmotivated muscle-specific breakdowns (e.g., naming individual deep muscles before the reader knows why they matter). Save detailed quantitative numbers for the Results section.
- No Overly Busy Diagrams as Figure 1: Keep Figure 1 focused and immediately legible as a high-level paper overview; move dense multi-module wiring/block diagrams to the Supplementary or graphical abstract.
- Spell Out Every Acronym on First Use: Never use an acronym (e.g.,
- Methods (Lead with What We Actually Did — Never Belabor What We Didn't Do):
- Present the complete, best-performing formulation as a cohesive pipeline from inputs to outputs in flowing narrative paragraphs.
- Make the Loss, Targets, & Stage Rationale Unmistakable to a Naive Reader: Always state explicitly what the parameter vector $\theta \in \mathbb{R}^P$ controls before differentiating with respect to it, what exact loss function is being minimized, and what measured signals are being fit. Whenever a multi-stage procedure is used (e.g., teacher-to-student distillation), state up front why the first stage cannot be deployed directly at test time (or, if it can, apply Occam's Razor and remove the extra stage). When synthetic validation data is used, explain in Methods how synthetic controllers/trials are parameterized so the validation is falsifiable and non-circular.
- Do Not Belabor Pruned Pathways or Replaced Precursors in Main Methods: Never spend main-text Methods paragraphs or equations explaining pathways we pruned (e.g., collinear static spindle length feedback $\kappa_i^L$), legacy four-sphere contact models we replaced, or speculative mono- vs. bi-articular arguments we did not experimentally test. Focus the main Methods 100% on what we actually did; mention pruned components in 1–2 sentences of prose in the Discussion and relegate collinearity/Fisher pruning math to
appendix.md. - Distinguish Native Simulator Physics vs. Custom Intercessions: Clearly state what is provided natively by the physics engine (e.g., standard MJX contact/COP readout used strictly as a validation signal) versus where we interceded with custom biomechanics (e.g., a custom muscle or reflex model, which should be backed by an ablation).
- Results (Standard
# ResultsHeading: Primary Benchmark $\to$ Physical/Clinical Validation $\to$ Ablations & Lesions):- Always use the standard filename
results.mdand top-level heading# Results(not# Experiments and Resultsorresults_and_lean.md) so MyST Studio parses section keys cleanly. - 3.1 Primary Head-to-Head Comparison: Compare our unified method against established external baselines.
- 3.2 Physical, Anatomical, or Clinical Validation: Validate against independent gold-standard measurements (CT holdout scans, dual AMTI forceplates, C3D optical markers, whole-body DEXA, GaitRite, or clinical scales).
- 3.3 Ablation & Lesion Studies ("What Mattered?"): Systematically remove or swap key components (or compare against simpler formulations tested during our simplification cycle) so the reader sees empirically why each component earns its keep.
- Always use the standard filename
- Discussion & Candid Limitations (
Limitations and Future Directions):- Include a dedicated, concrete subsection candidly exposing: (a) physical observability limits (what cannot be inferred from the modality alone), (b) training data / field-of-view coverage gaps (e.g., truncated distal forearms in clinical trunk CTs, adult body priors on infants), (c) biomechanical/sensor simplifications & compute costs (e.g., rigid prosthetic foot segments, optimization wall-clock time), and (d) how each limitation directly motivates the next step.
3.1 Direct, Punchy, Point-Driven Prose & The "3–4 Main Points Spine" (Zero Throat-Clearing)
AI drafts often eliminate bullet lists only to replace them with winding, 45-word multi-clause paragraphs that bury the point in passive throat-clearing. R. James Cotton's authentic pre-2026 (<= 2025) prose is direct, punchy, and tightly organized around a small number of explicit points:
- Define the 3–4 Main Points Before Writing (
methods_outline.md):- Every paper must state its 3–4 core points (punchlines) at the top of
methods_outline.md, paired with: (a) the exact dataset, subject/trial count ($N$ participants / $n$ trials), and fraction of the full cohort vs. pilot data used, (b) the primary metric/effect size supporting the point, and (c) the specific figure that visually proves it. - Frame engineering papers around an empirical discovery arc: "Before starting this study, we did not know how to do $X$ optimally (or whether $A$ was necessary over $B$); through targeted experiments and ablations, we now know $Y$."
- Every paper must state its 3–4 core points (punchlines) at the top of
- One Point Per Paragraph (Topic Claim $\to$ Mechanism/Evidence $\to$ Punchy Takeaway):
- Sentence 1 (Direct Point, $<22$ words): Open every paragraph with a direct, active-voice claim or question. Cut all throat-clearing ("In order to address the aforementioned challenges associated with...", "It is important to note that within the context of...").
- Bad (Diffuse): "In order to mitigate the sensitivity of monocular 3D human mesh recovery to partial clinician-hand occlusions during clinical spasticity assessments, we incorporate an uncertainty-aware canonical landmark weighting scheme."
- Good (Direct & Punchy): "Clinician hands frequently occlude the patient's forearm during passive stretch exams. Weighting each canonical landmark by its predicted Laplace uncertainty prevents occluded vertices from corrupting the elbow angle trajectory."
- Middle Sentences: Walk through the math or empirical numbers step by step without stacked participial tails ("..., leveraging X, thereby enabling Y, while simultaneously mitigating Z").
- Final Sentence: State the concrete takeaway or transition cleanly to the next step.
- Sentence 1 (Direct Point, $<22$ words): Open every paragraph with a direct, active-voice claim or question. Cut all throat-clearing ("In order to address the aforementioned challenges associated with...", "It is important to note that within the context of...").
- Ban Unnatural Framework Buzzwords, Inline Number Dumps in Prose, & Walls of Tables (
ICK!):- Never invent pompous structural labels like "Three-Pillar Architecture", "Dual-Paradigm Synergistic Framework", or "Four-Tiered Holistic Pipeline". Describe what the code does in plain biomechanics and optimization language.
- Put Numbers in Programmatically Generated Tables (and Figures), Not Inline Prose Dumps: There should not be tons of numbers in running text—most quantitative results belong in compact, programmatically generated tables (generated by committed scripts in
docs/papers/<paper_slug>/scripts/from committed evaluation JSONs) or data-driven figures. Running prose should focus on the scientific insight, physical mechanism, and what the comparison proves (referencing{numref}\tab-xyz`/{numref}`fig-xyz`, with at most a single headline anchor number where helpful), rather than stringing parenthetical metric dumps ((RMSE = X ± Y, r = Z, p = W vs. A ± B)`) through every sentence. This guarantees that numbers are programmatically generated and stay 100% synchronized while producing much cleaner, more readable prose. - Distinguish True Scientific Ablations From Trivial Bug Fixes / Early Suboptimal Choices: Not every iteration needs to be an ablation table or figure. Reserve formal ablations strictly for core architectural components where high-quality quantitative metrics are needed to prove why our final algorithm requires that component. Early engineering lessons or suboptimal parameterizations with hindsight (e.g., that an early oversized
bofoot.stlfoot mesh was bad) belong as at most a single concise sentence in the Discussion, never a dedicated ablation table. - 100% Locked-Down
methods_outline.md& Zeroscratch/Code Dependencies: Every paper must maintain a 100% locked-downmethods_outline.md(with exactfile:linepointers, formulas, hyperparameters, and inputs/outputs matching the manuscript and production code). All code discussed in the manuscript must be migrated out ofscratch/into committed production modules (src/) or intodocs/papers/<paper_slug>/scripts/(if it only drives paper-specific evaluations, tables, or figures) and committed to git. - No Walls of Tables: Main-text tables must be compact ($\le 6\text{–}8$ rows, $\le 5\text{–}6$ columns) and answer one focused question. If a table exceeds 8 rows or 6 columns, convert the core comparison into a Tufte figure (slope graph, forest plot, or complexity ladder) and move the exhaustive breakdown to
appendix.md.
- Canonical Fleet Terminology & Intellectual Lineage:
- Use
CSE(CanonicalSapiensEqx) strictly for the vision backbone that predicts dense canonical surface queries and uncertainties, andCAMBRstrictly for the bilevel monocular and multi-view biomechanical reconstruction method that optimizesMHRmeshes andBodyModelsskeletons from canonical queries. - Ground
CAMBRin the intellectual lineage of our prior differentiable biomechanics (MuJoCo/bilevel IK) andPBL(Probabilistic Body Localization) papers: keypoints are robust to appearance shifts, and optimizing a biomechanical skeleton through a bilevel framework yields substantially higher anatomical accuracy than feed-forward mesh regression (B3D) alone.
- Use
3.2 Point-Driven Visual Cadence: Overview (TikZ) $\to$ Qualitative Biomechanical Overlays $\to$ Tufte Quantitative & Occam's Razor Figures (Flexible, Non-Rigid)
Every figure in a manuscript must exist to prove one of the paper's main points to a reader who skims all figures before reading the text. Often in our papers it makes sense to follow a 3-tier visual cadence—while keeping this pattern explicitly non-rigid whenever a paper's specific claims align better a different way:
- Tier 1 — Method / Architecture Overview Figure (Often Standalone TikZ / Vector Schematic + Optional Mini-Renders):
- Show the conceptual and mathematical pipeline at a glance (
Inputs -> Canonical Queries / Latent Representations -> Physics / Optimization -> Outputs), explicitly contrasting the Primary Best Pathway against any Ablation Pathway with inline loss symbols ($\mathcal{L}{\text{NLL}}$, $\mathcal{L}{\text{prior}}$). - Exemplars:
fig1_cambr_architecture_pathways.png(2-pane CAMBR vector schematic contrasting Pathway A vs. Ablation Pathway B),fig_fascia_flow_architecture_and_samples.png(Panel a Eulerian–Lagrangian flow diagram paired with anatomical samples), andbga_fig1_architecture_overview.png. - Hard Rule: Build pipeline overviews in TikZ/vector PDF or tightly audited SVG—never ship boxy matplotlib rectangle diagrams with clipped text or arrow collisions (avoid right-edge text clipping).
- Show the conceptual and mathematical pipeline at a glance (
- Tier 2 — Qualitative Biomechanical Overlay & Physical Convincer Figure:
- Immediately show that the method works visibly on real movement data across views, occlusions, or clinical conditions.
- Mandatory Visual & Geometric Standards:
- Camera Distortion Correction (
multi_cameraJAX): Project 3D meshes, skeletons, and wheels using full OpenCV radial/tangential camera distortion coefficients ($[k_1, k_2, p_1, p_2, k_3]$ and exact principal point $(c_x, c_y)$) viamulti_camera's high-performance JAX camera model—and/or undistort the video images first—so overlays align pixel-accurately out to image edges. - Standard GPU Cycles Mesh-Over-Skeleton Recipe: Render the native forward-kinematics
MHRskin as a semi-transparent pearl envelope (alpha = 0.28–0.32,roughness = 0.45), internal.stlbones with Bilateral Bone Hue Coding (Left = Steel-Blue #7FA8D4,Right = Coral-Red #D98880,Axial/Midline = Warm Ivory #E8E2D5), and CT-calibrated skull offset (skull_offset_from_mhr+gmat). Never use frame-0 posed-spacecKDTreeproximity skinning. - Three-Gate PHI & Publication-Image Policy (
sheprd-publication-consentskill): (1) Verify REDCap publication-image consent (recording == "agree") via Sheprd $\to$PosePipeAgent / 1 (eqx wK:p1)(redcap_consentskill, cached in<repo>/data_cache/publication_image_consent.json, or use public-consentASBparticipants; never includedisagree/not_answeredclinical participants); (2) blur all visible faces even for consented participants (MHR3D cranial projection); and (3) never commit participant images outside the specific publication figures directory (docs/papers/<slug>/figures/ordocs/manuscript/figures/). - Time-Locked Waveforms & Insets: Pair image overlays with 3D studio views, high-magnification anatomical insets (e.g., $2.6\times$ hand/finger or ribcage zoom), and synchronized kinematic/GRF/EMG/reflex waveforms that never crop at panel boundaries.
- Camera Distortion Correction (
- Exemplars:
tremor_multiview_finger_overlay_keyframes.png,hop_test_triple_hop_skeleton_image_overlay_keyframes.png,fig_two_person_cse_mamma_contact.png, andfig_fascia_flow_architecture_and_samples.png(Panels b–d).
- Tier 3 — High-Density Tufte Quantitative Result & Occam's Razor Ablation Figures:
- Present quantitative validation and complexity ablations with transparent background (
transparent=True), serif typography (Palatino/GT Sectra Fine/TeX Gyre Pagella+stixmath,pdf.fonttype = 42), despined axes, direct on-canvas labels, subject-level paired trajectories (Npatients vs. trials to prevent pseudoreplication), and an explicit Occam's Razor Complexity Ladder whenever a simplification/ablation was tested. - Exemplars:
fig_ssrp_occams_razor_ablation.png(M1–M5 complexity ladder showing CV $r$, BIC minimum, and paired $d_z$),fig_real_stroke_vs_nonstroke_ssrp.png(6-panel paired clinical validation), andfig1_clinical_gaitlab_emg_grf.png(8-panel EMG/GRF/CoP validation).
- Present quantitative validation and complexity ablations with transparent background (
- Multi-Contrast Color Scheme Templates Across Manuscript PDF (
tufte-print), Companion Interactive 3D Viewers (dark-studio), and Conference-Room TV / Slides (bright-room-tv&sralab-blue-deckinpasture/config/viz_color_themes.json):- Use
tufte-print(transparent=True,#111111primary ink,#333333secondary ink, Left =#1655a1, Right =#c0392b, Measured =#0b8e93) for manuscript.svg/.pdf/.pngfigures. - Whenever generating companion interactive 3D HTML visualizers (
slot1_*.htmlmodeled ongaitlab/slot1_gaitlab_m5_flagship_viewer.htmlandhop_test/slot1_hop_test_m5_viewer.html) or Deckwerk presentations, wire the 1-click theme switcher (🌙 Dark|☀️ TV Bright|🏛️ SRALab Blue) usingpasture/config/viz_color_themes.jsonand the June 2026 Shirley Ryan AbilityLab Brand Guidelines (PMS 2145 Deep Blue #1655a1,PMS 185 Red #ed1c2c,PMS 1505 Orange #f36c21,Custom Dark Teal #0b8e93,PMS Warm Gray 11 #6e6259). Inbright-room-tvmode, strictly ban low-contrast#64748b/#94a3b8grays, use3.0 pxplot strokes, and switch the 3D scene floor to warm alabaster (#f8fafc -> #dbe4ee) to eliminate conference-room TV mirror glare.
- Use
- Non-Rigidity Guardrail:
- Never force a figure that does not serve the paper's core points. If a paper's 3–4 main points do not align with a TikZ overview + qualitative overlay sequence, adapt the figure architecture directly to the points being communicated.
4. Multi-Cycle Paper Development & Improvement Workflow (Cycles 0–5)
We develop and polish papers through an iterative, six-stage loop that tightly couples pre-2026 voice alignment, code-to-paper verification, and experimental simplification:
Cycle 0 (RAE MyST Scaffold & BUILD.md Provenance)
│
▼
Cycle 1 (Pre-2026 Zotero Voice & Literature Grounding — year <= 2025 only)
│
▼
Cycle 2 (Round-1 Three-Subagent Review Triad: Code/Jargon + Pre-2026 Voice + Peer-Reviewer/Simplification)
│
▼
Cycle 3 (Experimental Occam's Razor Loop: Methods Smell-Test -> Fast GPU Simplification Ablation)
│
▼
Cycle 4 (Round-2 Deep Audit: Code/Eval Bugs Exposed by Writing, Participant-Level DoF & Label-Blindness)
│
▼
Cycle 5 (Tufte Serif Figures, Dual ISR/IEEE PDF Build, Visual Page Inspection & Redline Diff PDF)
Cycle 0 — Scaffold from RAE templates/myst-paper & Stamp Provenance
- Initialize
docs/papers/<paper_slug>/usingtemplates/myst-paper/scripts/scaffold_myst_paper.sh. - Record the baseline git commit SHA, branch, and evaluation script commands in
BUILD.md.
Cycle 1 — Pre-2026 Voice Calibration & Mandatory Zotero Library Grounding (127.0.0.1:23119)
- Query
http://localhost:23119/api/users/0/items?q=Cotton&limit=100filtered toyear <= 2025and read 1–2 closest pre-2026 human-written manuscripts by R. James Cotton (e.g.,BiomechanicalConfidenceIntervals.md,Differentiable Biomechanics.md,Biomechanical Follower.md,Causal Framework for Precision Rehab.md) before drafting or rewriting prose. - Mandatory Zotero Library Grounding & Unambiguous Zotero Item Key (
zotero_key):- Every reference cited in our manuscripts must be grounded in the user's Zotero library (
http://localhost:23119/api/users/0/items). - Populate
references.bibdirectly from Zotero (http://localhost:23119/api/users/0/items/<ITEM_KEY>?format=bibtex) and include the unambiguous 8-character Zotero item identifier on every.bibentry (zotero_key = {<ITEM_KEY>}) so every reference can be deterministically audited against Zotero (curl -s "http://localhost:23119/api/users/0/items/<ITEM_KEY>"). - If a relevant external paper is not yet in Zotero, you must: (1) verify that the paper is 100% real via Crossref / PubMed / Semantic Scholar / arXiv (confirming exact DOI/PMID/arXiv ID, authors, title, venue, and year—never guess or synthesize BibTeX fields), (2) tag the entry with
zotero_status = {pending_user_import}+ verifieddoi = {...}, and (3) explicitly report the verified reference + DOI/URL to the user so they can add it to Zotero.
- Every reference cited in our manuscripts must be grounded in the user's Zotero library (
Cycle 2 — Round-1 Three-Subagent Review Triad
Spawn three parallel subagents with non-overlapping responsibilities:
- Subagent 1 — Code-to-Paper, Statistical-Unit & Jargon Auditor:
- Verifies every equation, symbol, sample size $N$, and table number against the exact production Python scripts (
<!-- CODE: file:line -->) and JSON evaluation outputs. - Strips all internal run tags (
O_v48,fits_v47,carry3sh,"CT skeleton"on non-CT cohorts) and checks that external baselines are faithfully represented. - Code-Complexity Smell Test: Flags any Methods passage where describing the code requires listing multi-stage heuristics, magic thresholds, or unablated sub-modules.
- Verifies every equation, symbol, sample size $N$, and table number against the exact production Python scripts (
- Subagent 2 — Pre-2026 James Cotton Voice, Narrative Flow & Candid Limitations Editor:
- Explicitly compares the draft against James's pre-2026 (
<= 2025) human-written papers (never 2026 AI drafts). - Converts
- **Bold Header:**bullet-list slop in Methods, Results, and Discussion back into cohesive narrative paragraphs. - Rewrites cold equation dumps into step-by-step inline prose introductions, strips AI hype adjectives, enforces "Best Method First $\to$ Ablations That Mattered", moves secondary tables to
appendix.md, and ensures the Limitations section is candid and specific.
- Explicitly compares the draft against James's pre-2026 (
- Subagent 3 — Adversarial Peer-Reviewer, Simplification Experiment Designer & Figure-to-Point Critic:
- Visually inspects every figure and rendered PDF page via
read_fileto verify that each figure proves one unmistakable scientific claim with zero text overlaps or aspect-ratio distortions. - Simplification Mandate: For every complex component or multi-stage rule in Methods, asks: "Have we tested whether a simpler, single-objective formulation works just as well or better?" and designs a fast GPU ablation (
15-Minute Rule).
- Visually inspects every figure and rendered PDF page via
Cycle 3 — The Experimental Occam's Razor Loop ("Don't Guess—Test It")
- Methods-Drafting as an Architectural Smell Test:
- When a Methods paragraph feels clunky or apologetic (
"First we apply heuristic A with threshold $\tau_1$, then for stationary tracks we switch to heuristic B..."), treat it as an architectural smell in the code, not merely a writing problem.
- When a Methods paragraph feels clunky or apologetic (
- Run the Fast GPU Simplification vs. Complexity Experiment:
- We are not dogmatically anti-complexity—we are against untested, unearned complexity.
- Implement the cleaner single-objective formulation alongside the complex one and run a targeted head-to-head GPU evaluation.
- Promote on Win/Tie, or Document as an "Ablation That Mattered":
- If the simpler formulation matches or beats the complex one: Promote the simpler code to production, delete the dead heuristics, and present the clean formulation as our primary Method.
- If the richer formulation wins significantly: Keep it in the primary Method and report the simpler baseline in the Ablation Study ("What Mattered?") subsection of Results so every piece of complexity is empirically justified.
Cycle 4 — Round-2 Deep Verification & Anti-Lab-Notebook Clutter Purge (Bugs & Administrative Clutter Exposed by Writing)
Writing the paper alongside the code routinely exposes both subtle evaluation bugs and "lab-notebook administrative clutter" that would never appear in a peer-reviewed journal article. Before finalizing any manuscript (or its supplement), run a dedicated Round-2 verification pass checking:
- Statistical Unit of Analysis & Degrees of Freedom: Never treat repeated trials, strides, or frames from the same participant as independent samples in $t$-tests or $p$-values (e.g., 59 trials across $N=12$ participants must use participant-level paired statistics or mixed-effects models with $df = N - 1$, never trial-level $df = 58$).
- Strict Label-Blindness at Inference: Verify in code that no evaluation metric or test-time inference path leaks ground-truth clinical labels, ground-truth body mass/height (when evaluating video-only prediction), or post-hoc per-trial bias subtraction without calling it out explicitly.
- Metric & Baseline Parity: Verify that baselines and our model are evaluated on the exact same frames, coordinate conventions (
KAD1-IMRvs.CMR), and unit scales (mmvs.cm). - Gate 5 — Mandatory Anti-Lab-Notebook & Irrelevant Administrative Detail Purge (Main Text AND Supplement):
- Zero Raw Calendar Visit Dates (
YYYY-MM-DD) or Session Logs: Never include tables or prose listing raw calendar visit dates (e.g.,2024-07-03, *2024-07-17*,TF70 2025-03-20... held only clinical tests) in either the main text or the supplement. Report clinical protocol intervals in elapsed days/weeks/months (e.g.,"following a 6-week home acclimation period") and aggregate cohort flow counts ($N$ participants, $n$ visits, $n$ walking trials). - Zero Internal Recording IDs, Filenames, Commit Hashes, or Checkpoint Codenames: Strip raw trial recording hashes (
20240915_112405_cam01,ps9n02cm,ASB_001), git commit hashes (4ebfbe9), CLI flags (--use-v49), and internal run tags (O_v48_s0,fits_v47,carry3sh,Method-137) from all prose, tables, and figure labels. Use clean participant pseudonyms (P1–P6orTF1–TF6) and descriptive scientific names. - Zero Data-Cleaning / QA Diary Notes: Replace narrative data-wrangling logs (
"Two trials had no accepted strides because the participant stood still, and three sessions had no multi-camera capture...") with a single standard CONSORT-style cohort accounting sentence in Methods/Results ("From 79 recorded walking trials across 6 participants, 72 met steady-state gait screening criteria (910 strides total)."). - Mandatory Pre-Build Automated Regex Linter: Before compiling any ISR or IEEE PDF (
./build_isr.sh/./build_ieee.sh), run this automated regex check across all manuscript.mdfiles and verify zero offending matches:grep -rnE \ '(\b202[0-6]-[01][0-9]-[0-3][0-9]\b|[0-9]{4}_202[0-6][01][0-9][0-3][0-9]_[0-9]{6}|JLH000[0-9]{4}|Method[ -]137|fits_v[0-9]+|O_v4[0-9]|carry3sh|dvltqft|tf_newbase|_2026[01][0-9][0-3][0-9]\.(stl|npz|xml|json)|\b(TF|TR|KD|P0|p)[0-9]{2,3}\b|\*Status:\* PLANNED|\bTODO\b|\bTENTATIVE\b)' \ docs/papers/<paper_slug>/*.md
- Zero Raw Calendar Visit Dates (
- Gate 6 — Cross-Portfolio De-Duplication ("One Canonical Home Per Method"):
- Every core mathematical formulation (
ProbabilisticReconstructionTBME 2026 pointwise Laplace marginals,FASCIACT/MRI anatomy &MhrFkOpt,CAMBRmulti-view front-end &SequenceIK,GaitLabDatasetPointwise Probabilistic + Implicit Kinetic RolloutM5) has one primary owner manuscript. - Downstream or clinical papers (
ProstheticComponentry,Spasticity,CUET,Tremor,WheelchairPropulsion,SARA-PULSE,KinTwin) must cite the upstream foundation paper in 2–4 concise sentences and focus 100% of their Methods space on their domain-specific methodological extension—never re-deriving shared pipeline equations across multiple papers.
- Every core mathematical formulation (
- Gate 7 — Mandatory AI Usage Acknowledgment (
acknowledgments.md):- Every 2026+ ISR manuscript must include an explicit AI usage statement in
acknowledgments.md(or the Acknowledgments section ofmanuscript.md/sections/acknowledgments.md) acknowledging the use of Claude (Anthropic) and Antigravity for code development, running experiments, and manuscript preparation, and stating that the manuscript was extensively reviewed by the authors:"AI-assisted coding and research agents, including Claude (Anthropic) and Google Antigravity, were used for code development, running experiments, and assistance with manuscript preparation. The manuscript was extensively reviewed and verified by the authors, who take full responsibility for its content."
- Every 2026+ ISR manuscript must include an explicit AI usage statement in
- Gate 8 — Mandatory Zotero Reference & Identifier (
zotero_key) Audit:- Every
.bibentry cited in the manuscript must carryzotero_key = {<8-char-Zotero-key>}verified againsthttp://localhost:23119/api/users/0/items/<ITEM_KEY>. Any citation not yet in Zotero must be verified against Crossref/PubMed/arXiv for real DOI/metadata, taggedzotero_status = {pending_user_import}, and surfaced in a clean table of DOIs/BibTeX for the user to import into Zotero.
- Every
- Gate 9 — Mandatory IEEE Font Compliance (
pdf.fonttype = 42, Zero Type 3 Bitmap Fonts, &pdffontsAudit):- IEEE Xplore / PDF eXpress strictly rejects Type 3 bitmap fonts (which Matplotlib emits by default when
pdf.fonttype = 3) as well as non-embedded fonts (emb = no). - Every Matplotlib figure script must explicitly configure:
plt.rcParams.update({ "pdf.fonttype": 42, # Embed TrueType (Type 42) outlines — NEVER Type 3 (3) "ps.fonttype": 42, # Embed TrueType in PostScript/EPS "svg.fonttype": "none", # Keep text as <text> elements using standard installed serif fonts "font.family": "serif", "font.serif": ["Times New Roman", "Liberation Serif", "DejaVu Serif"], "mathtext.fontset": "stix", # STIX vector math glyphs (avoids Type 3 Computer Modern bitmaps) "text.usetex": False, # Avoid dvipng/Type 3 TeX bitmap fallbacks }) - Both
./build_isr.shand./build_ieee.shmust runpdffonts <output>.pdfafter compilation and fail the build if any font hastype == "Type 3"oremb == "no".
- IEEE Xplore / PDF eXpress strictly rejects Type 3 bitmap fonts (which Matplotlib emits by default when
Cycle 5 — SVG-First Tufte Serif Figures, Inkscape PDF Pipeline, TikZ Policy, & Redline Diff PDF
- SVG-First Figure Generation (
scripts/generate_paper_figures.py) & TikZ Preservation Policy:- Generate
.svg+.pdf+.pngfor all programmatic figures: Save every Matplotlib / programmatic figure in all three formats (fig.savefig(f"{stem}.svg", transparent=True, bbox_inches="tight"),.pdf, and 300 DPI.png). - Reference
.svgin MyST Markdown (): Referencing.svgin.mdfiles gives crisp, infinitely zoomable vector rendering in MyST Studio (http://127.0.0.1:7319) and HTML builds, while MyST's LaTeX/PDF exporter automatically converts.svg -> .pdfviainkscape(or our lightweight~/.local/bin/inkscapeCLI wrapper aroundrsvg-convert -f pdf) during./build_isr.shand./build_ieee.sh. - Keep Existing TikZ Diagrams as
.tikz.tex/.pdf(Do Not Convert TikZ $\to$ SVG): Standalone or inline TikZ schematics (*.tikz.texcompiled directly bypdflatexto.pdf+.pngfor web preview) are already native LaTeX vector graphics with Type 1/TrueType fonts. ConvertingTikZ -> PDF -> SVG -> Inkscape -> PDFintroduces unnecessary double-conversion and can degrade math glyph embeddings or clipping boxes—leave TikZ diagrams untouched as.tikz.tex/.pdf. - Use serif typography (
Times New Roman/Liberation Serif/DejaVu Serif,mathtext.fontset = "stix",pdf.fonttype = 42) with mathematical symbols identical tomethods.md. - Use left-aligned 2-line panel headers (bold panel letter + claim on line 1; muted
#475569subtitle with $N$, units, and statistic on line 2), despined top/right axes, and1:1native pixel aspect ratios (aspect="equal") on all camera crops and 3D renders.
- Generate
- Dual ISR & IEEE Compilation +
pdffontsAudit + Page-by-Page Visual Inspection:- Run
./build_isr.shand./build_ieee.sh(ensuringinkscapeor thersvg-convertinkscapeshim is onPATH), verifypdffontsreports zeroType 3fonts and 100%emb = yes, render pages to PNG (pdftoppm -png -r 150 <paper>_isr.pdf page), and visually inspect every page (or preview live in Sheprd MyST Studio via./sheprd mystonhttp://127.0.0.1:7319).
- Run
- Word-Level Redline Diff PDF (
scripts/diff_commits_pdf.py):- Whenever completing an editing cycle or preparing a revision for co-author review, generate a publication-style inline word-diff PDF comparing the baseline commit/tag against the updated manuscript:
python scripts/diff_commits_pdf.py <baseline_commit_or_tag> WORKTREE -f manuscript.md -o exports/manuscript_diff.pdf
- Whenever completing an editing cycle or preparing a revision for co-author review, generate a publication-style inline word-diff PDF comparing the baseline commit/tag against the updated manuscript:
Cycle 6 — Addressing Inline MyST Studio Comments (status: "addressed") & Finalization (myst-paper-finalize)
- Never Delete
<!-- tandem-comments -->Blocks While Addressing Review Comments:- When addressing inline user comments stored in
<!-- tandem-comments ... -->at the bottom of.mdfiles (or queried via./sheprd myst-comments <paper_id>), never strip or delete the comment entries during iterative drafting. - Instead, update each addressed comment entry in-place inside
<!-- tandem-comments -->so MyST Studio renders an emerald-green addressed highlight (✓💬) over the revised prose together with your explanation:- Set
"status": "addressed"and"resolved": true. - Preserve the original highlighted text in
"originalQuote". - Update
"quote"to a verbatim substring (60–180characters) of the newly rewritten sentence in that.mdfile so the highlight attaches cleanly in Live Paper and RTF views. - Set
"addressedBy"(e.g.,"claude-pb124-fascia-flow (eqx w6:p3)"),"addressedAt"("YYYY-MM-DD HH:MM"), and"agentReply"(concise 1–3 sentence explanation of the prose/figure change and any literature or metric verification).
- Set
- When addressing inline user comments stored in
- Finalizing and Stripping Comment History (
myst-paper-finalize):- Only strip
<!-- tandem-comments -->history when the user explicitly asks to clear addressed comments or finalize the manuscript for submission (see.agents/skills/myst-paper-finalize/SKILL.md):./sheprd myst-comments <paper_id> # Inspect open & addressed comments ./sheprd myst-comments finalize <paper_id> --addressed # Strip only addressed comments ./sheprd myst-comments finalize <paper_id> --all # Strip all comment history for submission
- Only strip
JAX-Compatible Configuration
Overview
This skill sets up typed configuration dataclasses that work cleanly with JAX tracing,
Equinox modules, and tyro CLI parsing. The core challenge: JAX's jit traces code and
caches compiled versions keyed on the static components of its inputs. Configs must
cleanly separate dynamic values (JAX array leaves that can change without recompilation)
from static values (Python values baked into the trace).
Refer to references/pytree-registration.md for the full pytree registration pattern
and references/tyro-integration.md for CLI configuration with tyro.
Decision Flow
-
Will this config be passed into a
jit/filter_jitfunction?- YES → Register it as a JAX pytree (see references/pytree-registration.md)
- NO → Plain
@dataclassis fine
-
Is the config part of an
eqx.Module(model, layer, etc.)?- YES → Use
eqx.field(static=True)on structural fields - NO → Use pytree registration with explicit dynamic/static field sets
- YES → Use
-
Does the config need CLI parsing?
- YES → Use tyro (see references/tyro-integration.md)
- NO → Plain
@dataclasswith defaults
The Golden Rules
Rule 1: Separate Static from Dynamic
Static fields (changing them SHOULD trigger recompilation):
int— loop bounds, array shapes, sequence lengthsbool— control flow switchesstr— mode selectors, pathsEnum— architecture variantsNone | float— nullable fields (structural change)
Dynamic fields (changing them MUST NOT trigger recompilation):
float— loss weights, learning rates, thresholds, hyperparameters- Anything used in pure arithmetic, not control flow
Rule 2: Wrap Scalar Floats into Arrays
CRITICAL: JAX does not properly trace bare Python float values as dynamic
pytree leaves. A bare float in a pytree leaf position gets treated as a static
value by the tracer, defeating the purpose of making it dynamic.
# WRONG — bare float stays static, recompiles on every change
children_dict[key] = value # value is 0.5 (Python float)
# RIGHT — wrapped into a 0-d array, properly traced as dynamic
children_dict[key] = jnp.asarray(value, dtype=jnp.float32)
Always convert dynamic float fields to jnp.float32 arrays in the tree_flatten
function. The tree_unflatten function receives JAX arrays and can pass them
directly to the dataclass constructor (they behave like floats in arithmetic).
Rule 3: Validate Outside JIT
Config validation (range checks, incompatible option detection) MUST happen at
startup, never inside a traced function. Call config.validate() once before
entering any JIT boundary.
config = tyro.cli(MyConfig)
config.validate() # Fail fast here, not inside jit
train(model, config) # Now safe to pass into traced code
Quick Reference: Field Classification
| Field Type | Static/Dynamic | Why |
|---|---|---|
int | Static | Often controls loops or shapes |
bool | Static | Controls if branches in trace |
str | Static | Mode selection, paths |
Enum | Static | Architecture variants |
float | Dynamic | Hyperparameters, weights, thresholds |
float | None | Static | Nullability is structural |
tuple[float, ...] | Static | Length is structural |
jax.Array | Dynamic | Already a JAX type |
Equinox Module Config Fields
When config values live inside an eqx.Module, use eqx.field(static=True):
class MyModel(eqx.Module):
# Dynamic — learnable parameters (JAX arrays)
encoder: eqx.nn.Linear
decoder: eqx.nn.Linear
# Static — architecture config (baked into trace)
hidden_dim: int = eqx.field(static=True)
num_layers: int = eqx.field(static=True)
mode: ArchitectureMode = eqx.field(static=True)
dropout_rate: float = eqx.field(static=True) # Static if used in control flow
Note: In eqx.Module, even float fields that control dropout probability or
similar structural behavior should be static=True. The dynamic-float rule applies
to pytree-registered dataclasses where you explicitly want to tune hyperparameters
without recompilation.
Common Mistakes
| Mistake | Fix |
|---|---|
Passing a plain @dataclass config into jit | Register it as a pytree or make it an eqx.Module |
Bare float as pytree leaf | Wrap with jnp.asarray(value, dtype=jnp.float32) in tree_flatten |
Validating config inside jit | Move validation to startup, before any traced calls |
Using eqx.field(static=True) on a float you want to sweep | Remove static=True; or use pytree registration instead |
Changing an int field and expecting no recompilation | Ints are always static — this is correct behavior |
float | None as dynamic field | Keep it static — None vs float is a structural change |
| Not testing retrace behavior | Write a test that changes dynamic fields and asserts no retrace |
File Organization
src/my_package/
├── configs.py # All config dataclasses + pytree registration
├── model.py # eqx.Module with eqx.field(static=True)
└── train.py # tyro.cli() entry point
scripts/
├── train.py # tyro.cli(TrainConfig)
├── eval.py # tyro.cli(EvalConfig)
└── sample.py # tyro.cli(SampleConfig)
Keep all configs in a single configs.py. Pytree registration goes at the bottom
of the same file, immediately after the dataclass definitions.
References
rae:jax-memory-and-retracing— Diagnosing OOMs, memory leaks, and JIT retracing at runtime (the diagnosis-side complement to this skill's design-time rules)references/pytree-registration.md— Full pattern for registering configs as JAX pytreesreferences/tyro-integration.md— CLI configuration with tyro + JAX pytree configsenforce-guidelines/references/python-standards.md— Python coding standardsenforce-guidelines/references/coding-standards.md— TDD, fail-fast
JAX JIT & Fusion Hygiene: Don't Run Hot Code Eager or Under-Fused
Overview
XLA only optimizes what you hand it inside a single jit. Code outside a jit
runs eager: every primitive op dispatches separately from Python, with no
fusion, no kernel-launch coalescing, and full host round-trips. On a deep model
this is dramatically slower than the same computation compiled once — yet it is
easy to miss, because eager code works, produces correct numbers, and shows no
error. The only symptom is that the GPU is starved and the wall-clock is high.
Three failure families:
- Eager hot path — an expensive forward (a ViT/backbone, a decode head) is
called as a plain
model(x)per item instead of througheqx.filter_jit. Per-op Python dispatch dominates. Worked example below: a deep ViT head called eager once per input crop; wrapping it ineqx.filter_jitwas ~11x faster. - Under-fused — the work is jitted, but split across many tiny jit calls
(or a Python
forloop calling a small jit per element), so XLA never sees enough to fuse and you pay a launch/dispatch tax per op or per iteration. - Wrong boundary — jit is present but placed too deep (inside the loop body only) or too wide (across a batch axis that then OOMs). Boundary placement is a speed↔memory trade, not a free win.
This skill gives symptom → diagnosis → fix recipes for each, plus the
filter_jit partition rules, static_argnums/donate_argnums, how to measure
honestly, and the parity caveat (jit ≠ eager bit-for-bit).
Quick Triage Table
| Symptom | Most likely cause | Section |
|---|---|---|
Hot forward called as plain model(x) per item; GPU ~idle, wall high | Eager hot path — wrap in eqx.filter_jit | 1 |
for item in items: jit_small(item) — per-iteration dispatch tax | Under-fused loop — fuse body, scan, or vmap | 2 |
Dozens of @jit helpers each called once per step | Too-fine jit boundaries — hoist into one outer jit | 2 |
jit is inside the loop body only; outer glue is eager jnp ops | Boundary too deep — jit the largest pure block | 3 |
Fused a batch axis with vmap, now OOM | Boundary too wide — chunk it; see memory skill | 3, 6 |
| Numbers shifted slightly after adding jit; a bit-equality test fails | Expected: XLA fusion reorders fp — gate on the real metric | 5 |
| "Is this actually faster?" unclear from one timer | Async dispatch / cold compile not controlled | 4 |
| Every call recompiles; latency spikes | Static arg or shape varies → retrace | jax-memory-and-retracing §1 |
1. Eager Hot Path — Wrap the Forward in eqx.filter_jit
How it happens
An Equinox Module is callable, so out = model(image, mask) just works and
returns correct values. But with no enclosing jit, every primitive inside the
forward dispatches separately from Python — for a deep ViT that is thousands of
host-driven kernel launches per call, no two of which XLA is allowed to fuse. The
result is correct, silent, and slow.
JAX's own guidance: "give the XLA compiler as much code as possible, so it can
fully optimize it" (jit-compilation guide).
A bare model(x) gives XLA nothing.
Worked example — a deep ViT head called eager per item
A common anti-pattern: a deep landmark/segmentation head called eager, once per
input crop, in a Python loop over items (crops, frames, detections). A
representative microbench of such a head — same GPU / same process, warm,
block_until_ready:
eager model(image, mask): ~138 ms/crop
eqx.filter_jit(model)(image, mask): ~12 ms/crop -> ~11x faster
The fix is a one-liner — wrap the per-item forward:
@eqx.filter_jit
def _jit_forward(model, image, mask):
return model(image, mask)
When the input shape is fixed across items, it compiles once and every
subsequent item reuses the fused executable. filter_jit caches on the model's
static structure, so a stable model object across calls compiles exactly once.
Diagnosis
- Grep the hot path for
model(/self.<submodule>(/.forward(calls that are not lexically inside an@eqx.filter_jit/@jax.jitfunction. - Look for any
Module.__call__invoked inside a Pythonfor/whileover items (crops, frames, people, cameras, hypotheses). - Confirm it's hot: tie it to the wall-share. Wrapping a 0.1%-of-wall helper buys nothing; wrapping the per-item forward of a phase that dominates wall is the win.
Fix
Wrap the largest pure forward in eqx.filter_jit (not the tiny inner layers).
Keep the model object stable across calls so the trace is reused — don't
rebuild or re-partition the module each iteration. Inputs that vary in value but
not shape stay traced automatically (§4 partition rules).
2. Under-Fused — Fuse Many Small Ops / Per-Element Calls into One Block
How it happens
Two shapes of the same mistake:
- Tiny-jit soup. Each helper is individually
@jit-decorated and called once per step. XLA optimizes each in isolation and you pay a dispatch/launch tax at every boundary between them — work that one enclosing jit would fuse. - Per-element Python loop around a small jit.
for x in xs: y = jit_f(x)drives the GPU one tiny launch at a time from Python, leaving it idle between launches (the classic "low GPU util, host-bound" profile).
JAX measured the dispatch cost directly: redefining/re-dispatching trivial jitted functions in a loop ran ~454 ms vs ~2.7 ms for the cached/fused form — a
100x gap that is pure overhead (jit-compilation guide).
Fix — pick by loop shape
- Independent elements, same shape →
vmapinside one jit. Batch the element axis so XLA fuses across it:eqx.filter_jit(jax.vmap(f))(xs). One launch, full occupancy. Caveat (§6): vmapping a heavy forward over a large batch axis materializes every element's activations at once and can OOM — e.g. a large ViT/backbone vmapped over a wide batch axis (many parallel streams/views) can exceed device memory at every batch size. Chunk the batch if it doesn't fit; seejax-memory-and-retracing. - Sequential dependence (state carried step→step) →
lax.scan. A Python loop inside jit is unrolled into a giant graph (slow to compile, large);lax.scancompiles to one XLAWhileand "is useful for reducing compilation times" (lax.scan docs). Note scan's main win is compile time / graph size, not always runtime — on some hardware the rolled loop blocks a few XLA optimizations. - Many small jitted helpers → one outer jit. Drop the inner
@jits and wrap the composite at the top. Define jitted functions once at module scope, never inside a loop or inside a per-call function/method body (def decode(...): @eqx.filter_jit def fwd(...):) — the cache keys on the function object's identity and closure, so a function-local@eqx.filter_jitorjax.jit(jax.vmap(lambda ...))builds a brand-new function object on every outer call and recompiles from scratch (guide). Worst of all, if the outer function loops over $N_{\text{batches}}$ batches (e.g. $T=120$ frames in batches of 8), the ~9 s cold XLA compile on batch 0 is amortized over the 120 frames (9000 ms / 120 + 5 ms ≈ 80 ms/frame), masquerading as a plausible steady-state per-frame cost (~18x slower than true steady state) and rendering any warmup call of the outer function useless! Pass call-local arrays (query_points, camera intrinsicsK, extrinsics) as explicit traced arguments to a module-scope@eqx.filter_jitfunction instead of capturing them in a function-local closure.
# WRONG — host-bound: one tiny launch per element, GPU idle between them.
results = [jit_f(model, x) for x in xs] # xs all same shape
# RIGHT — fuse the element axis into one jit (watch peak memory, §6).
results = eqx.filter_jit(jax.vmap(jit_f, in_axes=(None, 0)))(model, jnp.stack(xs))
3. Boundary Placement — Largest Pure Block, but Mind Memory
The right boundary is the largest contiguous pure-functional block: ideally one jit wrapping preprocess-compute-postprocess, with only true I/O / host control (decode, DB, Python branching on data) outside it.
- Too deep — jit only the innermost layer while the surrounding glue stays eager jnp ops. Each glue op dispatches from Python. Hoist the boundary outward until it encloses the whole pure block.
- Too wide — jit (or vmap) across an axis whose fused activations don't fit.
Speed and memory pull opposite ways here. Don't keep widening blindly; chunk the
axis (
lax.mapover chunks) so peak ∝ chunk, not ∝ batch. Cross-referencejax-memory-and-retracing§"Reduce-after-fan-out". - Block differentiates a deep sub-forward — if the pure block trains through a
large frozen network (perceptual / teacher / discriminator loss), the backward
stores that network's full activation tape and the block can OOM at
jit_stepeven on a 2x card.jax.checkpoint(remat) the sub-forward to recompute it in backward. Seejax-memory-and-retracing§"Backward-tape OOM". - Value-dependent control flow can't go inside jit directly — restructure with
jnp.where/lax.cond, or keep that branch outside and jit the two pure arms.
4. filter_jit Partition Semantics, static_argnums, donate
What eqx.filter_jit does
"All JAX and NumPy arrays are traced, and all other types are held static",
at the PyTree-leaf level — so a single argument can mix traced array leaves
and static non-array leaves (Equinox transformations).
This is why you wrap an eqx.Module directly: its array params are traced, its
static structure (and any eqx.field(static=True) config) is cached as part of
the compile key.
Consequences to internalize:
- A Python scalar that varies per call is static → it retraces on every new
value. To make it traced, pass
jnp.asarray(x). (This is the #1 retrace leak — seejax-memory-and-retracing§1.) eqx.field(static=True)is for genuine config only. Making a JAX array static is almost always a bug; it bakes the array's values into the compile key and recompiles when they change. Keep weights/data as normal (traced) leaves.- Keep the model object identity/structure stable across calls so the cached trace is reused. Re-partitioning or rebuilding the module each step defeats it.
static_argnums / static_argnames (plain jax.jit)
Mark an argument static to allow value-based control flow inside the function — but "JAX will have to re-compile the function for every new value", so it "only [works] if the function is guaranteed to see a limited set of static values" (guide). Use it for a handful of fixed configs; never for a per-frame index or per-call count.
donate_argnums / buffer donation
If an input is not needed after the call and matches an output's shape and
dtype, donate its buffer so XLA reuses that memory for the output — "This will
reduce the memory required for the execution by the size of the donated buffer"
(buffer donation). The
hard rule: "It is not allowed to donate a buffer that is used subsequently" —
reuse raises an error. Donation is positional-args only, and follows pytrees.
Best fit: in-place-style state carried through a step (optimizer/tracker state)
where the old buffer is dead after the update. eqx.filter_jit(donate="all") /
"all-except-first" apply this with leaf-level filtering.
5. Parity Caveat — jit ≠ eager Bit-for-Bit
Adding jit changes the numbers slightly. XLA fuses and reorders
floating-point ops, so a jitted forward is not byte-identical to the eager
one. Typical magnitude for a deep vision forward: outputs shift by ~1e-4–1e-3 in
their native units (e.g. sub-pixel on normalized coordinates, ~1e-3 on
probabilities) — negligible for the task, fatal for a == test.
Therefore:
- Do not gate a jit change on bit-equality — it will fail for a correct change. An "it'll be bit-equal" expectation is usually wrong.
- Gate on the metric that matters end-to-end — your task's accuracy/error
measure (e.g. a reprojection error, an MPJPE, a downstream consumer's values) —
within a tolerance, not
==. - Flag any jit change that feeds a downstream exact-match assertion or a byte-identical stored/cache key; loosen those to metric tolerances.
6. How to Measure (so the win is real, not noise)
Eager-vs-jit and fused-vs-unfused A/Bs are only trustworthy if you control the confounds — same hardware, warm, synchronized:
- Same GPU, same process. Never compare GPU-A eager vs GPU-B jit (arch/clock confound). Run both arms back-to-back on one card.
- Warm, post-compile. The first call pays a one-time XLA compile (seconds to minutes). Do a warmup call, then time. Report warm and cold separately.
block_until_ready()— mandatory. JAX dispatch is async; without it you time the dispatch, not the compute (guide).jax.block_until_ready(jit_f(model, x)) # warmup + compile t0 = time.perf_counter() for _ in range(reps): out = jax.block_until_ready(jit_f(model, x)) ms = (time.perf_counter() - t0) / reps * 1e3- Watch GPU util as the inefficiency signal, not the score. Low util on a
high-wall phase = host-bound / under-fused (the thing to fix); the score is
still wall-clock. (
nvidia-smi dmon, or a profiler trace viajax.profiler.trace→ TensorBoard for a per-op breakdown.) - Confirm zero compilations during the timed window (
compiles_during_timed == 0) — Doing a warmup callsync(fn())beforefor _ in range(reps): sync(fn())does not guarantee that the timed calls are warm iffn()internally builds a function-local@eqx.filter_jitor retraces! Attach a programmaticjax_log_compileslogging handler (_CompileCounter) and assertcompiles_during_timed == 0over the timed repetitions. Repeated compiles mean a retrace or function-local@jitrebuild, not steady-state compute (→jax-memory-and-retracing).
Pre-flight Checklist for a Hot Inference or Training Path
- The expensive forward is inside
eqx.filter_jit/jax.jit, not eager - No
Module.__call__/ heavy op in a bare Python per-item loop - One jit wraps the largest pure block; defined at module scope (never inside a per-call function/method body, loop, or fresh lambda/partial)
- Independent same-shape elements vmapped (chunked if the batch OOMs)
- Sequential state loops are
lax.scan, not Python-unrolled; state-independent gathers, goal encodings, and PRNG splits are hoisted outsidelax.scanasxs(rae-jax-profiling§11) - Multi-step training/collection loops fuse $K \in [20, 100]$ iterations + on-device ring-buffer updates (
dynamic_update_slice_in_dim) inside one@eqx.filter_jit(donate="all")lax.scan(rae-jax-profiling§12) - Mixed-precision MLP hidden layers set
preferred_element_type=jnp.bfloat16(notfloat32) so backward cotangent GEMMs (dX,dW) stay onbf16tensor cores while master weights, normalizers, state integrators, heads, and losses stayfloat32(rae-jax-profiling§8) - Kinematic tree Jacobians use
@jax.custom_jvpwith a static binary ancestor mask andPrecision.HIGHESTon $3\times 3$ matmuls rather thanjax.jacfwdthrough the tree recurrence (rae-jax-profiling§9) - No per-call Python scalar crossing the jit boundary (pass
jnp.asarray) -
static=Trueonly on real config; weights/data stay traced - Benchmarked warm, same-GPU, same-process, with
block_until_ready - Gated on your task metric within tolerance, NOT bit-equality
- Programmatic compile counter (
jax_log_compiles+jax.monitoringlistener on"/jax/core/compile/backend_compile_duration") assertscompiles_during_timed == 0during warm timing
Related Skills
rae-jax-profiling— canonical RAE operational checklist coveringpreferred_element_type=jnp.bfloat16backward GEMMs (§8),@jax.custom_jvpanalytic tree Jacobians withPrecision.HIGHEST(§9),jax.monitoringpmaprecompile detection & sharding-flip fixes (§10), loop-invariantlax.scanhoisting & 1-GEMM unrolled BPTT weight gradients (§11), and multi-iterationlax.scanprogram granularity + persistent cache hygiene (§12).jax-memory-and-retracing— the complement to this skill. Fusion buys speed but raises peak memory (a widevmap/jit materializes all activations at once → OOM; chunk it) and a mis-set boundary can retrace (a static scalar/shape that varies). Use that skill when fusing makes a path OOM or recompile, and for the no-retraceassert_max_tracesregression test that locks the win in.jax-config— static/dynamic config separation and pytree registration; the design-time complement that prevents most static-vs-traced boundary mistakes.
JAX Memory & Retracing: Diagnose and Avoid OOMs, Leaks, and Recompiles
Multi-device runs add their own OOM and hang modes — unreplication residency (an "unreplicated" tree that is still resident on every device), collective deadlocks, and schedule scaling. If the OOM lands at a checkpoint, validation or rendering boundary, or the process is alive but doing no work, start at
jax-multi-gpu.
Overview
Three failure families account for most "mysterious" crashes in long-running JAX/Equinox programs:
- Retrace leaks — a jit'd function silently recompiles on every call, and every compiled variant is retained, holding device and host memory.
- Host-RAM OOM — Python-side accumulation (frames, masks, caches) grows until the Linux OOM-killer SIGKILLs the process with no traceback.
- GPU OOM — the device allocator fails, either loudly
(
RESOURCE_EXHAUSTED) or disguised as an unrelated error raised mid-operation.
This skill gives symptom → diagnosis → fix recipes for each, plus the compilation-cache, CUDA/cuDNN packaging, and precision-tiering hazards that feed them.
Quick Triage Table
| Symptom | Most likely cause | Section |
|---|---|---|
| Every call to a jit'd fn is slow; occasional 10-100x latency outliers | Retracing (static Python scalar, changing shapes, or function-local @jit) | 1, 2 |
| Batch/sequence helper reports ~50–100 ms/frame "warm" when the inner model is ~5 ms/frame | Function-local @eqx.filter_jit / jax.jit(vmap(lambda ...)) rebuilt on every outer call, amortizing a multi-second compile over the batch loop | 1, 2 |
| GPU memory grows step over step inside a "fixed-size" loop | Retained compiled executables from retraces, or an unbounded feature cache | 1, 2 |
| Process disappears mid-run, log ends abruptly, no Python traceback | Host OOM-killer (SIGKILL) — usually unbounded host-side accumulation | 4 |
XlaRuntimeError: RESOURCE_EXHAUSTED | GPU OOM — JAX raised it; catchable and diagnosable | 5 |
OOM at jit_step/first backward; worsens with more views/loss terms; still OOMs on a 2x-bigger card | Un-checkpointed backward tape through a large sub-forward (perceptual/teacher/discriminator) in the loss | 5 |
| OOM during model loading, error blames a checkpoint leaf / tree mismatch | Device allocator died mid-deserialize; wrapped as a structure error | 5 |
| Worked yesterday; today jit'd fns fail with version-mismatch / deserialize errors | Stale persistent compilation cache after a cuDNN/driver/jaxlib change | 3 |
libcudnn load errors or cudnn-graph build failures after adding a dependency | Two cuDNN wheels (cu12 vs cu13) colliding, or cuDNN below what jaxlib needs | 6 |
| Memory roughly doubled with no code change | jax_enable_x64 flipped on (float64 default) | 7 |
1. Retracing / Recompilation Leaks
How the leak happens
jax.jit (and eqx.filter_jit) cache compiled executables keyed on the
function object identity plus the static parts of the call: array
shapes, dtypes, pytree structure, and every non-array Python value. Three
common ways to leak compiles:
- Python scalar treated as static.
eqx.filter_jittreats anything that is not a JAX/NumPy array as static — including plain Pythonints such as a frame index, sequence length, or count. Every new value → a fresh trace, a fresh XLA compile, and a new retained executable. (jax.jitdoes the same forstatic_argnumsarguments, and for any int that changes a shape.) - Pytree leaf whose shape changes between calls. E.g. a detector returns
(N_kept, H, W)whereN_keptdepends on a score threshold; every novelN_keptrecompiles all downstream jit'd consumers (resize, IoU, matching). - Function-local
@eqx.filter_jit/@jax.jitrebuilt on every outer call (and hidden by inner-loop amortization). A sequence or batch helper (def decode(head, images, K, ...):) defines@eqx.filter_jit def fwd(...)orsolve = jax.jit(jax.vmap(lambda ...))inside its body to capture call-local variables (qp,k_jax) in a closure. Within a singledecode()call over $T=120$ frames ($15$ batches of $8$), batch 0 pays a full XLA compile (~9 s) and batches 1..14 hit the local cache (~4.5 ms/frame). Averaged over $T=120$ frames,decode()reports ~80–85 ms/frame — hiding a 9-second cold XLA compile inside what looks like a plausible per-frame forward time (~19x slower than true steady state)! Worse, as soon asdecode()returns, the localfwdis garbage-collected, so every subsequent call todecode()(every video clip, trial, or "warm" benchmark repetition) builds a fresh function object and recompiles from scratch.
Each retained executable holds device memory (baked-in constants, workspace) and host memory (the lowered program). In a per-frame loop this looks exactly like a memory leak — GPU usage stair-steps upward until OOM — and each new variant also costs hundreds of ms to seconds of compile time, visible as latency spikes.
Symptoms
- Per-step time never reaches a fast steady state, or shows sporadic ~10-100x outliers on specific iterations.
- GPU (and host) memory grows monotonically in a loop that should have a fixed working set.
JAX_LOG_COMPILES=1prints a compilation line on calls after the first.
Fixes
- Make varying values traced, not static. Pass
jnp.asarray(x)instead of a bare Python scalar. If the value feeds Python control flow, restructure withjnp.where/lax.condmasks so the trace is value-independent. - Keep leaf shapes static: pad to a fixed maximum + validity mask.
Instead of returning
(N_kept, ...), return top-K with fixedK = N_MAXand a(N_MAX,)boolvalidmask; downstream ops gate results onvalidso padded slots are inert. Select top-K in pure JAX (jnp.argsort(-scores)[:N_MAX]), notnp.nonzeroon host. - Never branch shapes on Python ints. Anything like
x[:n]orjnp.zeros((n, d))withnvarying per call must either be bucketed (roundnup to a small set of fixed sizes, e.g.(8, 16, 32)) or padded to a max. Bucketing bounds compiles to the bucket count. - State objects carried across steps must have static shape. A tracker/accumulator pytree passed in and out of a jit'd step must keep identical leaf shapes every step — preallocate fixed-capacity slots and a ring-buffer index rather than appending.
- Define
@eqx.filter_jit/@jax.jitfunctions once at module scope (or once in__init__), never inside a per-call function or method body. Pass call-local arrays (query_points, camera intrinsicsK, extrinsics) as explicit traced arguments instead of capturing them in a function-local closure. - If you intentionally discard a set of compiled variants (e.g. after a
resolution change), call
jax.clear_caches()to release them.
# WRONG — num_frames is a Python int: filter_jit treats it as static,
# so every distinct value compiles (and retains) a new executable.
@eqx.filter_jit
def step(model, frame, num_frames):
...
# RIGHT — traced 0-d array; one compile covers all values.
step(model, frame, jnp.asarray(num_frames))
# WRONG — output shape depends on data: every novel count retraces consumers.
keep = np.nonzero(scores >= threshold)[0]
return masks[keep]
# RIGHT — fixed (N_MAX, H, W) + valid mask; shape never changes.
order = jnp.argsort(-scores)[:N_MAX]
return masks[order], scores[order] >= threshold
# WRONG — @eqx.filter_jit defined inside decode() to close over qp and k_jax:
# every call to decode() creates a brand-new `fwd` function object and pays a
# full ~9 s XLA compile on batch 0 (amortized across the batch loop to look like
# ~85 ms/frame instead of the true ~4.5 ms/frame steady state!).
def decode(head, images, K, boxes, valid, *, batch_size=8):
qp, k_jax = head.query_points, jnp.asarray(K, jnp.float32)
@eqx.filter_jit
def fwd(model, ims, bxs, ok):
return jax.vmap(lambda im, b, v: forward_one(model, qp, im, b, v, k_jax))(ims, bxs, ok)
return [fwd(head.device_model, ...) for ...]
# RIGHT — hoisted to module scope with qp and k_jax passed as traced arguments;
# compiles once across the entire process lifetime.
@eqx.filter_jit
def _decode_batch_jit(model, qp, k_jax, ims, bxs, ok):
return jax.vmap(lambda im, b, v: forward_one(model, qp, im, b, v, k_jax))(ims, bxs, ok)
1b. The Inverse: a Cached Trace Silently Absorbing a Change
Section 1 is about retracing too often. The same cache key causes the opposite failure, and it is far harder to see: jit does not retrace when you needed it to, so a change you made has no effect and the code appears to confirm it.
How it happens
The cache is keyed on the static parts of the call — shapes, dtypes, pytree structure, non-array Python values. It is not keyed on module-level constants, config values, or any other global the traced code read. Change one of those and call the function again with the same argument structure, and you get the previous executable back.
# WRONG -- the mutation is absorbed by the cache; the assertion tests nothing.
config.EPS = 1e-5 # was 1e-6
out = model(x) # same shapes -> cached trace, old EPS
assert allclose(out, golden) # PASSES. Nothing was mutated in the executable.
Where it bites hardest: mutation tests
A mutation test exists to prove a guard has teeth: break the thing it guards, watch it fail. If the mutation is absorbed by a trace, the test passes and is recorded as proven when it is inert — the worst outcome, because a test nobody distrusts is worse than no test.
Fix
Force a re-trace, then confirm the mutation is observable before trusting any arm that passes:
# Reach the undecorated body, so no cache is consulted at all.
type(model).__call__.__wrapped__(model, x)
# Or clear the caches between arms.
jax.clear_caches()
# Or build a fresh module/closure per arm, so the cache key differs.
# And prove the harness can see a mutation at all -- run one arm you KNOW must
# fail, and watch it fail, before crediting an arm that passes.
The same cache defeats memory measurement
A module baked into a closure is a compile-time constant; the same module passed as a traced argument is not. The two produce different executables and different peak-memory figures for the same computation. Measure in the calling convention you ship, or the number describes a program you do not run.
Relatedly, jax.device_get-style peak counters (peak_bytes_in_use, and torch's
max_memory_allocated) are process-lifetime high-water marks. Several arms in
one process all report the maximum over every arm run so far, which silently
yields an identical figure for arms that differ. One arm, one process.
2. Detecting Retraces (Equinox/JAX tooling)
JAX_LOG_COMPILES=1 — locate and count compilations
JAX_LOG_COMPILES=1 python my_script.py 2>&1 | grep -c "Compiling"
Every line after warmup is a retrace. The logged function name tells you which jit boundary is unstable; correlate the timestamps with your loop index to find what changed.
eqx.debug.assert_max_traces — hard-fail on unexpected retraces
Wrap the inner function, then jit. Order matters: assert_max_traces
counts traces of the function it directly wraps, so jit-then-assert would
count outer dispatches (every call), not real compilations.
inner = eqx.debug.assert_max_traces(step_fn, max_traces=1)
jit_step = eqx.filter_jit(inner)
...
eqx.debug.get_num_traces(inner) # introspect the count at any time
Gate it behind an env var so production pays zero overhead and CI can opt in:
def _maybe_guard(fn, registry, name):
if not os.environ.get("MYPKG_ASSERT_NO_RETRACE"):
return fn
inner = eqx.debug.assert_max_traces(fn, max_traces=1)
registry[name] = inner # so tests can query get_num_traces(inner)
return inner
self._jit_step = eqx.filter_jit(_maybe_guard(step_fn, self._guards, "step"))
A no-retrace regression test
The highest-value test in any per-frame/per-step pipeline: exercise the path that historically changed shapes or scalars, and assert the trace count stayed at 1. Run it on CPU with a synthetic model — it needs no weights.
def test_step_compiles_once_across_lengths(monkeypatch):
monkeypatch.setenv("MYPKG_ASSERT_NO_RETRACE", "1")
runner = make_runner() # small synthetic config, CPU is fine
for num_frames in (3, 5, 9): # values that used to retrace
runner.run(synthetic_frames(num_frames))
assert all(n == 1 for n in runner.trace_counts().values())
assert_max_traces makes the failure loud at the moment of the second
trace (with both call signatures in the error), which is far easier to debug
than a downstream OOM hours later.
Gotcha — assert_max_traces on a function-local definition is a guard that
cannot fail across outer calls. If a helper builds @eqx.filter_jit inside
its own function body (def decode(...): @eqx.filter_jit def fwd(...)),
wrapping fwd with assert_max_traces(max_traces=1) inside decode()
creates a brand-new guard instance on every decode() call with its trace
counter reset to 0. Within one decode() call, all batches have identical
shape so the counter reaches 1 and passes — completely blind to the fact that
100 calls to decode() compiled 100 separate XLA executables! Always hoist
both the @eqx.filter_jit function and its assert_max_traces wrapper to
module scope, and pair them with a process-wide XLA compile counter in
benchmarks and integration tests:
import logging
import jax
class _CompileCounter(logging.Handler):
"""Process-wide XLA compile counter (catches even function-local @jit rebuilds)."""
n = 0
def emit(self, record):
msg = record.getMessage()
if "Finished XLA compilation" in msg or "Compiling " in msg:
_CompileCounter.n += 1
def enable_compile_count():
jax.config.update("jax_log_compiles", True)
h = _CompileCounter()
for name in ("jax", "jax._src.dispatch", "jax._src.interpreters.pxla", "jax._src.compiler"):
lg = logging.getLogger(name)
lg.addHandler(h)
lg.setLevel(logging.WARNING)
lg.propagate = False
Assert _CompileCounter.n - c0 == 0 across all post-warmup calls or timed
benchmark repetitions.
3. Persistent Compilation Cache Pitfalls
jax.config.update("jax_compilation_cache_dir", ...) persists compiled
executables across processes — a big win when cold compiles take minutes.
Two hazards:
- Stale-cache poisoning. The cache key does not capture every runtime dimension — notably the cuDNN/driver runtime can change without changing the key. After upgrading cuDNN (or jaxlib pulling a different one), the cache can serve an incompatible executable, failing with deserialize / version-mismatch errors that look like corruption. Fix: delete the cache directory after any CUDA/cuDNN/jaxlib/driver change. Treat "wipe the JAX cache dir" as a standard step in upgrade SOPs.
- Libraries must not enable it on import. Setting
jax_compilation_cache_diris a global config flip and a filesystem side effect (it silently accumulates GBs). A library that does this on import hijacks application policy. Fix: make it opt-in via an env var or explicit setup call owned by the application:
def setup_compilation_cache(cache_dir: str | None = None) -> Path | None:
"""Opt-in persistent compile cache. Off unless the caller asks."""
cache_dir = cache_dir or os.environ.get("MYPKG_JAX_CACHE_DIR")
if not cache_dir:
return None
path = Path(cache_dir).expanduser()
path.mkdir(parents=True, exist_ok=True)
jax.config.update("jax_compilation_cache_dir", str(path))
jax.config.update("jax_persistent_cache_min_compile_time_secs", 1.0)
return path
A persistent cache only pays off if nothing retraces — pin that first with the no-retrace tests from section 2, otherwise the cache just grows.
4. Host-RAM OOM: the Traceback-less SIGKILL
Recognize it
The process disappears — log ends mid-line, shell reports Killed or
exit code 137 (128+SIGKILL), and there is no Python traceback. This is
the Linux/cgroup OOM-killer. PYTHONFAULTHANDLER cannot help: SIGKILL is
not interceptable, so faulthandler prints nothing. Do not burn time on
batch-size or GPU mem-fraction knobs — this class is host RAM, not GPU.
Confirm:
dmesg -T | grep -i "killed process" # if you have access
# Otherwise: watch host RAM during a repro
while sleep 5; do free -g | awk '/Mem:/{print strftime("%T"), $3" GB used"}'; done
# Or track the process itself:
while sleep 5; do ps -o rss= -p <PID> | awk '{print strftime("%T"), $1/1048576" GB RSS"}'; done
Monotonic growth toward the machine's limit = accumulation leak. A flat plateau that is simply too high = oversized working set (datasets, workers).
The two canonical causes
(a) Accumulating O(T) full-resolution arrays over a sequence. The natural consumer pattern for a streaming model — "append every frame's masks/frames/ features to a list" — buffers the entire video in host RAM. A single full-HD float32 mask is ~8 MB; per-object, per-frame, this caps runs at roughly a thousand frames and then SIGKILLs.
Fixes, in order of preference:
- Stream, don't accumulate. Consume each frame's output inside the loop (write to disk, update running stats) and let it go. Provide an iterator API so O(1)-memory consumption is the easy path.
- Store compactly when you must keep everything. Binary masks:
threshold →
np.packbits→zlib.compressis lossless, stdlib-only, and ~1000x smaller than float32; decode lazily on access. Frames/features: uint8, downsampled, or memory-mapped to disk. - Bound every cache with a real budget. An "it's only a few MB" LRU
becomes hundreds of GB when shard sizes grow 100x. Size caches as
entries x measured_entry_size x num_workersagainst actual host RAM, and re-check the bound whenever data size changes. Never ship an unboundeddictcache on a data path.
(b) Compiled-executable buildup from retracing (section 1) — XLA compilation itself can also transiently spike host RAM by many GB on large models, so an almost-full host dies during a compile, which is confusing: the leak was the dataset cache, but the killer struck inside JIT.
5. GPU-OOM Diagnosis Discipline
Two distinct presentations
- Catchable:
jaxlib.xla_extension.XlaRuntimeError: RESOURCE_EXHAUSTED. JAX raised it; you get a traceback and can reason about which allocation failed. - Disguised: the allocator fails midway through a larger operation and
surfaces as an unrelated error. Classic case: model deserialization OOMs
partway through the leaf list and presents as a tree-structure /
leaf-mismatch error naming whatever leaf it died on. If a "structure
mismatch" appears only on busier GPUs or at lower memory fractions,
suspect OOM, and check the exception's
__cause__chain forRESOURCE_EXHAUSTED.
Tell GPU from host, leak from working set
Sample both memories over time; the divergent one is your problem:
# GPU, every 5 s:
nvidia-smi --query-gpu=timestamp,memory.used --format=csv -l 5
# Host, every 5 s:
while sleep 5; do free -m | awk '/Mem:/{print strftime("%T"), $3}'; done
- GPU grows stepwise per iteration → retained executables (section 1) or a per-step device-array cache you forgot to clear.
- GPU flat but high, fails only at one phase → working set too large for
that phase; reduce batch/resolution or free the previous phase's arrays
(drop references, then
jax.clear_caches()if compiled variants pin them). - Host grows, GPU flat → section 4.
Reduce-after-fan-out: scan/chunk, never one wide vmap
A common phase-OOM is a fan-out you immediately reduce: run the same model
on N variants (augmentations, hypotheses, MC samples, ensemble members) and
collapse the N outputs to a statistic (mean, std, max, top-k). The natural
jax.vmap(f)(variants) materializes all N forwards' activations at once,
so the peak scales with N even though the result is tiny. On a large model
(a ViT/backbone × a real batch) a modest N (e.g. 8) is enough to exhaust the
device — the failing allocation names the fan-out's executable and is
N-proportional.
The fix is to stream the fan-out and reduce online, so only chunk forwards
co-reside:
# WRONG — all N augmented forwards live at once; peak ∝ N.
outs = jax.vmap(lambda k: model_forward(augment(x, k)))(keys) # (N, B, ...)
sigma = jnp.std(outs, axis=0)
# RIGHT — lax.map over chunks; peak ∝ chunk, result identical.
def chunk_fwd(ck): # ck: (chunk,) keys
return jax.vmap(lambda k: model_forward(augment(x, k)))(ck)
outs = jax.lax.map(chunk_fwd, keys.reshape(n_chunks, chunk, *keys.shape[1:]))
sigma = jnp.std(outs.reshape(-1, *outs.shape[2:])[:N], axis=0)
chunk is a pure memory↔throughput knob — it does not change the numerical
result (same N samples), so set it small enough to fit the phase and leave N
at whatever the statistic needs. For a true O(1)-peak reduction (no (N, …)
stack at all), carry a running accumulator through lax.scan (e.g. Welford for
mean/variance) instead of stacking then reducing.
Gotcha — a chunk default >= N silently collapses to the wide vmap. If
the chunking helper defaults chunk to the same value as the fan-out count (or
the caller omits it), n_chunks == 1 and lax.map runs a single group over all
N — i.e. the exact wide vmap you were avoiding, with no error, just the OOM.
Default chunk to a small constant (2–4), and treat chunk >= N as the
unguarded wide-vmap case in the docstring so callers know it OOMs large models.
Backward-tape OOM: gradient-checkpoint large frozen sub-forwards used in a loss
A loss term that forwards a large network on the prediction — a perceptual /
feature loss (VGG / DINO / CLIP), a frozen teacher, a discriminator — is
differentiable wrt that prediction, so reverse-mode AD must store the
sub-network's entire activation tape for the backward pass. The weights being
frozen does not help: freezing zeroes the weight gradient, but the
input-gradient path still needs every intermediate activation. Run the
sub-forward K times per step (once per view, timestep, MC sample, or loss
target) and the backward-tape peak scales with K x depth x tokens, dwarfing
the rest of the step.
The tell that distinguishes this from a working-set OOM: the failure strikes
at jit_step (the first backward), worsens as you add loss terms or views, and
does not fit even on a much larger card — a 2x GPU OOMs the same way, because
the cost is the tape, not a one-off buffer. (A working-set OOM, by contrast,
shrinks with batch/resolution and does fit when you upsize the card.) If a
perceptual/feature loss makes a run need >2x its base memory, suspect this before
reaching for a bigger GPU.
Fixes, in order:
jax.checkpoint(remat) the sub-forward. Wrap the frozen network's forward so its activations are recomputed in the backward instead of stored — trading compute for memory. Peak drops fromK x full-tapeto roughlyK x one-layer. Value-identical; the single highest-leverage fix.stop_gradientthe target branch. Ind(f(pred), f(gt)),f(gt)needs no tape — wrap it so only thepredbranch is differentiated.- Cut
K. Score the term on fewer views/samples; a perceptual loss on the single matched view is usually as effective as on all of them at a fraction of the memory.
# WRONG — frozen backbone forward on every view; full tape per view -> peak ∝ K.
def feat_loss(pred, gt, mask):
return jnp.mean(1 - cos(backbone(pred * mask), backbone(gt * mask)))
total = sum(feat_loss(pred[v], gt[v], mask[v]) for v in range(K)) # OOMs at jit_step
# RIGHT — remat the differentiated forward, stop-grad the target, score one view.
backbone_remat = jax.checkpoint(backbone)
def feat_loss(pred, gt, mask):
f_gt = jax.lax.stop_gradient(backbone(gt * mask)) # no tape
return jnp.mean(1 - cos(backbone_remat(pred * mask), f_gt)) # recomputed in bwd
total = feat_loss(pred[0], gt[0], mask[0]) # K = 1
This is the backward-pass dual of the forward-peak fan-out above: §"Reduce-after-fan-out"
chunks the forward activations of N parallel variants; remat recomputes the
backward activations of a deep sub-forward. A loss with both (a perceptual term
under a vmap over views) needs both fixes.
Standardized Cross-Project Toolchain & XLA Flags (body_models.environment_check)
Rather than scattering ad-hoc os.environ["XLA_FLAGS"] blocks across training scripts, projects in the biomechanics ecosystem share a single, pure-metadata environment checker and configurator in body_models.environment_check:
- Before JAX backend initialization (top of entrypoints /
conftest.py): Callconfigure_environment()(orapply_recommended_flags()) to idempotently setXLA_PYTHON_CLIENT_PREALLOCATE=false, append version-specific XLA workaround flags (such as--xla_gpu_experimental_gemm_fusion_v2=falseonjaxlib >= 0.12to prevent the Triton GEMM nested-vmapbackward miscompileopenxla/xla#51186), and configure partitioner workarounds (JAX_USE_SHARDY_PARTITIONER=falseonjaxlib == 0.11.2). - Passive runtime verification (
check_environment()/find_gotchas()): Automatically invoked once per process whenForwardKinematicsis constructed (and callable directly by any library). Readsimportlib.metadataandos.environwithout importingjaxor touching devices, emitting aToolchainWarningif any known JAX / XLA / cuDNN / NCCL gotcha is active and unmitigated.
Allocator hygiene
XLA_PYTHON_CLIENT_PREALLOCATE=false # see true usage; play nice on shared GPUs
XLA_PYTHON_CLIENT_MEM_FRACTION=0.65 # leave headroom for transients & cohabitants
Preallocation off makes nvidia-smi meaningful for diagnosis. Choose a
mem fraction that leaves room for allocator transients — peak usage during
loading/compile can briefly exceed steady state by 2x.
Two loading/lifecycle patterns that prevent avoidable GPU OOMs:
- Deserialize weights to CPU first, then upload once. Loading a
checkpoint directly onto the device while a randomly-initialized skeleton
of the same model is still resident doubles peak device memory. Build /
deserialize under
jax.default_device(jax.devices("cpu")[0]), drop the skeleton, thendevice_putthe array leaves. - Own your background jobs. Backgrounded GPU processes that lose their
launcher frequently orphan and keep their device memory. Before
relaunching, find and kill your own stale PID (never broad
pkillon a shared machine); before blaming contention, check whether the squatter is yours.
6. CUDA/cuDNN Dependency Hazards
There is one libcudnn.so soname per process — two providers cannot
coexist.
- Torch pulling CUDA-13. Default GPU torch wheels can be CUDA-13 builds
that depend on
nvidia-cudnn-cu13, which file-collides withjax-cuda12'snvidia-cudnn-cu12and breaks JAX's cudnn-graph paths. Fix: if torch is only used for weight conversion or host-side I/O, resolve it from the CPU wheel index (carries no cuDNN) and put the conversion deps behind an optional extra. Only co-install GPU torch when it genuinely runs on the GPU, and then match CUDA majors. - cuDNN floor below what jaxlib needs. JAX's declared constraint (e.g.
cudnn>=9.x) can be looser than what its bundled XLA actually needs to build cudnn graphs (e.g. flash attention). The resolver then picks a too-old patch release and JIT fails at runtime. Fix: pin/floornvidia-cudnn-cu12(via a constraint, not a direct dep) at a version verified against your jaxlib, so fresh clones resolve a working runtime. - After any change in this layer, wipe the persistent compilation cache (section 3).
7. Precision / Memory Tiering
Precision is a memory knob with sharp numerical edges. Default tiering:
- float32 is the global default. Never enable
jax_enable_x64globally: it silently doubles every array and the autodiff tape — a fit that used to OOM at 2x memory is the classic signature. If some component needs float64 (reference-equivalence tests, a numerically delicate solver), gate it behind an explicit opt-in flag scoped to that component. - bfloat16 for big NN forwards (backbones, encoders) is usually safe — validate once against f32 (cosine similarity / max-abs on outputs) and you halve activation memory and often gain throughput.
- Never run ill-conditioned optimization in bf16 — second-order / line-search fits (LBFGS, IK, bundle-adjust) need f32 minimum; bf16's 8-bit mantissa destroys convergence. Tier within the program: bf16 forwards feeding an f32 optimizer is a sound and common split.
Rule of thumb: dtype choices are per-subsystem policy, set explicitly at each boundary — never a global config flip made on import.
Pre-flight Checklist for Long Sequence/Video Loops
- All jit'd step functions defined at module scope (never inside a
per-call function/method body) and guarded by a module-scope
assert_max_traces(max_traces=1)+ a no-retrace test over the shape-varying inputs - Warm benchmarks and multi-call integration tests assert
compiles_during_timed == 0via a process-widejax_log_compileshandler (catching any hidden function-local@jitrebuilds) - No Python scalar that varies per call crosses a
filter_jitboundary - Variable-count outputs padded to a static max with a
validmask - Per-frame outputs streamed or compressed, never appended raw
- Every cache has an explicit, measured budget
-
XLA_PYTHON_CLIENT_PREALLOCATE=false+ mem fraction with headroom - Persistent compile cache opt-in only; wiped after CUDA/cuDNN changes
- x64 off globally; bf16 only on validated NN forwards, never on fits
- Large frozen sub-forwards in a loss (perceptual / teacher / discriminator)
wrapped in
jax.checkpoint; target branchstop_gradient'd; scored on the fewest views/samples that suffice
Related Skills
rae:jax-config— static/dynamic config separation and pytree registration; the design-time complement to this skill's diagnosis recipes (a correctly-classified config prevents most retrace leaks up front).eqx-porting(project skill in repos that port PyTorch models to JAX/Equinox) — porting workflow, weight conversion, and parity testing; use this skill when a ported model's inference loop OOMs or retraces.
JAX Multi-GPU: Unreplication, Collective Hangs, and Whether Sharding Pays
Overview
Data-parallel multi-device training fails in ways single-device training does not, and the failures are quiet. Three families cover most of it:
- Unreplication / placement bugs. Pulling one replica of a replicated pytree back to one device is the most common multi-GPU operation and the easiest to get wrong. Done naively it leaves a full copy resident on every device, permanently — or, under the modern sharding API, silently returns the wrong numbers.
- Collective hangs. When one replica dies, the others block forever. The process stays alive, holds its GPUs, writes nothing, and never returns an exit code.
- Scaling that silently changes the experiment. Every step-counted hyperparameter means something different when a step consumes N times the data.
Underneath all three: your usual instruments lie on multi-GPU. Process existence, exit codes, log tails, GPU-utilisation snapshots and even the op name in the error each report "healthy" — or point at the wrong place — for a job that has been dead for hours.
Evidence tags. Claims below are marked [M] measured (reproduced on jax/jaxlib 0.10.1, CUDA plugin, in an environment you can re-run the snippet in), [C] cited (upstream docs or source, not re-measured here), or [F] field report (observed once on one project's hardware — directionally true, do not treat the number as a constant).
0. pmap is in maintenance mode — know which API you are in
pmap is in maintenance mode — not formally deprecated (it emits no DeprecationWarning
[M]), but the JAX 0.8.0 changelog is explicit [C]:
"
jax.pmapis in maintenance mode and we encourage all new code to usejax.shard_mapdirectly."
and jaxlib 0.10.1's own jax.pmap docstring opens with [M]:
"Old way of doing parallel map. Use
jax.shard_mapinstead." "pmapis now implemented in terms ofjitandshard_map." — with a pointer to the pmap migration guide.
Note the recommended successor is shard_map, not "jit + sharding" generically.
JEP 14273 gives the reason that matters
here [C]: "the sharding of pmap outputs can't be expressed as Shardings/OpShardings, due
to pmap's stacking rather than concatenative semantics." That mismatch is the root of §1.
[M] jax.device_put_replicated and jax.device_put_sharded are already gone from
jax 0.10.1; code that calls them will AttributeError. The replacement is an explicit mesh:
devices = jax.devices(); n = len(devices)
mesh = jax.sharding.Mesh(np.asarray(devices), ("devices",))
row = jax.sharding.NamedSharding(mesh, jax.sharding.PartitionSpec("devices"))
replicated = jax.tree.map(
lambda x: jax.device_put(jnp.broadcast_to(x[None], (n, *jnp.shape(x))), row), tree)
[M] Also present in 0.10.1 and worth knowing before you write a shim: jax.shard_map,
jax.smap, jax.make_mesh, jax.set_mesh, jax.sharding.reshard,
jax.sharding.make_single_device_sharding, jax.sharding.AxisType / auto_axes /
explicit_axes.
Which API you are in changes the shape of the bug, not whether you have one — see §1.4.
Nothing in §1 or §2 is pmap-specific: pmap now runs on the same jit+shard_map
machinery, and the residency behaviour reproduces identically through a plain
NamedSharding [M].
1. Unreplication: x[0] does not do what it looks like
Under pmap (or a pmap-shaped mesh layout) parameters carry a leading device axis. To
checkpoint, validate, or render you need one replica on one device. The obvious code is wrong:
single = jax.tree.map(lambda x: x[0], replicated) # WRONG
x[0] selects replica 0's values, but the output inherits the input's multi-device
sharding: PartitionSpec("devices") becomes PartitionSpec() on the same mesh — i.e. fully
replicated, a complete copy resident on every device in the job. It is not "uncommitted and
therefore free"; the bytes are really there.
Upstream documents exactly this, in the pmap migration guide's appendix [C]:
"JAX does not track semantic replication. It does not 'know' that the shard on device 1 is identical to the shard on device 0. Therefore, when you call
x[0], JAX must satisfy the strict semantics of array indexing… This triggers a global gather (or broadcast) of the entire array to all devices before slicing."
and explains why the idiom used to be free: PmapSharding "had a fast-path optimization in
jax.Array's __getitem__ allowing it to return an array with a SingleDeviceSharding" — a
fast path that was removed along with PmapSharding in jax 0.10 [C], because it "breaks
the semantics of array indexing". So code that was correct-and-cheap under old pmap became
correct-and-ruinous without a single line changing.
1.1 Prove it in twelve lines (no GPU needed)
[M] Runs on CPU with fake devices; the semantics are platform-independent.
import math, jax, jax.numpy as jnp, numpy as np
# XLA_FLAGS="--xla_force_host_platform_device_count=4" JAX_PLATFORMS=cpu
D = jax.devices(); mesh = jax.sharding.Mesh(np.asarray(D), ("d",))
row = jax.sharding.NamedSharding(mesh, jax.sharding.PartitionSpec("d"))
def resident(a): # device bytes actually held, summed over every device it sits on
return math.prod(a.sharding.shard_shape(a.shape)) * a.itemsize * len(a.sharding.device_set)
x = jax.device_put(jnp.zeros((len(D), 1024, 1024)), row) # one replica per device
for name, a in (("x[0]", x[0]), ("device_put(x[0], D[0])", jax.device_put(x[0], D[0]))):
print(f"{name:24s} logical {a.nbytes/2**20:5.1f} MiB resident {resident(a)/2**20:5.1f} MiB "
f" devices {len(a.sharding.device_set)}")
x[0] logical 4.0 MiB resident 16.0 MiB devices 4
device_put(x[0], D[0]) logical 4.0 MiB resident 4.0 MiB devices 1
jax.debug.visualize_array_sharding says the same thing in one line [M]:
x x[0] device_put(x[0], D[0])
┌──────────┐ ┌───────────┐ ┌───────┐
│ CPU 0 │ │CPU 0,1,2,3│ │ CPU 0 │
├──────────┤ └───────────┘ └───────┘
│ CPU 1 │ …
1.2 The fix, and why each guard is load-bearing
def unreplicate_to_single_device(tree, device=None):
"""Take replica 0 AND commit it to one device."""
target = device if device is not None else jax.devices()[0]
def take_replica_zero(leaf):
# Non-array and 0-d leaves carry no device axis. `3.0[0]` is a TypeError and
# `jnp.float32(3.0)[0]` is an IndexError, so an unguarded tree.map over a real
# Equinox module or config pytree crashes before it ever leaks anything. [M]
if not hasattr(leaf, "shape") or getattr(leaf, "ndim", 0) == 0:
return leaf
return jax.device_put(leaf[0], target)
return jax.tree.map(take_replica_zero, tree)
jax.device_put(x, device)is the whole fix. Its docstring: "If given, then the result is committed to the device(s)" [C]. Verified to produceSingleDeviceSharding(device=0)[M].- The pin must happen outside
jit. Adevice_putinside a traced function does not commit: with aDeviceor aNamedShardingit is silently ignored and the result comes back on all devices [M] (upstream: jax#6040, "sticky device (device_put) ignored in jit context"); with aSingleDeviceShardingit raisesValueError: … device_put inside jit with device ids [0][C]. A maintainer's summary [C]: "the recommendation is to usedevice_putoutsidejit, and if you want to constrain shardings of an intermediate inside jit, thenwith_sharding_constraintis the API." Single-device extraction is strictly a top-level operation —jit(..., out_shardings= SingleDeviceSharding(d0))under a multi-device mesh raises the incompatible-devices error too [C]. - Per leaf, not per tree.
jax.device_put(whole_tree, device)raisesTypeError: Argument 'x' of type <class 'str'> is not a valid JAX typeon the first non-array leaf [M] — so you need the map and the guards anyway, and mapping keeps the replicated intermediate one leaf wide instead of one tree wide. donate=Truedoes not rescue you.jax.device_put(x[0], D[0], donate=True)left the source array not deleted (is_deleted() == False) [M, CPU backend]; donation is documented as best-effort. Drop the reference instead.
1.3 Two symptoms, one bug
Loud. Mixing a properly pinned array with a naively "unreplicated" one inside a jit
fails immediately [M]:
ValueError: Received incompatible devices for jitted computation.
Got argument a … with device ids [0] on platform CPU and
argument b … with device ids [0, 1, 2, 3] on platform CPU
This is the good outcome — it names the bug. In practice it appears wherever a host-side
consumer combines a device-0 batch slice with a replicated model: an eager sanity forward, a
lax.cond / select_n over a validity mask, an overlay renderer [F].
Quiet. If nothing ever mixes device sets, the copies just sit there and shrink the margin until something else fails. [F] On a 2-GPU run, params + Adam state unreplicated this way charged ~10.5 GiB to each card, half of it on the card that never reads the result.
The leak is the Python local, not the copy
The copy is transient only if you let it be. This pattern pins memory for the rest of the run:
if global_step % save_every == 0:
model_to_save = unreplicate(model) # rebound only on the NEXT save
opt_to_save = unreplicate(opt_state) # ...so both stay live in between
save(model_to_save, opt_to_save)
Each local holds its arrays until reassigned, so between one save and the next the copy stays live.
Do not over-claim the magnitude — one field investigation measured it and got a much smaller number than it first estimated [M]. Bracketing the allocator cap on a trainer whose step wanted ~80.7 GB of a ~135 GB pool, with save / overlay / validation all firing on the same step, the window in which a boundary tipped an otherwise-passing step was under ~1.5 GB — not the 12-25 GB the docstring arithmetic above suggests. The persistent cost was small; the boundary behaved as a brief transient spike on a margin that was already knife-edge (the production failure map read 59% free against 58.4% needed). Its earlier "~25 GB/card pinned for the whole run" estimate was retracted.
The lesson generalises past this bug: the residency mechanism is real and worth fixing, but on a system this tight the thing that eventually kills you may be slow allocator drift over tens of thousands of steps rather than any single branch. In that investigation the production memory fraction was GREEN in a fresh short run at the same setting the dead arms had used, and the arms had survived several saves and a validation before dying — so no single event explained it. Fix the placement because it is correct; do not promise a specific margin recovery unless you measured one.
Rules
- Route every unreplication through one helper that commits to a device. Never open-code
x[0]. delthe locals at the end of the branch, or scope them in a function so they fall out.- Unreplicate the buffers too. Large constants replicated at setup (mesh templates, lookup
tables, per-step constants) carry a device axis as well. A host-side consumer receiving a
(D, G, V, C)array where it expects(G, V, C)fails deep inside an einsum, far from the cause.
1.4 Under the modern API the same mistake is worse
[M] In a jax.jit + NamedSharding data-parallel program, parameters are replicated
(PartitionSpec()), not stacked — there is no leading device axis to index. x[0] then
returns the first element of the data:
p = jax.device_put(jnp.arange(4.), NamedSharding(mesh, P()))
p.shape # (4,) — no device axis
p[0] # 0.0 — a DATA element, not a replica. Silently wrong.
The fix needs no indexing at all, and it is the single most useful line in this skill:
single = jax.device_put(params, jax.devices()[0]) # or
single = jax.device_put(params, jax.sharding.SingleDeviceSharding(jax.devices()[0]))
Both produce SingleDeviceSharding [M]. So: migrating off pmap removes the residency
trap entirely — but only if you also delete the x[0]. Porting the indexing forward converts
a memory bug into a correctness bug. Grep for [0] in every unreplication path as part of the
migration.
The mode matters too [M]. On an Auto mesh (what Mesh(devices, ("d",)) gives you by
default) x[0] silently performs the gather described above. Give the mesh Explicit axis
types and the same expression raises instead:
ShardingTypeError: … out sharding could not be resolved unambiguously
(or would require collectives on inputs). Use .at[...].get(out_sharding=) …
That is a good reason to adopt explicit axis types on new code: the failure becomes loud. In
Explicit mode the resharding API is jax.sharding.reshard / jax.reshard, and
with_sharding_constraint degrades to an assertion rather than a constraint [C].
1.5 Do not unreplicate at all if your checkpointer is distributed
§1 assumes a single-writer serialiser that reads one complete tree from one device. That is a
property of your checkpointing code, not a law. A distributed checkpointer (Orbax) inverts it:
"each process first writes its own data independently to its corresponding subdirectory" and
then merges metadata, and save is "called on all available controller processes" [C].
device_get / process_allgather / gathering to one device appear nowhere in its idiom.
So decide once, and write it down: either you gather to one device and pay §1's attention tax, or you write shards in place and never unreplicate. Doing the first while believing you are doing the second is how the residency bug survives review.
1.6 Assert placement; don't hope for it
Put this in a test, not in a comment. It runs on CPU fake devices in seconds.
def assert_single_device(tree, what):
for path, leaf in jax.tree_util.tree_leaves_with_path(eqx.filter(tree, eqx.is_array)):
n = len(leaf.sharding.device_set)
assert n == 1, f"{what}{jax.tree_util.keystr(path)} lives on {n} devices"
resident = sum(resident_bytes(l) for _, l in ...)
assert resident == sum(l.nbytes for _, l in ...) # no hidden replicas
Then take the same measurement across a whole save cycle: building the checkpoint copies
must add exactly one tree's worth of device bytes to jax.live_arrays(), and dropping them must
return you to the baseline. That catches the lifetime half of the bug, which the per-leaf
assertion cannot see.
Gotcha [M]: compute resident bytes from
arr.sharding.shard_shape(arr.shape) × itemsize × len(arr.sharding.device_set), not from
arr.addressable_shards — reading addressable_shards materialises single-device Array
objects that then show up in jax.live_arrays() and corrupt the very accounting you are doing.
Use x.sharding.num_devices == 1 as the assertion. is_fully_replicated is not a
single-device test — it is True both for a one-device array and for an N-device replicated one
[M], which is precisely the state you are trying to detect. x.devices() works but is
undocumented [C]. Other useful diagnostics: x.sharding, x.committed, x.device (returns a
Device for single-device arrays and a Sharding otherwise), len(x.addressable_shards), and
jax.debug.visualize_array_sharding(x) [M]. And
jax.transfer_guard("disallow") names the destination sharding of any transfer it blocks,
which is a cheap way to catch an unintended host round-trip [M]:
JaxRuntimeError: INVALID_ARGUMENT: Disallowed host-to-device transfer:
aval=ShapedArray(float32[3]), dst_sharding=SingleDeviceSharding(device=CpuDevice(id=0) …)
1.7 Why it surfaces as a giant unrelated allocation
The leaked memory does not fail on its own — it shrinks the margin until something else fails. [F] In one case the failing request was the pmapped train step's peak workspace, ~58% of the whole pool, dying with 59% free. Diagnostic tells:
- the request size varies with model/loss configuration → it is a computation's workspace, not a fixed-size copy of parameters
- the allocator map shows plenty free (
****____...____****) → not exhaustion, a margin problem - the op name in
requested by opis blank → no attribution; static reading beats more runs
Do not chase this by shrinking the memory fraction. That makes the failure move earlier, not away — the same request then fails at first-step validation instead of at step 40,000.
Related trap: JAX dispatch is async, so a device error is raised at the next synchronisation
point and blamed on whatever executable is being awaited there, which is often a small,
innocent function. [F] A failing linear-algebra op inside a pmapped train step surfaced as
JaxRuntimeError … [executable_name='jit_extract_single_device_model'] — a function that is a
pure per-leaf slice and contains no such op. Before believing the named op, dump the HLO of the
step (--xla_dump_to) and enumerate its custom calls, or force synchronisation
(block_until_ready) immediately after each stage to localise it.
2. Collective hangs: one replica dies, the job does not
When a replica OOMs or aborts mid-collective, the surviving replicas block in the rendezvous forever:
E rendezvous.cc:116] [rank=1] Acquire clique: devices=2:[0,1]; local_participants=2
... has been waiting for 10 seconds and may be stuck.
Expected 2 threads to join the rendezvous, but not all of them arrived on time.
The warning prints once. Then nothing. The process stays alive, holds every GPU, writes no metrics and never exits. XLA's default is to warn and then wait forever.
Note the vocabulary: in a single-process job (one process, several local devices, no
jax.distributed) "rank=0/rank=1" are XLA's per-device threads, not processes. There is no
second process to kill.
2.1 Arm a timeout — but arm the RIGHT one
One function (xla/service/rendezvous.cc) emits that message for four different rendezvous
points, each with its own timeout pair. The name in the message is the whole triage [C]:
| name in the log | terminate flag | default |
|---|---|---|
Acquire clique / Initialize clique | --xla_gpu_nccl_termination_timeout_seconds | −1 = never |
first call … collective operation | --xla_gpu_first_collective_call_terminate_timeout_seconds | 40 s |
thunk initialization completion … | --xla_gpu_executable_terminate_timeout | 30 s |
| whole-execution deadlock (not a rendezvous) | --xla_gpu_execution_terminate_timeout | inf |
So an Acquire clique hang lasts forever by default while a first-collective-call hang already
self-terminates. [F] That is why, on an Acquire clique stall, only
--xla_gpu_nccl_termination_timeout_seconds ended the process and the other *_terminate_* flags
looked inert — they arm rendezvous the job never reached. Match the flag to the name in the log;
do not bisect the family.
A set that parses on jaxlib 0.10.1 [M] and covers all four:
export XLA_FLAGS="\
--xla_gpu_nccl_termination_timeout_seconds=600 \
--xla_gpu_execution_terminate_timeout=10m \
--xla_gpu_execution_progress_tracking=16 \
--xla_gpu_executable_warn_stuck_timeout=60 --xla_gpu_executable_terminate_timeout=600 \
--xla_gpu_first_collective_call_warn_stuck_timeout_seconds=60 \
--xla_gpu_first_collective_call_terminate_timeout_seconds=600"
Four things to know before you touch these — the details, defaults and verbatim help strings are
in references/collective-hang-toolkit.md:
- The terminate timeout dumps every thread's stack (
DumpAllThreadStacksToLog) before itsLOG(FATAL)[C]. That is the best artefact you will get; an externalkillgets you none. - A malformed flag VALUE and an unknown flag NAME are different fatals, and both kill the job
at first device use with a
F…line that reads like a driver fault [M]. The single most expensive instance:--xla_gpu_execution_terminate_timeouttakes a duration string (10m,600s) — passing=60fails, and reading that failure as "not a flag" discards the one knob whose stated purpose is "to prevent undetected deadlocks". NCCL_TIMEOUT/NCCL_ASYNC_ERROR_HANDLING/NCCL_BLOCKING_WAITare PyTorch variables. Absent from NCCL's documented variable list [C] and from the bundledlibnccl.so.2[M]; in a JAX job they do nothing. UseNCCL_DEBUG/NCCL_DEBUG_SUBSYS/NCCL_DEBUG_FILE.- Not every stuck warning is a hang. The same code logs "Warning above was a false-positive.
Perhaps the timeout is too short" when threads merely arrived late [C] — uneven per-device
work under
pmapproduces exactly this. Raise the warn timeout; do not chase a bug.
When to set them. XLA_FLAGS is parsed once, at the first XLA backend initialisation —
not at import jax [M]: with a bogus flag set, import jax succeeds and the abort only
arrives when the first array touches a device. Setting it before importing jax is still the right
rule, but audit the guard from the program's own log of its effective XLA_FLAGS, never from
import order. Also drop RLIMIT_CORE to 0 first — the abort writes a core dump, one measured
at 17.5 GB into a repo root [F], unreadable without driver symbols.
2.2 A dead process is not a released process
[M/F] An exception escaping main does not reliably end a multi-device job: normal
interpreter shutdown joins the PJRT worker threads, and a thread parked in a rendezvous never
joins. A run that "crashed" can hold every GPU indefinitely. Exit hard:
def run_entrypoint(main):
try:
main()
except BaseException:
traceback.print_exc()
sys.stderr.flush(); sys.stdout.flush()
os._exit(1) # sys.exit() would block joining a parked PJRT thread
2.3 Detecting a hang: the only signal that does not lie
| signal | verdict |
|---|---|
| process exists | useless — a hung job lives forever |
| exit code | useless — never arrives |
| GPU memory allocated | useless — a hung job holds it all |
| GPU utilisation snapshot | useless — a healthy train loop reads 0% between steps |
| metrics-file mtime | misleading — healthy runs batch writes and can look 80+ min stale |
| CPU-tick delta | reliable — /proc/<pid>/stat fields 14+15 (utime+stime) |
[F] A hung process burned ~1 tick per 15 s with both GPUs at 0% util while still holding 165 GB and 131 GB; a healthy trainer burns thousands in the same window. The regimes are three orders of magnitude apart, so the threshold is not delicate.
Three properties a watchdog needs that the tick delta alone does not give you:
- Debounce. Require several consecutive quiet polls (e.g. 40 × 15 s). One quiet poll is a long checkpoint write or a host-side pause, not a hang.
- A startup grace period. Shuffle-buffer fill plus compilation can exceed 20 minutes; a watchdog armed at t=0 kills healthy launches.
- A fast-failure cap. A death inside the grace period is a launch failure (bad config, OOM at step 1), not a stall. Stop relaunching after a few.
Prefer an in-process heartbeat to an external prober. A daemon thread that watches a
beat(position) timestamp knows where the loop was, and can os._exit() — releasing the GPUs
even though a PJRT thread will never join. Layer it: (1) the XLA termination flag for rendezvous
stalls, (2) the heartbeat watchdog for every other stall, (3) the hard-exit entrypoint of §2.2.
Before you kill a hung run, ask it where it is. Register
faulthandler.register(signal.SIGUSR1, all_threads=True, chain=False) at startup; then
kill -USR1 <pid> dumps every thread's stack to stderr without killing the process. That dump is
usually the only record of where a hang was. (chain=False matters: chaining re-raises to the
default handler, which terminates.)
2.4 Multi-process adds a heartbeat — but not a cure
If you run multi-process, jax.distributed does "health checking, ensuring that all processes
shut down if any process dies" [C]. Read the default carefully anyway — JAX's
fault tolerance page [C]:
"By default, if a process fails while participating in a distributed computation (like
jnp.sum), then the rest of the processes participating in the computation will get stuck forever."
Heartbeats fate-share the processes; they do not unwedge an in-flight collective. You still
want §2.1. The knob is heartbeat_timeout_seconds (default 100) — not heartbeat_timeout
[M] — and the survivors' fatal message carries the most useful operational hint in the stack:
"Disable Python buffering, i.e. python -u, to be sure to see all the previous output" [C].
Without -u the dying process's traceback, the only thing that says why, is lost unflushed.
Prevention beats detection: "it's crucial that all processes… run the same computation in the
same order… beware of any control flow that depends on jax.process_index() and includes
communication" [C]. multihost_utils.assert_equal on shapes/step counters/config hash before
entering a jit, plus named sync_global_devices barriers to bisect divergence, is the documented
mismatch check. Single-process pmap on local devices gets none of this — hence §2.1–2.3.
For the full API, the coordinator's own error strings, live inspection of a wedged job (NCCL RAS
on localhost:28028, gdb thread apply all bt, profiler TraceMe stall counts, per-rank HLO
diffing) and what upstream issues actually say, see
references/collective-hang-toolkit.md.
3. Scaling a recipe without changing the experiment
Every step-counted quantity must be divided
If a step consumes N times the samples, everything counted in steps reaches its boundary on N times the data. Halving two of three is worse than halving none, because the arm looks subtly wrong rather than obviously wrong.
steps_per_epoch ÷ N
unfreeze/freeze schedules ÷ N
lr_decay_updates / cosine horizon ÷ N <-- the one that gets missed
warmup_steps ÷ N (usually)
eval/checkpoint cadences ÷ N (if you want them at the same data volume)
Missing the LR horizon means the schedule completes on N× the data, so at any matched sample count the arm sits at a higher LR — less decayed and often a scaled-up peak. It presents as "multi-GPU is mysteriously less sample-efficient".
Verify from the run's own effective config, not the launch flags. Whether a resume honours a
config file is a property of your trainer — many rebuild config from the checkpoint and re-apply
only a whitelist of CLI overrides, so a passed --config is silently ignored. Read the value the
process logs at startup (or logs to your experiment tracker), never the command line.
Divisibility is a hard startup gate
Every batch bucket's B must divide the device count — including eval buckets, which are easy
to forget and fail only at startup on the larger device count.
Decide whether sharding pays before you build for it
Two numbers settle it, and neither is throughput.
Critical batch size. Above it, more parallelism buys wall-clock and loses sample efficiency. If your critical batch fits on one device, more devices cannot help — and if data is your binding constraint rather than time, going past it is strictly bad.
The standard model is McCandlish, Kaplan & Amodei, An Empirical Model of Large-Batch Training
(arXiv:1812.06162) [C]. Its printed tradeoff law is
(S/S_min − 1)·(E/E_min − 1) = 1 with B_crit ≡ E_min/S_min, equivalently (Appendix D, fixed
batch) the more usable S/S_min = 1 + B_crit/B and E/E_min = 1 + B/B_crit. At exactly
B = B_crit you pay 2× the minimum steps and 2× the minimum data — a turning point, not a
free lunch. Two conditions a naive fit violates [C]: re-tune the LR at every batch size
before fitting (and fit in log space), and remember B_crit "typically increases by an order of
magnitude or more over the course of training" — so it exists only relative to a stated
performance target.
You can measure it instead of fitting it, almost for free. The gradient-noise estimator needs only the gradient norm before and after the cross-device average [C]:
S ≡ (|G_small|² − |G_big|²) / (1/B_small − 1/B_big) # B_small = per-device, B_big = global
— "we can compute these with minimal effort by computing the norm of gradient before and after
averaging between devices." Two extra scalars in a pmap/shard_map step, and you get B_crit
from one run instead of a sweep. Say measured if you did this and curve fit if you fitted
the tradeoff law to (arm, target) observations; they are different claims.
Learning rate does not scale with a single rule — it scales with an exponent that depends on
the optimizer, in the saturating form ε(B) = ε*/(1 + B*/B)^α: α = 1 (linear) for
SGD/momentum, 0.5 < α < 1 for Adam/RMSProp, α = 0.5 for the classic variance-preserving √ rule
[C]. Because it saturates, an unbounded lr *= sqrt(ratio) overshoots at large B. Warmup is
not optional and should be gradual, because "the linear scaling rule breaks down when the
network is changing rapidly, which commonly occurs in early stages of training" [C]. The
largest study of this (Shallue et al., 35 workloads) found the optimum "did not always follow
either a linear or square root scaling heuristic" and that LR and momentum must be tuned
jointly [C] — so treat the rule as a starting point for a re-tune, never as a substitute.
Two traps around the comparison itself [C]:
- The budget is not neutral. An epoch/example budget only ever probes the perfect-scaling regime and favours small batches; a step budget favours large ones. Say which you used.
- Check you are not simply IO-bound first. Tuning Playbook: "if the batch size doubles, the training throughput should also double… If this is not the case then the training pipeline has a bottleneck such as I/O or synchronization between compute nodes." Its usual conclusion is worth hearing before you build anything: "the largest batch size supported by the available hardware will be smaller than the critical batch size… a good rule of thumb is to use the largest batch size possible." (Gradient accumulation buys none of this — it "does not provide any throughput benefits".)
Citations, exact equations and the recent CBS literature: references/scaling-and-batch-size.md.
Decompose any speedup you quote. "2 GPUs gave 1.71×" is usually two changes at once — the extra devices and a bigger batch. Measure the batch increase on one device first:
1.71 = 2 (the doubled batch) / 1.16 (the sharding tax)
[F] That decomposition showed the second device contributed a 16% overhead and nothing else, and that the whole useful batch range fitted on a single card. Comparing an N-GPU arm at batch 2B against a 1-GPU arm at batch B measures the batch, not the hardware.
An N-GPU arm is not the same objective as a 1-GPU arm
- Masked means. Gradients are
pmean'd, but a loss term computed assum / max(count,1)makes mean-of-per-device-means ≠ global masked mean whenever counts differ across shards. - Percentiles are wrong. A
pmeanover per-device percentiles is not a global percentile. Means are fine; treatp50/p90/p99as approximate. - Contrastive losses change meaning. Gathering across devices changes the negative-pool size, which changes the objective. Check whether your resume path force-enables such a gather — resume and fresh launch then optimise different things.
- The random stream is not device-count invariant. The standard idiom derives a per-device key
inside the mapped function,
fold_in(rng, step)thenfold_in(step_key, lax.axis_index(...))(axis_indexis free — it lowers toReplicaIdplus arithmetic, no communication [C]). It is reproducible at a fixed device count and not across a change in device count, so a 2-GPU and a 4-GPU arm see different augmentation/dropout noise even at matched global batch. Note this is community practice: no official JAX page documents PRNG handling across devices [C, negative]. Two rules regardless: init must use an identical key on every device or the replicas diverge on step 0; dropout/augmentation must use a varying key or you lose batch diversity. - There is a noise floor even at N=1. [F] Two runs of the same step implementation with
identical model/state/batch/key diverged after two steps: ~4e-7 absolute on parameters, ~1e-3
relative on losses. Run an A-vs-A control to establish that floor before reading any small
1-GPU-vs-N-GPU gap as signal. OpenXLA's determinism page
names two GPU sources — autotuning picking a different winning kernel across compiles
(pin with
--xla_gpu_autotune_level=0or persisted AOT autotune results) and nondeterministic ops accumulating via atomics — and offers--xla_gpu_exclude_nondeterministic_opsat "substantial runtime throughput loss", which hard-fails compilation for ops with no deterministic lowering [C].--xla_gpu_deterministic_opsalso exists in the plugin [M] but is absent from that page [C, negative]. - Do not say "NCCL is nondeterministic". The reduction tree is fixed at communicator init, so an all-reduce is generally run-to-run reproducible on an unchanged configuration; what it is not is invariant to configuration — change the device count, rank order, algorithm, protocol or message size and float non-associativity gives you different bits [C, inferred from NCCL issue reports; no NVIDIA guarantee found]. For 2-GPU vs 4-GPU that is the operative statement: the arms are not bitwise comparable even with identical seeds and data order.
4. Instruments and knobs that behave differently on several devices
Buffer donation matches on the sharded aval
Donation (jax.jit(donate_argnums=...), eqx.filter_jit(donate=...)) aliases an input buffer to
an output of the same shape and dtype. On several devices the match is made on the per-device
shard, not the global shape [C, read from jax/_src/interpreters/mlir.py]. Consequences
that produce no error at all:
- Same global shape but a different sharding → different shard shapes → no aliasing, no
memory saving, only
UserWarning: Some donated buffers were not usable: …. - Memory kinds must match (
devicevspinned_host); the source comment is explicit that cross-memory donation is not supported yet. - An
AUTO-layout input donated toward a concrete-layout output raises. - Keyword arguments are never donated —
f(params=p)silently donates nothing [C]. jit+shard_map+donate_argnumswas genuinely broken until jax 0.4.23 (jax#15368, fixed 2024-02) [C].
Because every one of these degrades to a warning, promote it in CI:
warnings.filterwarnings("error", message="Some donated buffers"). Otherwise a sharding change
silently removes an optimisation you are still budgeting memory for.
jax_log_compiles misses pmap — use jax.monitoring and normalize _device_put_sharded vs. 'i'-mesh
[M] Under modern JAX (>= 0.4.30 / 0.10+), jax_log_compiles=True does not emit log
records for jax.pmap compilations. Always attach a jax.monitoring listener on
"/jax/core/compile/backend_compile_duration" (rae-jax-profiling §10) to count every jit,
pjit, shard_map, and pmap backend compile.
Furthermore, legacy jax.device_put_replicated(x, devs) tags leaves with
NamedSharding(mesh={'_device_put_sharded': N}), whereas jax.pmap(..., axis_name='i') returns
leaves with NamedSharding(mesh={'i': N}):
- Epoch-2 recompile: Epoch 1 compiles
pmapfor_device_put_shardedinputs, and Epoch 2 recompiles the entirepmapstep (~40–60 s) because the carry returned by Epoch 1 now has mesh'i'. - Every-eval recompile with
donate_argnums: If a single-device evaluator uses@functools.partial(jax.jit, donate_argnums=(0,))oneval_statewhile consuming unreplicated parameters fromtraining_state, any leaf whose sharding flips between_device_put_shardedand'i'across calls causes the evaluator to recompile on every single evaluation (~30 s×100evals =50 minwasted). - Fix: Normalize initial replicated state through a 1-step
pmap(lambda s: s, axis_name='i')identity (or construct all arrays on a single explicitMesh), strip weak types, and commit unreplicated evaluator inputs viaunreplicate_to_single_device(§1.2) before calling anydonate_argnumsevaluator.
Catch accidental host round-trips with the transfer guard
jax.transfer_guard [C] takes
allow / log / disallow / log_explicit / disallow_explicit, plus directional variants
(jax_transfer_guard_host_to_device, _device_to_device, _device_to_host). Two properties make
it the right tool here:
"disallow"blocks only implicit transfers — a strayfloat(loss),print(array)ornp.asarray(x)inside the step — while still allowing explicitdevice_put/device_get, so checkpointing keeps working. Usedisallow_explicitonly for the innermost loop.- The error names the destination sharding, which is exactly what you want when diagnosing
placement [M]:
INVALID_ARGUMENT: Disallowed host-to-device transfer: aval=…, dst_sharding=SingleDeviceSharding(…)
Caveat [C]: the context manager is thread-local and "a newly spawned thread will use the global option" — a guard set in the training thread does not cover a background prefetch thread. Set the global flag if you mean the whole process.
jax.debug.print is not free on several devices
Under pjit/sharded jit, "a global synchronization occurs that will materialize values on a
single device"; under pmap the prints "might be reordered" and ordered=True raises
[C]. A debug print left in a sharded step costs a device-wide sync every step. (A plain Python
print under jit is free precisely because it only runs at trace time.)
Profiling
jax.profiler.trace("/tmp/prof") around a block_until_ready()'d call — the doc is explicit
that without it async dispatch ends the trace before the work runs [C]. For a job already
running, jax.profiler.start_server(9999) plus python -m jax.collect_profile <port> <ms> (or
XProf's capture-by-address) beats restarting it. The TensorBoard plugin is now xprof; HLO Op
Stats and the trace viewer are where all-reduce/all-gather time actually shows up.
Negative finding [C]: the JAX profiling page contains no multi-process guidance at all —
single-host multi-GPU is covered (each local device is its own row), multi-process is not, and you
fall back to per-process capture.
Knobs upstream actually recommends
From GPU performance tips [C]:
- Single-host multi-GPU NCCL tuning:
NCCL_LL128_BUFFSIZE=-2,NCCL_LL_BUFFSIZE=-2,NCCL_PROTO=SIMPLE,LL,LL128— intra-node only, "don't seem useful for multi-host communication yet". --xla_gpu_enable_latency_hiding_scheduler(default off) to overlap collectives with compute. It changes the live-range structure, so re-check peak memory when you flip it.- Collective combiner thresholds default to 256 bytes and should be raised
"high enough so as to combine at least a Transformer Layer's weight":
--xla_gpu_all_gather_combine_threshold_bytes,--xla_gpu_reduce_scatter_combine_threshold_bytes,--xla_gpu_all_reduce_combine_threshold_bytes. - Multi-process: one process per GPU, not one per node.
multihost_utils is for multi-process, not multi-device
jax.experimental.multihost_utils (sync_global_devices, process_allgather,
broadcast_one_to_all, assert_equal, host_local_array_to_global_array) is "Utilities for
synchronizing and communication across multiple hosts" [C]. On a single-host box with several
GPUs jax.process_count() == 1, every device is local, and none of it is needed — reaching for it
there is a common cargo-cult. When you are multi-process, assert_equal and
sync_global_devices are the cheap way to catch the mismatched-collective class of hang (processes
disagreeing about what to call, or in what order) before it becomes a rendezvous stall.
5. Operational habits that pay for themselves
- Never mutate a venv while runs are live. Replacing a package unlinks shared objects the running processes have mmap'd; they die silently, staggered, with no diagnostic. Clone the venv, mutate the clone, launch new runs against it.
- Never kill by pattern.
pgrep -f <name>matches your own command line, and on a shared box a name match can take out someone else's job. Resolve explicit PIDs (ps -eo pid,args), then signal those. - Stagger launches. Two simultaneous XLA compilations of a large program can push load high enough that something gets killed with no traceback. Start one, let it compile, then the next.
- Budget the cold start. Shuffle-buffer fill plus multi-device compilation can take 15–25 minutes before the first step [F]. Judging a launch at 20 minutes has twice produced a false "it hangs".
- Some warnings are permanent noise.
VMM cuMemCreate with FABRIC+POSIX_FD … CUDA_ERROR_NOT_PERMITTEDappears on every multi-GPU start under container capability limits and falls back automatically [F]. Confirm collectives work (psum+ a large all-reduce) once, then stop reading it. (Some donated buffers were not usableis not in this category — see §4.) - Grep for the mechanism, not the string. Command-buffer formation is a thunk-level pass:
optimized HLO contains
command_bufferzero times even when buffers are being built [F]. UseTF_CPP_VMODULE=command_buffer_conversion_pass=3. - Checkpoint atomically. Stage every member to
.tmpand rename, metadata last, so "metadata exists" implies a complete set. A non-atomic writer withkeep_last=1has a window (measured at 23–43 s on 12 GB checkpoints [F]) in which a death leaves zero resumable checkpoints.
Quick reference
| Symptom | Likely cause | Section |
|---|---|---|
| Alive, 0% util, no metrics, holds GPUs | collective hang after a replica died | §2 |
Acquire clique / rendezvous.cc / Expected N threads to join | same | §2 |
| Hung job survives a crash and keeps its GPUs | PJRT thread never joins at shutdown | §2.2 |
Received incompatible devices for jitted computation | one operand pinned, one still replicated | §1.3 |
| Huge OOM at checkpoint/validation boundary | unreplication residency ate the margin | §1 |
requested by op with a blank name | not attributable — read the code, don't rerun | §1.7 |
| Error blames a jit'd fn that has no such op | async attribution at the next sync point | §1.7 |
| Request size varies by model/loss config | it is a computation's workspace, not a copy | §1.7 |
Smaller MEM_FRACTION makes it fail sooner | margin problem, not a leak in that request | §1.7 |
Ported off pmap and the numbers changed | x[0] on a replicated (not stacked) array | §1.4 |
AttributeError: device_put_replicated | removed in jax 0.10 — build an explicit mesh | §0 |
Unknown flag in XLA_FLAGS at first device use | the flag name does not exist | §2.1 |
Check failed: tsl::Flags::Parse … usage: --foo=BAR | the flag exists; the VALUE is malformed | §2.1 |
| Warning says "unstuck … false-positive" | straggler, not a hang — raise the warn timeout | §2.1 |
ShardingTypeError … out sharding could not be resolved | x[0] on an Explicit-mode mesh | §1.4 |
NCCL_TIMEOUT etc. had no effect | PyTorch variables, absent from NCCL | §2.1 |
| N-GPU arm less sample-efficient than 1-GPU | a step-counted schedule was not divided | §3 |
| Startup: batch not divisible by device count | eval bucket, usually | §3 |
| Runs die silently just after a package change | venv mutated under live processes | §5 |
Some donated buffers were not usable on a sharded jit | donation matched on shard shape, not global shape | §4 |
| A sharded step got slow after adding a debug print | jax.debug.print forces a global sync | §4 |
Pre-flight checklist before launching on N devices
- Every unreplication goes through one helper that
device_puts to a device — no barex[0] - That helper guards non-array and 0-d leaves
- A test asserts single-device placement and total resident bytes across a save cycle
-
steps_per_epoch, LR horizon, warmup and freeze/unfreeze schedules all divided by N - Every batch bucket, including eval, is divisible by N
- Timeouts armed for the rendezvous you can actually hit, set before
jaxis imported and echoed in the log (--xla_gpu_nccl_termination_timeout_seconds,--xla_gpu_execution_terminate_timeout) - Every new
XLA_FLAGSentry smoke-tested against a one-line program that touches a device -
RLIMIT_COREdropped to 0;faulthandleron SIGUSR1 - Heartbeat watchdog with debounce + startup grace + fast-failure cap; entrypoint
os._exits - Checkpoints written atomically, metadata last
- The speedup you plan to quote is decomposed into batch effect vs sharding tax
- Critical-batch estimate says another device can help at all
Related skills
jax-memory-and-retracing— the single-device half of the memory story: retrace leaks, host-RAM OOM, allocator diagnosis. Read it first if the OOM is not at a checkpoint/validation boundary.jax-jit-and-fusion— jit boundaries,donate_argnums, and honest warm/block_until_readybenchmarking, which §3's speedup decomposition depends on.jax-config— static/dynamic config separation, so the schedule constants in §3 have one source of truth to divide.
References
references/collective-hang-toolkit.md— the full XLA timeout-flag table with defaults and verbatim help strings, the flag-naming traps, real vs PyTorch NCCL variables, NCCL RAS,jax.distributedfailure semantics, and how to inspect a job that is wedged right now.references/scaling-and-batch-size.md— critical-batch-size and LR-scaling literature with exact equations, the caveats that invalidate a naive fit, and recent work on how CBS actually scales.
Pose DataJoint Query Reference
CRITICAL: NEVER Modify Database Entries
DO NOT update, delete, or alter any DataJoint database entries. This includes:
update1(),delete(),drop()on any table- Modifying settings lookup tables (KinematicReconstructionSettingsLookup, ProbabilisticReconstructionSettingsLookup, KineticReconstructionSettingsLookup, KeypointSet, etc.)
- Altering any computed table entries
Database entries are shared state used by the entire lab. Changing a settings entry changes it for everyone and invalidates prior results computed with those settings.
Overview
The biomechanics pipeline has two parallel systems:
- Multi-Camera (MMC) - Lab-based, multiple synchronized cameras, uses
participant_id(string) - Monocular (PBL) - Phone-based portable recordings, uses
subject_id(integer)
Both produce kinematic outputs (qpos, joints, sites) but have different table hierarchies.
Quick Reference: Package Imports
# Shared: Video and 2D/3D pose estimation
from pose_pipeline.pipeline import Video, VideoInfo, TopDownPerson, LiftingPerson
# === MULTI-CAMERA (MMC) ===
from multi_camera.datajoint.sessions import Session, Recording, Subject
from multi_camera.datajoint.multi_camera_dj import (
MultiCameraRecording, SingleCameraVideo, PersonKeypointReconstruction
)
from body_models.datajoint.kinematic_dj import KinematicReconstruction
from body_models.datajoint.dataset import fetch_keypoints # For synchronized KR+keypoint fetch
# === MONOCULAR (PBL) ===
from portable_biomechanics_sessions.emgimu_session import (
Subject as PBLSubject, # Note: different from MMC Subject!
Session as PBLSession, # Note: different from MMC Session!
FirebaseSession
)
from body_models.datajoint.monocular_dj import MonocularReconstruction
Key Spaces (CRITICAL - Different Per Pipeline!)
Multi-Camera Key Space
# Session: participant_id is STRING
session_key = {'participant_id': '104', 'session_date': date(2023, 7, 21)}
# Video: video_project + filename
video_key = {'video_project': 'CLINIC_GAIT', 'filename': 'trial_001.27.mp4'}
Monocular Key Space
# Subject/Session: subject_id is INTEGER, project is part of key
subject_key = {'subject_id': 301, 'project': 'HLL'}
# Session adds timestamp
session_key = {'subject_id': 301, 'project': 'HLL',
'session_start_time': datetime(2024, 1, 15, 10, 30, 0)}
# AppVideo links to Video table
app_video_key = {**session_key, 'app_start_time': ...,
'video_project': 'HLL', 'filename': '0301_gait.mp4'}
DataJoint Operators
| Operator | Meaning | Example |
|---|---|---|
& | Restrict (filter) | Video & 'video_project="HLL"' |
* | Join tables | Session * Recording * MultiCameraRecording |
- | Set difference | Video - TopDownPerson (videos without poses) |
.proj() | Select attributes | Table.proj('field1', 'field2') |
Fetching Data
# fetch1() - Exactly ONE row (raises error if 0 or >1)
timestamps, qpos = (Table & key).fetch1('timestamps', 'qpos')
# fetch() - Multiple rows as arrays
all_keys = (Table & restriction).fetch('KEY') # List of dicts
values = (Table & key).fetch('field_name') # Numpy array
# fetch(as_dict=True) - Multiple rows as list of dicts
records = (Table & key).fetch(as_dict=True)
Multi-Camera (MMC) Queries
Count Videos in MMC Project
from pose_pipeline.pipeline import Video
from multi_camera.datajoint.multi_camera_dj import MultiCameraRecording, SingleCameraVideo
# Videos linked to multi-camera recordings
count = len(Video & SingleCameraVideo & (MultiCameraRecording & 'video_project="CLINIC_GAIT"'))
Count Sessions/Participants in MMC
from multi_camera.datajoint.sessions import Session, Recording
from multi_camera.datajoint.multi_camera_dj import MultiCameraRecording
import numpy as np
# Sessions for a participant (participant_id is STRING!)
count = len(Session & {'participant_id': '104'})
# Unique participants in a project
participants = np.unique(
(Session & (Recording & (MultiCameraRecording & 'video_project="CLINIC_GAIT"'))).fetch('participant_id')
)
print(f"Participants: {len(participants)}")
Get MMC Kinematic Reconstruction
from body_models.datajoint.kinematic_dj import KinematicReconstruction
from datetime import date
# CRITICAL: Always specify kinematic_reconstruction_settings_num!
key = {
'participant_id': '102',
'session_date': date(2023, 7, 21),
'kinematic_reconstruction_settings_num': 137 # REQUIRED!
}
timestamps, qpos, joints, sites = (KinematicReconstruction.Trial & key).fetch1(
'timestamps', 'qpos', 'joints', 'sites'
)
# qpos: (T, 41) joint angles in radians
# joints: (T, N_bodies, 3) body positions in meters
# sites: (T, N_sites, 3) marker positions in meters
Get 3D Triangulated Keypoints (MMC)
from multi_camera.datajoint.multi_camera_dj import PersonKeypointReconstruction
key = {
'video_project': 'CLINIC_GAIT',
'video_base_filename': 'trial_20231215_143022',
'reconstruction_method': 0 # 0=Robust Triangulation
}
keypoints3d = (PersonKeypointReconstruction & key).fetch1('keypoints3d')
# Shape: (T, N_joints, 4) - [x, y, z, confidence], units: mm
Monocular (PBL) Queries
Count Videos in Monocular Project
from portable_biomechanics_sessions.emgimu_session import FirebaseSession
# AppVideo is a Part table of FirebaseSession
count = len(FirebaseSession.AppVideo & {'video_project': 'HLL'})
print(f"HLL monocular videos: {count}")
Count Subjects in Monocular Project
from portable_biomechanics_sessions.emgimu_session import FirebaseSession
import numpy as np
# Get unique subject_ids for a project
subject_ids = np.unique(
(FirebaseSession.AppVideo & {'video_project': 'HLL'}).fetch('subject_id')
)
print(f"Subjects with HLL videos: {len(subject_ids)}")
Count Monocular Videos Processed with Kinematic Reconstruction
from portable_biomechanics_sessions.emgimu_session import FirebaseSession
from body_models.datajoint.monocular_dj import MonocularReconstruction
# Videos that have monocular reconstruction
processed = len(
FirebaseSession.AppVideo
& (MonocularReconstruction.Trial & {'video_project': 'HLL'})
)
print(f"HLL videos with monocular reconstruction: {processed}")
# Videos NOT yet processed
all_videos = FirebaseSession.AppVideo & {'video_project': 'HLL'}
unprocessed = len(all_videos - MonocularReconstruction.Trial)
print(f"HLL videos needing processing: {unprocessed}")
Get Monocular Kinematic Reconstruction
from body_models.datajoint.monocular_dj import MonocularReconstruction
from datetime import datetime
# Monocular uses subject_id (INTEGER) and project
key = {
'subject_id': 301,
'project': 'HLL',
'session_start_time': datetime(2024, 1, 15, 10, 30, 0),
'monocular_reconstruction_settings_num': 1 # Specify method
}
# Get all trials for this session
trial_keys = (MonocularReconstruction.Trial & key).fetch('KEY')
for trial_key in trial_keys:
timestamps, qpos, joints, sites, rnc = (MonocularReconstruction.Trial & trial_key).fetch1(
'timestamps', 'qpos', 'joints', 'sites', 'rnc'
)
# qpos: (T, 40) joint angles - monocular has 40 DOF (vs 41 for MMC)
# rnc: (T, 3) camera rotation vector from phone attitude
print(f"Video: {trial_key['filename']}, frames: {len(timestamps)}")
List Monocular Projects
from portable_biomechanics_sessions.emgimu_session import FirebaseSession
import numpy as np
projects = np.unique(FirebaseSession.AppVideo.fetch('video_project'))
print(f"Monocular projects: {projects}")
Fetching Synchronized Keypoints + Reconstruction (MMC)
When you need 2D keypoints aligned frame-by-frame with KR qpos, use fetch_keypoints:
from body_models.datajoint.dataset import fetch_keypoints as bm_fetch_keypoints
from body_models.datajoint.kinematic_dj import KinematicReconstruction
trial_key = {'participant_id': '104', 'session_date': date(2023, 7, 21),
'recording_timestamps': '2023-07-21 14:06:37'}
full_key = {**trial_key, 'kinematic_reconstruction_settings_num': 137}
qpos = (KinematicReconstruction.Trial & full_key).fetch1('qpos') # (T, 41)
timestamps, kp_raw = bm_fetch_keypoints(trial_key, only_detected=True)
# kp_raw: (C, T, 87, 3) — guaranteed qpos[i] matches kp_raw[:, i]
assert qpos.shape[0] == kp_raw.shape[1] # ALWAYS verify
Do NOT fetch keypoints via TopDownPerson * VideoInfo and match timestamps — this produces silent frame offsets. See rae:fetching-synchronized-data for the full pattern including camera parameter reordering and contiguous segment selection.
Shared Queries (Work for Both Pipelines)
Get 2D Keypoints (Standalone, No KR Alignment)
For keypoints aligned with KR qpos, use the synchronized pattern above.
For standalone 2D analysis (no KR alignment needed):
from pose_pipeline.pipeline import TopDownPerson
key = {
'video_project': 'HLL', # Works for any project
'filename': 'trial_001.mp4',
'video_subject_id': 0,
'top_down_method': 0 # 0=MMPose
}
keypoints = (TopDownPerson & key).fetch1('keypoints') # Shape: (T, N_joints, 3)
Get 3D Lifted Keypoints
from pose_pipeline.pipeline import LiftingPerson
key = {**video_key, 'video_subject_id': 0, 'top_down_method': 0, 'lifting_method': 1}
keypoints_3d = (LiftingPerson & key).fetch1('keypoints_3d') # Shape: (T, N_joints, 4)
Count All Videos by Project
from pose_pipeline.pipeline import Video
from collections import Counter
projects = Video.fetch('video_project')
for project, count in Counter(projects).items():
print(f"{project}: {count} videos")
Table Relationships
Multi-Camera Hierarchy
Subject (participant_id) ← STRING
-> Session (participant_id, session_date)
-> Recording -> MultiCameraRecording (video_project)
-> SingleCameraVideo -> Video
-> PersonKeypointReconstruction (3D triangulated)
-> SessionCalibration.Grouping
-> KinematicReconstruction (method 137)
-> KinematicReconstruction.Trial (qpos, joints, sites)
Monocular Hierarchy
Subject (subject_id, project) ← INTEGER + project
-> Session (subject_id, project, session_start_time)
-> FirebaseSession
-> FirebaseSession.AppVideo -> Video
-> FirebaseSession.PhoneAttitude (phone orientation)
-> FirebaseSession.Gyro/Accel/Mag (IMU data)
MonocularReconstruction (subject_id, project, session_start_time, method)
-> MonocularReconstruction.Trial (qpos, joints, sites, rnc)
-> FirebaseSession.AppVideo (links video)
Common Mistakes
| Mistake | Fix |
|---|---|
MMC: {'subject_id': 104} | Use {'participant_id': '104'} (string!) |
PBL: {'participant_id': '301'} | Use {'subject_id': 301} (integer!) |
Keypoints2D table | Use TopDownPerson for 2D keypoints |
| Missing method for KinematicReconstruction | Add 'kinematic_reconstruction_settings_num': 137 |
| Missing method for MonocularReconstruction | Add 'monocular_reconstruction_settings_num': 1 |
fetch(unique=True) | Use np.unique(table.fetch('field')) |
create_virtual_module() | Direct import from modules |
| Mixing MMC Session with PBL Session | Import with alias: Session as PBLSession |
| Matching KR timestamps with VideoInfo timestamps | Use fetch_keypoints(only_detected=True) for frame-aligned data |
Assuming camera_params matches keypoint camera order | Reorder from Calibration order to SingleCameraVideo alphabetical order |
Method Numbers Reference
| Pipeline | Table | Method Field | Default |
|---|---|---|---|
| 2D Pose | TopDownPerson | top_down_method | 0 (MMPose) |
| 3D Lifting | LiftingPerson | lifting_method | 1 (VideoPose3D) |
| 3D Triangulation (MMC) | PersonKeypointReconstruction | reconstruction_method | 0 |
| Multi-Camera Kinematic | KinematicReconstruction | kinematic_reconstruction_settings_num | 137 |
| Monocular Kinematic | MonocularReconstruction | monocular_reconstruction_settings_num | 1 |
DataJoint Schema Names
Each repository stores tables in named MySQL schemas. Use these to connect directly or create virtual modules with dj.VirtualModule('alias', 'schema_name').
| Schema Name | Repository | Key Tables |
|---|---|---|
pose_pipeline | PosePipeline | Video, VideoInfo, TopDownPerson, LiftingPerson |
mocap_sessions | MultiCameraTracking | Subject, Session, Recording |
multicamera_tracking | MultiCameraTracking | MultiCameraRecording, SingleCameraVideo, PersonKeypointReconstruction, Calibration |
multicamera_tracking_annotation | MultiCameraTracking | VideoActivity (walking/standing labels) |
project_body_models | BodyModels | KinematicReconstruction, KinematicReconstruction.Trial, ProbabilisticReconstruction |
project_monocular_testing | BodyModels | MonocularReconstruction, MonocularReconstruction.Trial |
project_body_models_gait_cycles | BodyModels | GaitTransformer, GaitTransformer.WalkingSegment, GaitTransformer.Steps, GaitTransformerMonocular |
project_gdi | BodyModels | GDICycles, GDICyclesMonocular, GDIJointsLookup |
emgimu_sessions | PortableBiomechanicsSessions | Subject, Session, FirebaseSession, FirebaseSession.AppVideo |
openpbl_session_annotations | PortableBiomechanicsSessions | VideoActivity (PBL), WalkingType |
Connecting Without Code Access
import datajoint as dj
# Create virtual modules to access tables without installing the package
kinematic = dj.VirtualModule('kinematic', 'project_body_models')
gait = dj.VirtualModule('gait', 'project_body_models_gait_cycles')
sessions = dj.VirtualModule('sessions', 'mocap_sessions')
pose = dj.VirtualModule('pose', 'pose_pipeline')
mmc = dj.VirtualModule('mmc', 'multicamera_tracking')
# Then query as usual
keys = kinematic.KinematicReconstruction.fetch('KEY')
Gait-Specific Tables
For gait analysis (walking segments, step metrics, GDI), see the /gait-metrics skill which documents:
- Walking segment detection and validation
- Spatiotemporal gait metrics (step length, cadence, velocity)
- Gait Deviation Index (GDI) computation
- Joint ROM analysis per gait cycle/phase
Files to Explore
| Task | File |
|---|---|
| Video/TopDownPerson | PosePipeline/pose_pipeline/pipeline.py |
| MMC Session/Recording | MultiCameraTracking/multi_camera/datajoint/sessions.py |
| MMC MultiCameraRecording | MultiCameraTracking/multi_camera/datajoint/multi_camera_dj.py |
| MMC KinematicReconstruction | BodyModels/body_models/datajoint/kinematic_dj.py |
| PBL Subject/Session/FirebaseSession | PortableBiomechanicsSessions/portable_biomechanics_sessions/emgimu_session.py |
| PBL MonocularReconstruction | BodyModels/body_models/datajoint/monocular_dj.py |
| Gait Cycles/Walking Segments | BodyModels/body_models/datajoint/gait/gait_cycles_dj.py |
| GDI (Gait Deviation Index) | BodyModels/body_models/datajoint/gait/gdi_dj.py |
| Gait Step/Phase Metrics | BodyModels/body_models/datajoint/gait/gait_analysis.py |
| Gait Event Detection | BodyModels/body_models/biomechanics_mjx/gait/gait_transformer_cycles.py |
| Gait Metrics (standalone) | BodyModels/body_models/biomechanics_mjx/gait/gait_metrics.py |
| VideoActivity (MMC walking labels) | MultiCameraTracking/multi_camera/datajoint/annotation.py |
| VideoActivity (PBL walking labels) | PortableBiomechanicsSessions/portable_biomechanics_sessions/session_annotations.py |
RAE JAX Recompilation, Acceleration & Profiling Skill (rae-jax-profiling)
This skill consolidates the canonical Reproducible Agent Environment (RAE) standards from jax-jit-and-fusion, jax-memory-and-retracing, jax-multi-gpu, and jax-config — together with the general JAX acceleration principles established from StanfordASL/frax (Morton & Pavone 2026), XMimic (Bohlinger & Peters 2026), BodyModels, and BiomechanicalFollower (KinTwin / SuperMimic) — into a single operational checklist and verification protocol for /workspaces/eqx_pose.
Any newly implemented or compute-heavy JAX/Equinox algorithm must satisfy the rules below and pass both a single-compile assertion test (max_traces=1 + jax.monitoring backend compile listener) and a cold-vs-warm jax.block_until_ready() benchmark.
1. eqx.filter_jit & PyTree Dynamic vs. Static Separation
- Always decorate entry points with
@eqx.filter_jit, never bare@jax.jit, when arguments includeeqx.Moduleinstances, callable PyTrees, or dataclass configs.eqx.filter_jitautomatically partitions all JAX arrays (eqx.is_array) as dynamic traced leaves and treats non-array metadata as static. - Pass scalar hyperparameters/weights that vary across calls as traced
jnp.float32/jnp.int32arrays (or register config dataclasses via@register_config_dataclasswhere varying weights areArrayfields). Pythonfloat,int, andboolvalues are hashed statically byeqx.filter_jit— varying them across trials or ablation toggles (e.g.,use_sensors=TruevsFalse) triggers a full XLA recompilation unless converted to tracedjnp.float32multipliers (0.0vs1.0). - Never bake large frozen model weights as XLA constants inside closures. If a stage optimizer uses
jax.jitover a flat parameter vectortheta, split the frozen model viamodel_arrays, model_static = eqx.partition(model, eqx.is_array), passmodel_arraysin the traceddatadict, and reconstructmodel = eqx.combine(data["model_arrays"], model_static)inside the traced function. - Keep biomechanical/kinematic arrays in
jnp.float32throughouteqx_posepipelines.
2. Bucket-Padding Variable Sequence Lengths $T$ with Masks
Because XLA compiles a distinct executable for every unique tensor shape, passing raw trial lengths $T \in {187, 243, 312, \dots}$ to a jitted solver or trajectory fitter causes an $O(\text{num_trials})$ compilation storm and host/GPU OOM.
- Snap $T$ to a small geometric bucket set (e.g.,
BUCKETS = (64, 128, 256, 512, 1024, 2048)):def bucket_length(n: int, buckets: tuple[int, ...] = (64, 128, 256, 512, 1024, 2048)) -> int: for b in buckets: if b >= n: return b return ((n + buckets[-1] - 1) // buckets[-1]) * buckets[-1] - Edge-pad time/observation arrays to $T_{\text{bucket}}$ and attach a float32 validity mask
maskof shape(T_bucket,)(1.0for valid frames $0 \le t < T$,0.0for padded tail frames $T \le t < T_{\text{bucket}}$):- Use
mode="edge"when padding timestamps and poses so finite differences / Nadaraya-Watson kernels / Jacobians remain well-conditioned on padded rows. - Replace
jnp.mean(err)with masked reductions:def masked_mean(values: Array, mask: Array) -> Array: w = jnp.reshape(mask, (mask.shape[0],) + (1,) * (values.ndim - 1)) denom = jnp.maximum(jnp.sum(mask) * (values.size // mask.shape[0]), 1.0) return jnp.sum(values * w) / denom - For finite-difference velocity ($T-1$) and acceleration ($T-2$) regularizers, construct edge masks
vel_mask = mask[:-1] * mask[1:]andacc_mask = mask[:-2] * mask[1:-1] * mask[2:].
- Use
- Slice outputs back to
[:T]at the host boundary after optimization or inference completes.
3. Eliminating Function-Local @jit Rebuilds & Closure-ID Cache Misses
Two closely related closure anti-patterns silently defeat JAX/Equinox compilation caching:
3a. Function-Local @eqx.filter_jit / @jax.jit Rebuilt on Every Call (Amortized Over an Inner Batch Loop)
Defining @eqx.filter_jit def fwd(...) or solve = jax.jit(jax.vmap(lambda ...)) inside an entry-point function or method (def decode(...), def place_in_camera(...)) to capture local variables (qp, k_jax) in a closure builds a brand-new function object on every call to the outer function:
# BAD: `@eqx.filter_jit` is defined inside `decode()` to close over `qp` and `k_jax`.
# Every call to `decode()` creates a fresh `fwd` function object and triggers a full ~9 s
# XLA compile on batch 0. Averaged over T=120 frames (15 batches of 8), the 9 s compile is
# amortized across the loop (`9000 ms / 120 + 4.5 ms ≈ 80 ms/frame`), masquerading as a
# plausible per-frame forward time (~18x slower than true steady state) and throwing away
# the compiled executable as soon as `decode()` returns!
def decode(head, images, K, boxes, valid, *, batch_size=8):
qp, k_jax = head.query_points, jnp.asarray(K, jnp.float32)
@eqx.filter_jit
def fwd(model, ims, bxs, ok):
return jax.vmap(lambda im, b, v: forward_one(model, qp, im, b[None], v[None], k_jax))(ims, bxs, ok)
return [fwd(head.device_model, ...) for ...]
# GOOD: Hoisted to module scope with `qp` and `k_jax` passed as explicit traced arguments.
# Compiles once across the entire process lifetime (~4.5 ms/frame warm).
@eqx.filter_jit
def _decode_batch_jit(model, qp, k_jax, ims, bxs, ok):
return jax.vmap(lambda im, b, v: forward_one(model, qp, im, b[None], v[None], k_jax))(ims, bxs, ok)
Why
eqx.debug.assert_max_traceson a function-local definition fails silently: If you wrapfwdwitheqx.debug.assert_max_traces(max_traces=1)insidedecode(), a fresh guard object is created on everydecode()call with its trace counter reset to0. Within onedecode()call, all 15 batches have shape(8, H, W, 3)so the counter reaches1and passes — even when 100 calls todecode()compile 100 separate XLA executables! Always hoist both@eqx.filter_jitandassert_max_tracesto module scope.
3b. Passing Fresh Nested Closures to vmap / jit / Chunked Primitives
Creating a fresh nested Python closure on every call and passing it to a jitted or cached higher-order primitive misses any cache keyed on id(fn) or function identity:
# BAD: Creates a fresh function object `fk` with a new `id(fk)` on every call!
# Any cache keyed on `id(fn)` misses 100% of the time and leaks compiled executables.
def _landmarks_per_frame(state, fk_landmarks, frame_chunk):
shp, scl, go, gt, bp = state.unpack()
def fk(g, t, b):
return fk_landmarks(g, t, b, shp, scl)
return minibatch_vmap(fk, go, gt, bp, batch_size=frame_chunk)
Fixes:
- Always define
@eqx.filter_jit/@jax.jitfunctions at module scope (or bind once in__init__), passing call-local arrays (qp,k_jax,shp,scl) as explicit traced arguments. - Use callable
eqx.ModulePyTrees oreqx.Partial(or unroll closure cells into a PyTree before@eqx.filter_jit) so that captured arrays (shp,scl, model weights, index tables) are exposed as dynamic traced PyTree leaves while the underlying function code object is static and hashable. - Never key a compilation cache on
id(fn)without weakrefs or closure unwrapping. In Python, ephemeral closures either leak forever if stored in the cache or have theirid()reused after garbage collection for a completely different closure.
4. Eliminating Host Syncs (float(loss)) Inside Training Loops via jax.lax.scan
Calling float(loss_val) or np.asarray(grad) inside a Python for step in range(num_steps): loop forces a device-to-host synchronization every step, destroying JAX's asynchronous dispatch (~100–500 $\mu$s dispatch overhead per step) and preventing cross-step XLA fusion.
# BAD: Python loop + per-step device-to-host sync `float(loss_val)`
for _ in range(config.num_steps):
model, opt_state, loss_val, metrics = train_step(model, opt_state, data)
loss_history.append(float(loss_val))
Fix — Fuse steps with jax.lax.scan inside @eqx.filter_jit:
@eqx.filter_jit
def _run_training_scan(
model: Model,
opt_state: optax.OptState,
data: FitData,
optimizer: optax.GradientTransformation,
num_steps: int,
) -> tuple[Model, optax.OptState, Float[Array, "num_steps"], dict[str, Float[Array, ""]]]:
params, static = eqx.partition(model, eqx.is_inexact_array)
def _step(carry, _):
curr_params, curr_opt = carry
curr_model = eqx.combine(curr_params, static)
(loss_val, metrics), grads = eqx.filter_value_and_grad(loss_fn, has_aux=True)(
curr_model, data
)
updates, next_opt = optimizer.update(grads, curr_opt, curr_params)
next_params = eqx.apply_updates(curr_params, updates)
return (next_params, next_opt), (loss_val, metrics)
(final_params, final_opt), (losses, metrics_hist) = jax.lax.scan(
_step, (params, opt_state), None, length=num_steps
)
final_model = eqx.combine(final_params, static)
last_metrics = jax.tree_util.tree_map(lambda x: x[-1], metrics_hist)
return final_model, final_opt, losses, last_metrics
Then transfer losses to host once after the scan finishes: loss_history = np.asarray(losses, dtype=np.float64).tolist().
5. Single-Pass Nested jax.jvp & Precomputed Static Linear Operators
- Simultaneous $(x(t), \dot{x}(t), \ddot{x}(t))$ evaluation for scalar $t$:
Using nested
jax.jacfwdwith a separate primal call evaluates the network 3 times per timestamp. Replace with a single nestedjax.jvpalong unit scalar tangentone = jnp.ones_like(t):
This computes position, velocity, and acceleration in one single forward-over-forward pass.def evaluate_single_pos_vel_acc(fn, t: Float[Array, ""]) -> tuple[Array, Array, Array]: one = jnp.ones_like(t) def val_and_vel(s: Float[Array, ""]) -> tuple[Array, Array]: return jax.jvp(fn, (s,), (one,)) (val, vel), (_, acc) = jax.jvp(val_and_vel, (t,), (one,)) return val, vel, acc - Precompute static Vandermonde / Savitzky-Golay pseudoinverses on CPU with
@functools.lru_cache: Never calljnp.linalg.pinv(SVD) on a static-shape Vandermonde matrix inside a traced function. Compute the filter weights once in NumPy (@lru_cache) and convert to a smalljnp.asarray(..., dtype=signal.dtype)kernel forjnp.convolve.
6. Verifying xla_compile_count == 1 & compiles_during_timed == 0
Every hot jitted function or training loop must be verified against unexpected retracing and function-local @jit rebuilds:
- In unit tests (
eqx.debug.assert_max_tracesat module scope):
(Never placeinner = eqx.debug.assert_max_traces(step_fn, max_traces=1) jitted_step = eqx.filter_jit(inner)assert_max_traceson a function defined inside another function or method — its trace counter resets to0on every outer call!) - Process-wide XLA compile counter (
_CompileCounter) for benchmarks and end-to-end pipelines:import logging import jax class _CompileCounter(logging.Handler): """Counts XLA compilations across the entire process (catches even function-local @jit rebuilds).""" n = 0 def emit(self, record): msg = record.getMessage() if "Finished XLA compilation" in msg or "Compiling " in msg: _CompileCounter.n += 1 def enable_compile_count(): jax.config.update("jax_log_compiles", True) h = _CompileCounter() for name in ("jax", "jax._src.dispatch", "jax._src.interpreters.pxla", "jax._src.compiler"): lg = logging.getLogger(name) lg.addHandler(h) lg.setLevel(logging.WARNING) lg.propagate = False - Interactive diagnostics (
jax.explain_cache_misses&jax.log_compiles):with jax.explain_cache_misses(): out1 = jitted_fn(x1) out2 = jitted_fn(x2) # prints exact leaf/shape/id responsible if a cache miss occurs
7. Cold Compile vs. Warm block_until_ready() Profiling Protocol
Because JAX dispatch is asynchronous, wall-clock timings without jax.block_until_ready() only measure Python enqueue time, and warmup calls without a compile counter can silently hide function-local @jit rebuilds. Always benchmark on the same GPU and inside the same process:
import time
import jax
enable_compile_count()
# 1. Cold call (compilation + first execution)
t0 = time.perf_counter()
out_cold = jax.block_until_ready(jitted_fn(*args_1))
cold_ms = (time.perf_counter() - t0) * 1e3
# 2. Warm calls (steady-state execution; assert zero XLA compilations during timed window!)
c0 = _CompileCounter.n
warm_times_ms = []
for args_i in warm_inputs:
t0 = time.perf_counter()
_ = jax.block_until_ready(jitted_fn(*args_i))
warm_times_ms.append((time.perf_counter() - t0) * 1e3)
compiles_during_timed = _CompileCounter.n - c0
assert compiles_during_timed == 0, f"Expected 0 warm compiles, got {compiles_during_timed}!"
Report:
- Cold compile + execute latency (ms)
- Warm execution latency (ms, median / IQR)
compiles_during_timed(0required for warm timings) & total compilation count across $N$ trials / calls (1vs. $N$)- Numerical parity (
np.testing.assert_allclose/ relative error) against the reference implementation.
8. Indicating Precision in JAX: jax.default_matmul_precision, DotAlgorithmPreset, and PyTree Policies (Replacing Legacy jmp)
JAX controls precision at two distinct levels, and neither requires DeepMind's archived jmp (0.0.4, last released 2022 for Haiku + float16 loss scaling):
8a. Level 1 — Matmul Accumulator & Tensor-Core Precision (jax.default_matmul_precision & DotAlgorithmPreset)
Even when arrays have dtype jnp.float32, NVIDIA GPUs can execute matmul / einsum / dot_general using different hardware instructions. Native JAX (0.10+) provides two built-in ways to specify this without changing array storage dtypes:
- Scoped context manager or per-op
precision=(jax.default_matmul_precision):with jax.default_matmul_precision("highest"):(orprecision=jax.lax.Precision.HIGHESTon a singlejnp.matmul/jnp.einsum): forces full 24-bit IEEEfloat32FMA instead of 10-bitTF32. Use this for kinematic trees (ForwardKinematics), skinning (body_models/mhr/skin.py), and whitening/Cholesky paths.with jax.default_matmul_precision("bfloat16"):(or"tensorfloat32"): allowsfloat32matmuls to round inputs tobf16(orTF32) on tensor cores while inputs/outputs stayfloat32.
- Explicit
jax.lax.DotAlgorithmPreset(jax >= 0.4.30/0.10.2):- You can pass an explicit
<LHS>_<RHS>_<ACCUM>preset directly toprecision=(e.g.,precision="BF16_BF16_F32","BF16_BF16_BF16","TF32_TF32_F32","F32_F32_F32","F64_F64_F64").
- You can pass an explicit
8b. Level 2 — Array Storage & Activation Dtype Policy (param_dtype, compute_dtype, output_dtype)
To also halve activation bandwidth and guarantee that backward cotangent GEMMs ($\nabla_X = \Delta W^\top$ and $\nabla_W = X^\top \Delta$, which represent $2\times$ the FLOPs of forward) run with bf16 inputs and outputs on tensor cores, hidden activations themselves must be stored as jnp.bfloat16:
- Why
preferred_element_type=jnp.float32is a trap onbf16matmuls: NVIDIA tensor cores already accumulatebf16 × bf16products infp32registers internally even whenpreferred_element_type=jnp.bfloat16. Settingpreferred_element_type=jnp.float32forces the layer output tofloat32, which makes the backward pass's incoming cotangentfloat32and drops both backward GEMMs off thebf16tensor cores (~2xslower). - Why per-layer
jmp.Policy(body_models/implicit_representation.py) is suboptimal insidevmap: Callingself.jmp_policy.cast_to_compute((self.weight, self.bias, x))insideLinear.__call__whenjax.vmap(implicit_traj)(timestamps)is mapped over $T$ frames casts the weights inside the per-timestampvmapbody (and required forkingeqx.nn.Linear). - Preferred Equinox Pattern — PyTree
cast_to_computeonce outsidevmap(or a 12-line nativePrecisionPolicydataclass):
Cast the@dataclass(frozen=True) class PrecisionPolicy: param_dtype: jnp.dtype = jnp.float32 compute_dtype: jnp.dtype = jnp.bfloat16 output_dtype: jnp.dtype = jnp.float32 def cast_to_compute(self, tree): return jax.tree.map( lambda x: x.astype(self.compute_dtype) if eqx.is_inexact_array(x) else x, tree ) def cast_to_output(self, tree): return jax.tree.map( lambda x: x.astype(self.output_dtype) if eqx.is_inexact_array(x) else x, tree )eqx.ModulePyTree once at the forward boundary (model_bf16 = policy.cast_to_compute(model_f32)), run standardeqx.nn.Linear/w @ xlayers (wherebf16 @ bf16automatically usespreferred_element_type=jnp.bfloat16withfp32tensor-core accumulation), and cast the final output back viapolicy.cast_to_output(y). Reverse-mode AD automatically casts thebf16weight gradients back toparam_dtype(float32) at the boundary foroptax. - Keep precision-critical operations in
jnp.float32(orfloat64for Kalman/Gaussian covariances):- Master parameters and Optax/Adam moments (
mu,nu). - Running observation normalizers (
RunningMeanStd) andLayerNorm/RMSNormvariance reductions. - Autoregressive state integration (
next_state = state_f32 + delta_bf16.astype(jnp.float32)), policy distribution heads (mean,log_std,log_prob), value heads, GAE advantages, and losses.
- Master parameters and Optax/Adam moments (
- Gate with a single-batch
fp32vs.bf16gradient equivalence test (test_fast_equiv.py):- Assert finite losses/gradients and cosine similarity $> 0.999$ between
fp32andbf16parameter gradient vectors on an identical batch.
- Assert finite losses/gradients and cosine similarity $> 0.999$ between
9. Analytic @jax.custom_jvp Geometric Jacobians Over Kinematic Trees (frax Principle)
Calling jax.jacfwd(residual_fn)(q) through a general multi-branch biomechanical tree (e.g., 64 bodies, 94 hinge DOFs, 80+ markers/sites) forces forward-mode autodiff to propagate a $94$-column tangent array through every body rotation and translation in the tree — even though each joint column $j$ only moves its descendant bodies. This causes 20–50 s XLA compile times and 2–10x slower warm LM/Gauss-Newton steps.
Rules for Fast Differentiable Kinematics & Inverse Kinematics (frax @morton2026frax & BodyModels):
- Wrap
ForwardKinematicsin@jax.custom_jvpusing the closed-form geometric Jacobian and a static binary ancestor mask:- Precompute a static binary ancestor mask
M_ancof shape(n_bodies, nv)(or(n_sites, nv)) at model load time, whereM_anc[b, j] = 1.0iff DOF $j$ is on the kinematic path from the root to body/site $b$. - In the forward primal pass, compute world body/site positions $\mathbf{p}_b$, world joint anchor positions $\mathbf{p}_j$, and world joint axes $\mathbf{a}_j$.
- In the JVP rule, compute the exact geometric Jacobian in closed form without tracing through the tree recurrence: $$\frac{\partial \mathbf{p}b}{\partial q_j} = M{\text{anc}}[b, j] \begin{cases} \mathbf{a}_j \times (\mathbf{p}_b - \mathbf{p}_j) & \text{(revolute / hinge DOF } j\text{)} \ \mathbf{a}_j & \text{(prismatic / translation DOF } j\text{)} \end{cases}, \qquad \frac{\partial \boldsymbol{\omega}b}{\partial q_j} = M{\text{anc}}[b, j],\mathbf{a}_j^{\text{rot}}$$
- Keep
jax.jacfwdon the outer residual wrapper (camera_project, gauge weights, linear/polynomial joint couplings): becauseForwardKinematicsnow has a@jax.custom_jvprule,jacfwd(residual)automatically invokes the analytic geometric Jacobian for the tree and uses AD only for the thin outer residual assembly.
- Precompute a static binary ancestor mask
- Force
precision=jax.lax.Precision.HIGHESTon small $3\times 3$ kinematic matmuls on GPU:- On NVIDIA GPUs, JAX defaults
float32matmuls toTF32(10-bit mantissa), which introduces~5e-4to1.4e-3drift across chained $3\times 3$ body rotations and lever-arm cross products compared tomjx.kinematics/float64. - Always pass
precision=jax.lax.Precision.HIGHESTto small $3\times 3$ rotation/tangent matmuls inside FK and its@jax.custom_jvprule; this restores< 1e-6GPUfloat32parity at negligible runtime cost.
- On NVIDIA GPUs, JAX defaults
- Serial chains vs. shallow branched trees:
- For long serial chains (manipulators, continuum spines), use
jax.lax.associative_scanover $SE(3)$ homogeneous transforms (fraxparallel scan) to reduce sequential depth from $O(N)$ to $O(\log_2 N)$. - For shallow branched human skeletons (depth $\le 14$, width $64$), a depth-layered vector gather (
for d in range(1, max_depth + 1):updating all bodies at depth $d$ in parallel) avoids padding every body to max tree depth.
- For long serial chains (manipulators, continuum spines), use
10. Catching pmap/pjit Recompiles via jax.monitoring & Fixing Sharding-Flip Bugs
Two multi-device compilation traps frequently waste 25–50 minutes per training run without appearing in jax_log_compiles:
jax_log_compiles=Truedoes NOT logpmapcompilations in modern JAX (>= 0.4.30/0.10+):- To catch every
jit,pjit,shard_map, andpmapbackend compilation, register a listener on"/jax/core/compile/backend_compile_duration"in addition to the log handler:COMPILE_DURATIONS_S: list[float] = [] jax.monitoring.register_event_duration_secs_listener( lambda event, duration_s: COMPILE_DURATIONS_S.append(duration_s) if event == "/jax/core/compile/backend_compile_duration" else None )
- To catch every
- The
_device_put_shardedvs.pmap'i'-Mesh Sharding-Flip Recompile Trap:- Legacy
jax.device_put_replicated(x, devs)tags array leaves withNamedSharding(mesh={'_device_put_sharded': N}), whereasjax.pmap(..., axis_name='i')returns leaves tagged withNamedSharding(mesh={'i': N}). - Symptom A (Epoch-2 recompile): Epoch 1 compiles
pmapfor_device_put_shardedinputs, and Epoch 2 recompiles the entirepmapstep (~40–60 s) because the carry returned by Epoch 1 now has mesh'i'. - Symptom B (Every-eval recompile with
donate_argnums): If an evaluator uses@functools.partial(jax.jit, donate_argnums=(0,))oneval_statewhile consumingunreplicate(training_state.params), any leaf whose sharding flips between_device_put_shardedand'i'across calls causes the evaluator to recompile on every single evaluation (~30 s$\times$100evals =50 minwasted). - Fix: Normalize initial replicated state through a 1-step
pmap(lambda s: s, axis_name="i")identity (or construct all arrays directly on a single explicitjax.sharding.Mesh), strip weak types (jax.tree.map(lambda x: jnp.asarray(x, dtype=x.dtype) if hasattr(x, "dtype") else x, state)), and commit unreplicated single-device parameters viajax.device_put(leaf[0], jax.devices()[0])before entering anydonate_argnumsevaluator.
- Legacy
11. Hoisting Loop-Invariant Work Outside lax.scan & Deferred 1-GEMM Weight Gradients in Unrolled BPTT
When optimizing recurrent rollouts (lax.scan over horizon $H=16\text{--}64$, such as SuperMimic world-model imagined rollouts or PPO unrolls):
- Hoist all state-independent gathers, encodings, and PRNG splits outside
lax.scan:- Never slice reference windows via
dynamic_slice_in_dim, split PRNG keys, or run an MLP encoder on reference/goal inputs inside the $H$-step scan body if those inputs do not depend on the evolving recurrent state. - Compute the full
(H, B, ...)reference slice, encoded goal embeddings, and noise tensors once in vectorized(H, B, ...)form beforelax.scan, and pass them as the scanned sequencexs.
- Never slice reference windows via
- Deferred 1-GEMM-per-layer weight gradients for unrolled BPTT through shared MLPs (
XMimicfast_nets.py):- When backpropagating through an $H$-step
lax.scanthat applies the same MLP weights $W$ at every step, standard reverse-mode AD computes $H$ separate per-step weight gradients $\nabla_{W}^{(t)} = x_t^\top \delta_t$ and accumulates them. - For compute-heavy unrolled horizons ($H \ge 16$), a custom VJP that propagates only activation cotangents $\delta_t$ through the sequential recurrence and then computes $\nabla_W = \text{reshape}(X, (H\cdot B, D_{\text{in}}))^\top \text{reshape}(\Delta, (H\cdot B, D_{\text{out}}))$ after the loop replaces $H$ small GEMMs with one high-occupancy tensor-core GEMM per layer.
- When backpropagating through an $H$-step
- Eliminate redundant value-net forward passes in fixed-length rollouts:
- If
episode_lengthis fixed inside the scan andtruncationis identicallyFalseon steps $0 \dots T-2$,next_value[t]equalsvalue[t+1]. Evaluate the critic once per step plus one bootstrap call after the loop (T+1calls total instead of2T), cutting rollout critic compute in half.
- If
12. Multi-Iteration lax.scan Program Granularity, Persistent Cache Hygiene & Sim-vs-Learner Crossover
- Multi-iteration
lax.scanprogram granularity (K = 50–100updates per JIT call) with on-device ring buffers:- Calling separate
collect_step,replay_add,wm_step, andpolicy_stepJIT functions from a Pythonforloop drains the GPU at every boundary. - Fuse trajectory collection, on-device ring-buffer insertion (
jax.lax.dynamic_update_slice_in_dimalong the time/slot axis), uniform window sampling, and learner updates into a single pure carry function scanned over $K \in [20, 100]$ iterations inside one@eqx.filter_jit(donate="all")program. Return stacked scalar metrics of shape(K,)and log them on the host only once per $K$-iteration chunk.
- Calling separate
- Persistent XLA compilation cache hygiene (
JAX_INCLUDE_FULL_TRACEBACKS_IN_LOCATIONS=false):- Configure the persistent XLA cache before importing
jaxin training and benchmark entrypoints:import os os.environ.setdefault("JAX_COMPILATION_CACHE_DIR", "/workspaces/eqx_pose/.jax_cache") os.environ.setdefault("JAX_PERSISTENT_CACHE_MIN_COMPILE_TIME_SECS", "0") os.environ.setdefault("JAX_PERSISTENT_CACHE_MIN_ENTRY_SIZE_BYTES", "-1") # Omitting full Python tracebacks from HLO op metadata roughly halves cache load/save time: os.environ.setdefault("JAX_INCLUDE_FULL_TRACEBACKS_IN_LOCATIONS", "false")
- Configure the persistent XLA cache before importing
- Profile the Sim-vs-Learner scaling crossover before optimizing:
- Batched GPU physics (
MJX/MuJoCo-Warp) scales sub-linearly withnum_envsuntil GPU SMs saturate (e.g.,4xmore envs from512 -> 2048costs only1.7xwall time), whereas large MLP learners (8×1024value net over64minibatches) are GEMM-bound early and scale linearly (4.7xwall time for4xenvs). - At
num_envs >= 2048, the learner often consumes65–75%of total iteration wall time while physics is only25–35%. Always time(A) env-step scan onlyvs.(B) full training stepat the production batch/env count so you know whether the bottleneck is physics (mjx_impl="warp",njmax,nconmax, solver iterations) or the learner (bf16GEMMs, critic size,num_updates_per_batch).
- Batched GPU physics (
Overview
This skill creates a properly structured Python repository from scratch. It enforces
the standards in enforce-guidelines/references/repo-structure.md.
Use when:
- Creating a new Python project
- Converting an unstructured project to proper layout
- User says "new repo", "new project", "initialize", "scaffold"
Parameters
- name (required): Project name (lowercase, hyphens allowed)
- description (required): One-line project description
- package_name (optional): Python package name (defaults to name with underscores)
- author (optional): Author name (defaults to "James Cotton")
- extras (optional): Additional optional-dependencies groups to include
Steps
1. Validate Inputs
- You MUST verify project name is lowercase with hyphens only
- You MUST derive package_name from project name (replace hyphens with underscores)
- You MUST NOT proceed without a description
2. Create Directory Structure
mkdir -p src/{package_name}
mkdir -p tests
touch src/{package_name}/__init__.py
touch src/{package_name}/py.typed
touch tests/__init__.py
- You MUST use src/ layout
- You MUST create both src/ and tests/ directories
- You MUST include py.typed marker for type hint support
3. Create pyproject.toml
Copy the canonical template from templates/pyproject.toml in the RAE plugin and customize:
- Replace
your-project-namewith{name} - Replace
your_packagewith{package_name} - Replace author name/email
- Set the project description
- Add any project-specific dependencies
The template is the single source of truth for ruff, pytest, and coverage config. Do not hardcode these values — always read from the template file.
- You MUST set line-length = 120
- You MUST put pytest and ruff in dev optional-dependencies
- You MUST NOT put dev tools in main dependencies
4. Create .gitignore
# Python
__pycache__/
*.py[cod]
*.egg-info/
dist/
build/
.eggs/
# Virtual environments
.venv/
venv/
ENV/
# IDE
.idea/
.vscode/
*.swp
# Testing
.pytest_cache/
.coverage
htmlcov/
# RAE
scraps/
scratch/
.rae-version
# OS
.DS_Store
Thumbs.db
5. Create README.md
# {name}
{description}
## Installation
```bash
uv pip install -e ".[dev]"
Development
# Run tests
pytest
# Format code
ruff format .
# Lint code
ruff check .
### 6. Create Initial Test Files
Create `tests/conftest.py`:
```python
"""Shared test fixtures."""
Create tests/test_placeholder.py:
"""Placeholder test to verify pytest works."""
def test_placeholder() -> None:
"""Remove this test once real tests exist."""
assert True
- You MUST include type hints (-> None)
- You MUST create conftest.py for shared fixtures
7. Initialize Git (if not already)
git init
git add .
git commit -m "feat: Initialize {name} with RAE structure"
- You MUST NOT commit if already in a git repo with uncommitted changes
- You SHOULD offer to commit but confirm with user first
8. Create CLAUDE.md
Copy or adapt templates/CLAUDE.md to create a CLAUDE.md with project-specific instructions, always including the DataJoint read-only rule (if applicable), the Fail-Fast Field Access rule, and the no-Claude-session-links rule:
## CRITICAL: NEVER Modify Database Entries
**DO NOT update, delete, or alter any DataJoint database entries.** This includes:
- `update1()`, `delete()`, `drop()` on any table
- Modifying settings lookup tables (KinematicReconstructionSettingsLookup, ProbabilisticReconstructionSettingsLookup, KineticReconstructionSettingsLookup, KeypointSet, etc.)
- Altering any computed table entries
Database entries are shared state used by the entire lab. Changing a settings entry changes it for everyone and invalidates prior results computed with those settings.
## Fail-Fast Field Access & Anti-Defensive-Programming
Never use defensive `getattr(obj, "field", None)` or `dict.get("field", None)` (nor broad `try/except Exception: pass`) when accessing expected struct, dataclass, Equinox module, or dictionary fields. Use direct named access (`person.mhr` / `person.mhr.joint_coords` or `data["mhr"]`) so schema mismatches fail fast with `AttributeError` or `KeyError`. (In `WheelchairPropulsion`, a defensive `getattr(person, "mhr", None)` / `getattr(person.mhr, "joints_3d", None)` check on a nonexistent attribute name silently evaluated `mhr_hand21` to `None` on every frame without raising an error, silently disabling the 20-DOF finger fit.) Always check for and reject this anti-pattern during code review and PR review.
- You MUST include this section if datajoint is in the project's dependencies
- You SHOULD include it by default for any biomechanics/motion-capture project
9. Verify Structure
ruff format .
ruff check .
pytest
- You MUST run ruff format before completing
- You MUST run ruff check with no errors
- You MUST run pytest with all tests passing
10. (Optional) Add Devcontainer
If the user wants devcontainer support, ask which template to use:
Option 1: CPU-only (lightweight, faster startup)
- Copy from
templates/devcontainer-cpu/devcontainer.json - Uses
mcr.microsoft.com/devcontainers/python:3.11 - No GPU support, minimal dependencies
- Best for: web apps, APIs, general Python development
Option 2: GPU-enabled (CUDA + cuDNN for ML/CV)
- Copy from
templates/devcontainer-gpu/devcontainer.json - Uses
nvidia/cuda:12.6.0-cudnn-devel-ubuntu24.04 - Includes: GPU access, OpenCV dependencies, git-lfs, Jupyter
- Requires:
--env-file .envin project root - Best for: machine learning, computer vision, biomechanics
Both templates:
- Install Claude Code, pyright, RAE plugin, and full plugin suite via
postCreateCommand - Mount
~/.claudefor config persistence - Configure VSCode with ruff, Python testing, 120-char rulers
- Include Node.js via
"ghcr.io/devcontainers/features/node:1": {}(required for excalidraw rendering via npx) - No Dockerfile needed - everything via features and install script
Beads (bead-driven development): Ask the user whether they want beads enabled for this project. Beads is off by default. Only enable if the user explicitly opts in. If enabled, add the beads plugin to the devcontainer's postCreateCommand install list.
Note: Ensure ~/.claude exists on the host before starting: mkdir -p ~/.claude
GPU template also requires: Create a .env file in project root for secrets (gitignored)
Adding Optional Dependencies
If user requests specific libraries, add appropriate optional-dependencies:
OpenCV:
[project.optional-dependencies]
opencv = ["opencv-python>=4.0.0"]
opencv-headless = ["opencv-python-headless>=4.0.0"]
opencv-contrib = ["opencv-contrib-python>=4.0.0"]
PyTorch:
[project.optional-dependencies]
torch-cpu = ["torch>=2.0"]
torch-cuda = ["torch>=2.0"] # User installs with CUDA separately
Jupyter:
[project.optional-dependencies]
notebooks = ["jupyter>=1.0", "ipykernel>=6.0"]
Headless 3D rendering (meshes / point clouds / video overlays):
[project.optional-dependencies]
viz = ["pyrender>=0.1.45", "trimesh>=4.0", "PyOpenGL>=3.1", "opencv-python-headless>=4.0.0"]
This needs the EGL/Vulkan system libraries and ICD manifests from the GPU devcontainer
(libgl1, libegl1, libvulkan1, libosmesa6-dev, plus /usr/share/glvnd/egl_vendor.d/10_nvidia.json
and /etc/vulkan/icd.d/nvidia_icd.json and -e NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics)
— already in templates/devcontainer-gpu. Note that if a user restricts GPUs in runArgs with manual
--device=/dev/nvidia* flags, they must also pass --device=/dev/nvidia-modeset and --device=/dev/dri
for Vulkan/RTX rendering (e.g. Isaac Sim, SAPIEN). At runtime the project must set PYOPENGL_PLATFORM=egl
(and MUJOCO_GL=egl) before importing any GL library and verify it didn't fall back to the llvmpipe
CPU rasterizer. Point the user at the /efficient-rendering skill for the device-probing,
context-reuse, camera-convention, and video-compositing patterns rather than
re-deriving them here.
Examples
User: "/scaffold-repo my-analysis-tool A tool for analyzing motion capture data"
Agent:
- Creates directory structure
- Generates pyproject.toml with name="my-analysis-tool", package="my_analysis_tool"
- Creates .gitignore, README.md
- Creates placeholder test
- Runs ruff format, ruff check, pytest
- Reports success with next steps
User: "/scaffold-repo cv-processor Image processing pipeline --extras opencv-headless"
Agent:
- Creates standard structure
- Adds opencv-headless to optional-dependencies
- Completes verification
Semi-Formal Code Reasoning
This skill implements structured reasoning templates that force explicit evidence gathering before conclusions. Based on research showing 10+ percentage point accuracy gains over unstructured chain-of-thought on patch equivalence, fault localization, and code question answering tasks.
Reference: Ugare, Shubham, and Satish Chandra. 2026. "Agentic Code Reasoning." arXiv:2603.01896. Preprint, arXiv, March 4. https://doi.org/10.48550/arXiv.2603.01896.
Core Principle
Never conclude without tracing. The central failure mode in code reasoning is assuming behavior from names, signatures, or surface similarity. Semi-formal reasoning prevents this by requiring a certificate: explicit premises, concrete execution traces with file:line evidence, and a formal conclusion that follows from the traces. If you can't fill in the template, you haven't gathered enough evidence yet.
When to Use Which Template
| Task | Template | Trigger |
|---|---|---|
| Comparing two patches/diffs | Patch Equivalence | "are these equivalent", "same behavior", PR review with alternatives |
| Finding buggy code from test failures | Fault Localization | "find the bug", "why does this fail", "localize the fault" |
| Understanding code behavior | Code QA | "what does this do", "how does X work", "trace through this" |
| General code review | Code Review (adapted from Patch Equivalence) | "review this PR", "is this change safe" |
Pick the closest template. For ambiguous tasks, Code QA is the most general.
Template 1: Patch Equivalence
Use when comparing two code changes to determine if they produce the same behavior.
Structured Exploration Phase
Before filling the certificate, gather evidence systematically:
- Read both patches carefully. Identify every file and function touched.
- For each function called in the patches:
- Search for its definition in the repository (do NOT assume standard library semantics — names can be shadowed by module-level or project-specific definitions).
- Document:
FUNCTION: name | LOCATION: file:line | ACTUAL BEHAVIOR: ...
- Read the test patch (if available). Identify what behaviors are tested.
- Check for side effects: imports, module-level definitions, class inheritance, decorators, middleware, signal handlers — anything that could alter execution semantics.
Certificate Template
DEFINITIONS:
D1: Two patches are EQUIVALENT MODULO TESTS iff executing the
repository test suite produces identical pass/fail outcomes.
D2: The relevant tests are ONLY those in FAIL_TO_PASS and
PASS_TO_PASS (the existing test suite).
PREMISES (state what each patch does — cite file:line):
P1: Patch 1 modifies [file(s)] by [specific change, with line refs]
P2: Patch 2 modifies [file(s)] by [specific change, with line refs]
P3: The FAIL_TO_PASS tests check [specific behavior]
P4: The PASS_TO_PASS tests check [specific behavior, if relevant]
FUNCTION RESOLUTION TABLE:
| Function/Symbol | Assumed Meaning | Actual Definition (file:line) | Different? |
|-----------------|-----------------|-------------------------------|------------|
(Fill for EVERY function/method called in either patch)
ANALYSIS OF TEST BEHAVIOR:
For each FAIL_TO_PASS test:
Claim 1.1: With Patch 1, test [name] will [PASS/FAIL]
because [concrete execution trace through code]
Claim 1.2: With Patch 2, test [name] will [PASS/FAIL]
because [concrete execution trace through code]
Comparison: [SAME/DIFFERENT] outcome
For each relevant PASS_TO_PASS test:
Claim 2.1: With Patch 1, behavior is [description with trace]
Claim 2.2: With Patch 2, behavior is [description with trace]
Comparison: [SAME/DIFFERENT] outcome
EDGE CASES (only those exercised by actual tests):
E1: [Edge case] → Patch 1: [behavior] | Patch 2: [behavior] | Same? [Y/N]
COUNTEREXAMPLE (if NOT EQUIVALENT):
Test [name] will [PASS/FAIL] with Patch 1 because [traced reason]
Test [name] will [FAIL/PASS] with Patch 2 because [traced reason]
OR
NO COUNTEREXAMPLE EXISTS (if EQUIVALENT):
All tests produce identical outcomes because [traced reason]
FORMAL CONCLUSION:
By D1, since test outcomes are [IDENTICAL/DIFFERENT],
patches are [EQUIVALENT / NOT EQUIVALENT].
ANSWER: [YES/NO]
CONFIDENCE: [HIGH/MEDIUM/LOW] — [brief justification]
Key Anti-Patterns to Avoid
- Name assumption: Never assume
format(),open(),print(), etc. are builtins. Always check for module-level shadowing. - Skipping tests: Trace EVERY test, not just the ones that look relevant.
- Surface similarity: High textual similarity does NOT imply semantic equivalence. Patches can look identical but diverge on one edge case.
- Premature conclusion: If you find yourself writing the ANSWER before completing the ANALYSIS section, stop and go back.
Template 2: Fault Localization
Use when finding buggy code given a failing test, without execution information.
Structured Exploration Phase
Explore in a hypothesis-driven loop:
HYPOTHESIS H[N]: [What you expect to find and why it may contain the bug]
EVIDENCE: [What from the test or previously read files supports this]
CONFIDENCE: [high/medium/low]
-> Read the file
OBSERVATIONS from [filename]:
O[N]: [Key observation, with line numbers]
HYPOTHESIS UPDATE:
H[M]: [CONFIRMED | REFUTED | REFINED] — [Explanation]
UNRESOLVED:
- [What questions remain]
NEXT ACTION RATIONALE: [Why reading another file, or why enough evidence]
Certificate Template
## Phase 1: Test Semantics Analysis
PREMISE T1: The test calls [X.method(args)] and expects [behavior]
PREMISE T2: The test asserts [condition]
PREMISE T3: [Additional test constraints]
Key question: What is the EXPECTED vs OBSERVED behavior?
## Phase 2: Code Path Tracing
Trace from test entry point into production code:
METHOD: [ClassName.methodName(params)]
LOCATION: [file:line]
BEHAVIOR: [what this method does — VERIFIED by reading source]
RELEVANT: [why it matters to the test]
-> [next call in chain]
...
## Phase 3: Divergence Analysis
For each traced path, identify where implementation diverges from expectations:
CLAIM D1: At [file:line], [code] would produce [behavior]
which contradicts PREMISE T[N] because [reason]
CLAIM D2: ...
Each claim MUST reference a specific PREMISE and specific code location.
IMPORTANT — Distinguish ROOT CAUSE from CRASH SITE:
- The crash site is where the error manifests (e.g., StackOverflowError)
- The root cause is the code that creates the bad state
- Always trace BACKWARD from crash site to root cause
## Phase 4: Ranked Predictions
Rank 1 ([confidence]): [file:lines] — [root cause description]
Supporting claims: D[N], D[M]
Rank 2 ([confidence]): [file:lines] — [description]
Supporting claims: D[N]
...up to Rank 5
Key Anti-Patterns to Avoid
- Stopping at the crash site: The method that throws the exception is usually NOT the root cause. Trace backward to find what created the bad state.
- Single-turn predictions: Don't predict from test code alone. Read source files.
- Name-based guessing: Don't assume a method is buggy because its name is similar to the failing test. Trace the actual execution path.
- Ignoring indirection: Bugs are often in configuration classes, factory methods, or helper utilities — not in the method directly called by the test.
Template 3: Code Question Answering
Use when answering questions about code behavior, semantics, or design.
Certificate Template
FUNCTION TRACE TABLE:
| Function/Method | File:Line | Parameter Types | Return Type | Behavior (VERIFIED) |
|-----------------|-----------|-----------------|-------------|---------------------|
(Every function examined — MUST read source, not guess from name)
DATA FLOW ANALYSIS:
Variable: [key variable name]
- Created at: [file:line]
- Modified at: [file:line(s), or 'NEVER MODIFIED']
- Used at: [file:line(s)]
(Repeat for each key variable relevant to the question)
SEMANTIC PROPERTIES:
Property 1: [e.g., "HashMap is mutable", "this list is sorted"]
- Evidence: [specific file:line]
Property 2: ...
ALTERNATIVE HYPOTHESIS CHECK:
If the opposite answer were true, what evidence would exist?
- Searched for: [what you looked for]
- Found: [what you found — cite file:line]
- Conclusion: [REFUTED / SUPPORTED]
ANSWER: [Final answer with explicit evidence chain]
CONFIDENCE: [HIGH/MEDIUM/LOW]
Key Anti-Patterns to Avoid
- Answering from function names:
getUserName()might not return a user's name. Read the implementation. - Missing downstream handling: Even if you find an edge case, check whether downstream code already handles it before claiming it's a problem.
- Incomplete trace chains: If your answer depends on 5 functions, trace all 5. A chain is only as strong as its weakest link.
Template 4: Code Review (Adapted)
Use for general PR/diff review when there's no second patch to compare against.
Certificate Template
CHANGE SUMMARY:
C1: [file] line [N-M]: [what changed and why, based on PR description]
C2: ...
FUNCTION RESOLUTION TABLE:
| Function/Symbol | Called In Change? | Definition (file:line) | Semantics Verified? |
|-----------------|-------------------|------------------------|---------------------|
BEHAVIORAL ANALYSIS:
For each change C[N]:
- Pre-change behavior: [traced from code]
- Post-change behavior: [traced from code]
- Regression risk: [NONE / LOW / MEDIUM / HIGH] — [specific reason]
INVARIANT CHECK:
For each invariant or assumption the code relies on:
I1: [invariant] — PRESERVED / BROKEN by change C[N] because [reason]
I_FAILFAST (MANDATORY): Does the diff introduce any defensive `getattr(obj, "field", None)`,
`d.get("field", None)`, or broad `try/except Exception: pass` on known/expected schemas?
(Cite exact lines; reject in favor of direct `obj.field` / `d["field"]` access so schema
mismatches raise `AttributeError` / `KeyError` rather than silently disabling downstream
stages as in the `WheelchairPropulsion` `getattr(person, "mhr", None)` 20-DOF finger-fit bug.)
EDGE CASES:
E1: [input/state] -> Pre: [behavior] | Post: [behavior] | Safe? [Y/N]
CONCLUSION:
[APPROVE / REQUEST CHANGES / NEEDS DISCUSSION]
Summary: [1-2 sentence summary of findings]
Issues found: [list, or "none"]
General Guidance
Calibrating Depth
Not every analysis needs the full template. Use judgment:
- Quick check (high-confidence, small diff, single file): Fill the template mentally, write a condensed version. ~5 minutes equivalent.
- Standard analysis (moderate complexity, 2-5 files): Full template, moderate exploration. ~15-30 agent steps.
- Deep analysis (subtle semantics, cross-module dependencies, framework-specific behavior): Full template with extensive exploration. ~30-50 agent steps.
If the user asks for a quick review, provide a condensed version but flag if you spot something that warrants deeper analysis.
When Semi-Formal Reasoning Helps Most
The accuracy gains are largest when:
- Code spans multiple files with non-obvious dependencies
- Functions or symbols might be shadowed or overridden
- The difference between correct and incorrect hinges on a subtle semantic detail
- Framework-specific behavior differs from standard library behavior
- Multiple test cases need to be reasoned about independently
When It Helps Less
- Single-file, self-contained code snippets
- Pure syntax or formatting questions
- Cases where the model already has strong baseline accuracy
- When speed matters more than thoroughness (flag the tradeoff to the user)
Running Independent Probes
When you cannot execute repository code but need to verify language semantics (e.g., "what
does Python's format() builtin do with this input?"), you may write and run small
independent scripts that test general language behavior. This is legitimate static analysis —
you are verifying your understanding of the language, not executing the repository's code.
Presenting Results
After completing the certificate:
- Lead with the answer — don't make the user read the whole trace first.
- Summarize key findings in 2-3 sentences.
- Offer the full certificate as supporting evidence if the user wants to audit your reasoning.
- Flag confidence level and explain what would change your mind.
~80% ASD-STE100 Technical Specifications & Explanations (ste100-spec)
Use this skill whenever writing, auditing, or updating technical specifications (docs/SPEC.md, docs/specs/*.md, or docs/papers/<slug>/SPEC.md) or explaining complex codebases, algorithms, and pull requests in ~80% ASD-STE100 (Simplified Technical English).
Background (The Karpathy ~80% ASD-STE100 Pattern): ASD-STE100 is the international aerospace maintenance specification standard designed to eliminate ambiguity in safety-critical engineering documentation. Strict 100% ASD-STE100 restricts vocabulary to a ~900-word dictionary, which forces awkward circumlocutions for modern machine learning, computer vision, and biomechanics concepts. At ~80% strictness (as highlighted by Andrej Karpathy), you retain exact domain-specific technical nouns and symbols (
MHR,SequenceIK,Cholesky,ELBO,float32,multi_camera) while strictly enforcing ASD-STE100 grammar, sentence-length bounds, active voice, and one-idea-per-sentence structure. The result is dense, zero-fluff engineering prose that humans and LLMs can parse and verify at a glance.
1. Core ~80% ASD-STE100 Writing Rules
Every specification or STE-100 explanation MUST obey these six grammatical and structural constraints:
| Rule | Constraint | Why It Matters |
|---|---|---|
| 1. Active Voice Only | Every sentence must name the explicit actor (<Actor> <verb> <object>). Passive voice ("is computed", "are filtered", "was optimized") is forbidden. | Passive voice hides which module, function, or stage performs the operation. |
| 2. Strict Sentence Length Limits | Procedural / pipeline steps: $\le 20$ words per sentence. Descriptive / conceptual context: $\le 25$ words per sentence. | Short sentences prevent nested subordinate clauses and force explicit intermediate states. |
| 3. One Idea Per Sentence | Each sentence states one action, invariant, or causal relationship. Split compound sentences ("X does A while Y does B, which enables Z") into numbered sequential steps. | Prevents hidden assumptions and makes every sentence independently falsifiable. |
| 4. One Term Per Concept (Zero Synonym Drift) | Pick one exact technical term for each concept and use it identically throughout the document. Never alternate synonyms for stylistic variety. | Calling the same tensor pose, joint_angles, qpos, and kinematic_state across paragraphs causes implementation bugs. |
| 5. Maximum 3-Word Noun Clusters | Break up stacked 4+ word noun chains ("multi-camera keypoint trajectory optimization weight" $\to$ "weight for multi-camera trajectory optimization"). | Long noun clusters obscure which word modifies which noun. |
| 6. Explicit Shapes, Frames & Physical Units | State array shapes ((T, J, 3)), data types (float32), coordinate frames (world, camera, pelvis-root), and physical units (mm, deg, rad, N/kg, Hz) inline. Strip all marketing adjectives ("seamless", "robust", "cutting-edge", "novel", "crucially"). | Eliminates unit/frame mismatch bugs and AI filler prose. |
1.1 Before vs. After Examples (AI Slop vs. ~80% ASD-STE100)
| Standard / AI-Generated Prose (Forbidden in Specs) | ~80% ASD-STE100 Specification (Required) |
|---|---|
| "Optimization of the biomechanical trajectory is seamlessly performed across multiple views by leveraging our novel implicit representation to minimize reprojection errors alongside temporal smoothness priors." (26 words, passive, vague actor, marketing fluff) | 1. ImplicitTrajectoryNet maps normalized timestamps $t \in [-1, 1]$ to joint angles $\mathbf{q}_t \in \mathbb{R}^{49}$ (rad). (16 words)2. ForwardKinematics transforms $\mathbf{q}_t$ into 3D markers $\mathbf{X}_t \in \mathbb{R}^{M \times 3}$ (mm, world frame). (14 words)3. SequenceIK minimizes the sum of robust 2D reprojection loss and joint velocity penalties using L-BFGS. (16 words) |
| "When handling missing detections, a confidence threshold is applied and low-quality keypoints are masked out before triangulation occurs." (18 words, passive twice, unspecified threshold/shape) | 1. filter_keypoints zeros 2D detections with confidence $c_{t,k} < 0.30$ in keypoints_2d (T, C, K, 3). (16 words)2. triangulate_ransac requires at least two cameras with valid detections to estimate 3D point $\mathbf{x}_{t,k}$ (mm). (16 words) |
| "The handrim contact phase is robustly identified using kinematic heuristics and subsequently refined via an Extended Kalman Filter." (18 words, passive, hides exact signals) | 1. ContactClassifier predicts contact probability $p_t \in [0, 1]$ from 3D wrist-to-handrim distance (mm) and tangential speed (m/s). (18 words)2. PhaseEKF updates continuous propulsion phase $\phi_t \in [0, 2\pi)$ (rad) at 60 Hz. (14 words) |
2. Architectural Decision Framework: Single Unified Spec vs. Multi-Specification Suite
Before creating specification files in a repository or paper directory, decide whether the system belongs in one unified specification or a modular suite of specifications.
2.1 The 3-Question Split Test
Default to one unified specification (docs/SPEC.md or docs/papers/<slug>/SPEC.md). Split into multiple modular specifications (docs/specs/) if and only if at least one of the following three conditions holds:
- Independent Subsystem Contracts: Does the repository contain distinct subsystems with independent input/output contracts that can be executed and tested in isolation? (Example: (1) offline CT anatomical atlas construction, (2) multi-GPU flow-matching training, and (3) real-time single-camera clinical inference.)
- Independent Downstream Consumers: Do external repositories or users import or invoke one subsystem without running the others? (Example:
BodyModelsforward kinematics vs.MultiCameraTracking2D-to-3D triangulation.) - Complexity & Length Ceiling: Would a single end-to-end specification exceed 150 lines or 40 numbered procedural steps?
2.2 Decision Matrix
| System Archetype | Layout Choice | File Paths | Line & Step Budget |
|---|---|---|---|
| Single End-to-End Pipeline or Focused Paper (e.g., WheelchairPropulsion, ProstheticComponentry, single clinical validation or method paper) | Single Unified Spec | Repo: docs/SPEC.mdPaper: docs/papers/<slug>/SPEC.md | $\le 120$ lines total ($\le 30$ procedural steps) |
| Multi-Subsystem Platform or Coupled Method Suite (e.g., BodyModels + MhrFkOpt, multi-stage training + calibration + deployment repo) | Modular Spec Suite + Mandatory Top-Level Overview | docs/specs/00_overview.mddocs/specs/01_<subsystem_a>.mddocs/specs/02_<subsystem_b>.mddocs/specs/03_<subsystem_c>.md | 00_overview.md: $\le 35$ linesEach 0N_<module>.md: $\le 100$ lines |
MANDATORY RULE FOR MULTI-SPEC SUITES (
docs/specs/00_overview.md): Never create a directory of numbered spec files without00_overview.md.00_overview.mdmust stay under 35 lines and contain:
- A 2-sentence system purpose statement.
- A dataflow table or ASCII diagram showing the exact tensor/file handoff between
01_...,02_..., and03_....- Direct links to each module specification.
3. Canonical ~80% ASD-STE100 Templates
Starter templates are also available in references/templates.md.
3.1 Template A: Project / Repository Specification (docs/SPEC.md)
Use this structure to specify a repository's production pipeline in $\le 120$ lines:
# Technical Specification: <Repository or System Name>
> **Format:** ~80% ASD-STE100 (Active voice; $\le 20$ words per procedural step; $\le 25$ words per descriptive sentence; explicit shapes, frames, and units).
> **Code Anchor Commit:** `<git-sha>`
## 1. Purpose and Scope
- **System Actor:** `<package_name>` (`src/<package_name>/`)
- **Primary Function:** <1–2 active-voice sentences ($\le 25$ words each) stating what the system transforms from input to output.>
- **Out of Scope:** <1 sentence ($\le 25$ words) stating what upstream/downstream systems handle.>
## 2. Terminology and Coordinate Conventions
| Term / Symbol | Exact Definition | Shape & Type | Frame & Units |
| :--- | :--- | :--- | :--- |
| `<var_name>` | <Single-meaning definition> | `(T, J, 3)` `float32` | `world`, `mm` |
## 3. Input Contracts
1. `<Entrypoint>` accepts `<input_a>` (`<shape>`, `<dtype>`, `<units>`) from `<source>`.
2. `<Entrypoint>` validates that `<precondition>` holds; otherwise `<Entrypoint>` raises `ValueError`.
## 4. Sequential Pipeline Stages
*(Every step MUST use active voice and contain $\le 20$ words.)*
### Stage 1: <Stage Name> (`src/<pkg>/<module>.py:<func>`)
1. `<Actor>` loads `<input>` and normalizes `<field>` to `<range>` (`<units>`).
2. `<Actor>` computes `<intermediate>` (`<shape>`, `<dtype>`) using `<algorithm>`.
### Stage 2: <Stage Name> (`src/<pkg>/<module>.py:<func>`)
3. `<Actor>` projects `<intermediate>` into `<target_space>` (`<shape>`, `<units>`).
4. `<Actor>` minimizes `<objective>` over `<N>` iterations with step size `<lr>`.
## 5. Output Contracts and Invariants
- **Primary Output:** `<output_name>` (`<shape>`, `<dtype>`, `<units>`).
- **Hard Invariants:**
1. `<Tensor>` contains zero `NaN` or `Inf` values across all valid frames.
2. `<Physical constraint>` holds within tolerance `±<tol> <units>`.
## 6. Verification and Benchmark Gates
| Gate | Command | Pass Criterion |
| :--- | :--- | :--- |
| Unit & Parity Suite | `pytest tests/` | 100% pass, coverage $\ge 80\%$ |
| Primary Benchmark | `python scripts/<eval>.py` | `<Metric>` $\le <threshold> <units>` on `<cohort>` ($N = <count>$ participants) |
3.2 Template B: Paper Technical Specification (docs/papers/<paper_slug>/SPEC.md)
Write docs/papers/<paper_slug>/SPEC.md before drafting prose in methods.md and results.md, and keep it synchronized with the code and manuscript. While the manuscript (methods.md, results.md) uses flowing pre-2026 narrative paragraphs (per /isr-paper-writing), SPEC.md provides the crisp, unambiguous ~80% STE-100 engineering blueprint that prevents symbol drift, degree-of-freedom bugs, and code-to-paper mismatches.
# Paper Specification: <Paper Title> (`<paper_slug>`)
> **Format:** ~80% ASD-STE100 Engineering Blueprint for [`manuscript.md`](manuscript.md).
> **Code Provenance Commit:** `<git-sha>`
## 1. Core Problem and Thesis
1. **Measurement Gap:** <Active-voice sentence ($\le 25$ words) stating what clinical/biomechanical quantity existing systems fail to measure at scale.>
2. **Core Methodological Thesis:** <Active-voice sentence ($\le 25$ words) stating how our unified formulation solves this gap.>
3. **Statistical Unit of Analysis:** <State the exact independent unit, e.g., $N = 12$ participants ($df = 11$), never pooled trials or frames.>
## 2. Notation and Symbol Contract
*(Every symbol in `methods.md` and `results.md` MUST match this table without synonym drift.)*
| LaTeX Symbol | Code Variable (`file:line`) | Definition | Shape & Units |
| :--- | :--- | :--- | :--- |
| $\boldsymbol{\theta}_t$ | `qpos` (`src/model.py:L42`) | Generalized joint coordinates at frame $t$ | `(D,)` `rad` / `m` |
## 3. Unified Method Stages ($\le 20$ Words Per Step)
1. **Stage 1 (<Name>):** `<Component>` maps `<input>` to `<latent/state>` (`<!-- CODE: src/...:L10-L45 -->`).
2. **Stage 2 (<Name>):** `<Component>` predicts `<output>` by minimizing `<objective>` (`<!-- CODE: src/...:L50-L95 -->`).
3. **Inference Label-Blindness Invariant:** At test time, `<Model>` receives zero ground-truth clinical labels, forceplate signals, or per-trial bias corrections.
## 4. Primary Quantitative Claims and Evidence Anchors
| Claim ID | Manuscript Section | STE-100 Claim Statement ($\le 25$ words) | Metric & Value ($N$, $df$) | Reproduction Script & Output JSON |
| :--- | :--- | :--- | :--- | :--- |
| **C1** | `results.md` §3.1 | `<OurMethod>` reduces 3D joint error compared to `<Baseline>` on `<Cohort>`. | `<X.X ± Y.Y mm>` vs. `<A.A ± B.B mm>` ($N=12, p=0.002$) | `scripts/eval.py` $\to$ `outputs/eval.json` |
## 5. Ablations That Mattered
| Variant | What Changes Relative to Unified Method | Quantitative Effect |
| :--- | :--- | :--- |
| **Unified Method (Ours)** | Complete formulation (Stages 1–3). | `<Primary Metric>` |
| **Ablation 1 (`w/o <Component>`)** | `<Module>` removes `<term>`. | Error increases by `+<delta> <units>`. |
## 6. Explicit Scope Boundaries and Limitations
1. **Observability Limit:** `<System>` cannot observe `<unobservable quantity>` from `<modality>` alone.
2. **Cohort / Domain Limit:** Training data covers `<population>`; `<out-of-domain case>` requires `<adaptation>`.
3. **Compute Cost:** `<Hardware>` processes a `<duration>` trial in `<time>` (`<fps> Hz`).
3.3 Template C: Ad-Hoc ~80% STE-100 Explanation (Code, Bug, or PR Walkthrough)
When the user asks you to "explain how X works in STE-100", "summarize this PR in STE-100", or "give me a Karpathy STE-100 walkthrough", format your response directly using this structure:
### ~80% ASD-STE100 Walkthrough: <Topic / PR / Subsystem>
**Purpose:** <1 active-voice sentence ($\le 25$ words) stating what the code or change accomplishes.>
**Inputs:**
- `<arg_1>` (`<shape>`, `<dtype>`, `<units>`): <1-line description>
**Sequential Execution Steps ($\le 20$ words per sentence):**
1. [`<func_a>`](file:///abs/path/to/file.py#L10-L25) reads `<arg_1>` and validates `<condition>`.
2. [`<func_b>`](file:///abs/path/to/file.py#L30-L60) transforms `<state_1>` into `<state_2>` (`<shape>`, `<units>`).
3. [`<func_c>`](file:///abs/path/to/file.py#L65-L90) writes `<output>` to `<destination>`.
**Outputs & Invariants:**
- Returns `<output>` (`<shape>`, `<dtype>`, `<units>`) with invariant `<condition>`.
4. Evergreen Specification Maintenance Rule
Specifications in docs/SPEC.md, docs/specs/*.md, and docs/papers/<slug>/SPEC.md are evergreen engineering contracts, not historical changelogs:
- Same-Commit Synchronization Mandate: Whenever a code change or manuscript revision modifies input/output shapes, coordinate conventions, pipeline stages, loss terms, degrees of freedom, or headline benchmark numbers, you must update the corresponding STE-100 specification in the same commit / PR.
- Never Document Historical Evolution in
SPEC.md: Do not write "In v1 we used X, and in v2 we switched to Y" inside a specification. Document only what the current production code does today. Leave change history togit logand the Pull Request description. - Self-Audit Checklist Before Committing a Spec:
- Does every procedural step use active voice (
<Actor> <verb> <object>) and contain $\le 20$ words? - Does every descriptive sentence contain $\le 25$ words and express one idea?
- Are all marketing adjectives (
"robust","seamless","novel","cutting-edge") removed? - Does every tensor/quantity specify its shape, dtype, coordinate frame, and physical units (
mm,deg,rad,N/kg,Hz)? - Did you apply the 3-Question Split Test (single
docs/SPEC.md$\le 120$ lines vs. modulardocs/specs/00_overview.md+01_<module>.md)? - Does every stage and quantitative claim link to an exact
file:lineor reproduction script?
- Does every procedural step use active voice (
Tufte Data Visualization
Apply Edward Tufte's principles whenever generating or reviewing code that renders data visually. This skill covers chart generation, not slide/presentation design.
Workflow
Follow these steps in order when creating any chart:
Step 1: Identify the message
Before writing code, determine:
- The key finding or trend the chart must make visible.
- The comparison context — a baseline, prior period, target, or peer group. A number without context is meaningless.
- The chart type that best fits the data structure (see Chart type guidance below).
Step 2: Apply universal rules
Review the rules below. Every rule is a default — deviate only when the user explicitly requests otherwise.
Step 3: Apply library-specific config
Use the Library quick reference table to find the essential overrides for the target library. For complete code examples and helper functions, read ONE rule file from rules/ matching the library.
Step 4: Validate
Run through the validation checklist at the bottom of this file before presenting the chart.
Universal rules
Rules 1–14 cover static principles; 15–19 extend them for screens; 20–22 address content and formatting.
1. Remove top and right borders
No chart should have top or right axis lines, borders, or spines. The bottom and left axes are sufficient. Top and right lines are pure chartjunk.
2. Direct labels, not legends
Label each data series directly — at the endpoint of a line, on or beside a bar, next to a cluster. Remove the <Legend> component entirely. If there is only one series, the chart title provides that context; no label is needed.
3. No gridlines by default
The default is zero gridlines. For static charts where users need to read precise values, add horizontal-only gridlines at very low opacity (0.08–0.12). For interactive charts, prefer a contextual crosshair on hover instead (see rule 15). Never add vertical gridlines.
4. Range-frame axes
Axis lines should span only the range of the data, not from zero to some arbitrary maximum. The axis starts at (or near) the minimum data value and ends at the maximum.
5. No 3D effects
No perspective, no depth, no shadows on chart elements. Two-dimensional data gets two-dimensional representation.
6. No pie charts unless explicitly requested
Default to a horizontal bar chart sorted by value. If the user explicitly asks for a pie chart: maximum 4 slices, 2D only, start at 12 o'clock, direct percentage labels on each slice.
7. Aspect ratio ~1.5:1
Charts should be approximately 50% wider than tall. Standard sizes: 600x400, 750x500, 900x600. Exception: sparklines and small multiples may be more compact.
8. Gray first, highlight selectively
The default data series color is medium gray (#666). Use a single accent color to highlight the most important series or data point. Never use more than 4 distinct colors. Choose the right palette type: categorical (4-color muted) for unordered groups, sequential (single-hue ramp) for ordered magnitude, diverging (two-hue from center) for deviation from a midpoint. See rules/typography-and-color.md for hex values.
9. Transparent background for exported figures (PDF / SVG / PNG)
Default to a transparent background (transparent=True / facecolor="none") for all exported static figures (PDF, SVG, PNG) so they embed seamlessly in LaTeX manuscripts, Markdown docs, and slides without a colored bounding box. Only use #fffff8 (light) or #151515 (dark) as an HTML page background when building a standalone web page whose entire canvas shares that color.
10. Serif fonts for data
Use serif fonts for data labels, annotations, and chart titles: "ET Book", "Palatino Linotype", Palatino, "Book Antiqua", Georgia, serif. Sans-serif (system-ui, sans-serif) is acceptable only for small axis tick labels (11-12px).
11. No dual y-axes
Two y-axes on one chart create false implied correlations. Use small multiples instead — two charts stacked vertically with shared x-axis.
12. Annotate the notable
If the data contains a peak, trough, inflection point, or event boundary, add a text annotation pointing to it directly on the chart. Place annotations in the nearest clear space — offset from the data point with a short leader line if needed. When multiple annotations compete for space, keep only the most important; move others to a footnote or tooltip.
13. Show comparison context
Include at least one reference element: a reference line (average, target, prior period), a shaded band, or a second series. A chart showing one line with no context fails the "Compared to what?" test.
14. Minimal tooltips
Tooltips should be plain text with the data value and label. No colored background, no border, no arrow pointer, no shadow.
15. Progressive disclosure over static density
Default to the Tufte-clean overview — high data-ink, minimal chrome. Layer details through hover, tap, and click (values, annotations, comparisons). Don't frontload everything onto a single static view. A contextual crosshair on hover replaces permanent gridlines.
16. Accessible by default
3:1 contrast ratio minimum for chart elements against their background; 4.5:1 for text in charts. Never use color as the sole differentiator — pair with shape, pattern, or direct label. Provide a text alternative for every chart (aria-label with key finding, or companion data table). Interactive charts must be keyboard-navigable.
17. Responsive, not just resized
Charts must have a responsive strategy — fluid (percentage width + viewBox), adaptive (breakpoint-based layout changes), or hybrid. At narrow viewports, change chart type or layout (horizontal bars for categories, reduced tick density, abbreviated labels), don't just shrink.
18. Animate to explain, not to decorate
Transitions for data changes (sorting, filtering, time progression) are good — they help the viewer track transformations. Gratuitous entrance animations, bouncing, and decorative motion are chartjunk. Duration: 200–500ms, ease-out. Always respect prefers-reduced-motion.
19. Dark mode as first-class citizen
Design both light and dark palettes intentionally. Never invert colors. Reduce saturation in dark mode (bright colors "vibrate" on dark backgrounds). Respect prefers-color-scheme. Use semantic color tokens (--tufte-bg, --tufte-text, --tufte-series-default) so charts adapt automatically.
20. Titles assert findings
The chart title states the key insight, not the axis description. "Revenue Surged 23% in Q3" not "Revenue by Quarter, 2024". The subtitle can provide context ("vs. prior year, USD millions"). If the data has no clear finding, the chart may not be needed (see rule 22).
21. Format numbers for humans
Abbreviate large numbers: $1.2M not $1,200,000. Use thousand separators for mid-range numbers (12,450 not 12450). Match decimal precision to significance (don't show $4.2391M when $4.2M suffices). Right-align numbers in tables. Use consistent units and state them once (in the axis label or title), not on every data point.
22. Don't chart what a sentence can say
If the data is 1–2 numbers, write a sentence with inline context ("Revenue was $4.2M, up 23% from Q2"). If the data is a simple ranking of 3–5 items, consider a table. Charts earn their space by revealing patterns, trends, or distributions that text and tables cannot. A chart of two bars is almost always worse than a sentence.
Library quick reference
The universal rules above are sufficient for most charts. For complete code examples and library-specific helpers, read the appropriate rule file from the rules/ directory in this skill's folder. Only read ONE rule file per task.
| Library | Rule file to read | Essential config (apply even without reading the file) |
|---|---|---|
| Recharts | rules/recharts.md | <CartesianGrid stroke="none" />, remove <Legend />, <YAxis axisLine={false} tickLine={false} />, <Line dot={false} strokeWidth={1.5} /> |
| ECharts | rules/echarts.md | splitLine: { show: false }, legend: { show: false }, grid: { show: false }, use endLabel on series |
| Chart.js | rules/chartjs.md | grid: { display: false }, border: { display: false }, plugins.legend.display: false, use chartjs-plugin-datalabels |
| matplotlib | rules/matplotlib.md | spines['top'].set_visible(False), spines['right'].set_visible(False), spines['bottom'].set_bounds(min, max), font.family: serif, pdf.fonttype: 42, ps.fonttype: 42, svg.fonttype: "none", mathtext.fontset: "stix" |
| Plotly | rules/plotly.md | showgrid=False, showlegend=False, plot_bgcolor='#fffff8', zeroline=False |
| D3/SVG/HTML | rules/svg-html.md | .domain { display: none }, no <rect> backgrounds, stroke-opacity: 0.1 for any gridlines |
Chart type guidance
| Type | Key settings |
|---|---|
| Line | 1.5–2px stroke, dot={false} unless <7 points (then r=2), direct label at rightmost point |
| Bar | Prefer horizontal for categories, sort by value descending, direct value labels, #7a7a7a default fill |
| Scatter | Gray dots #999 r=3, highlight key cluster/outlier with accent, regression line if meaningful (dashed, thin) |
| Time series | Label events on chart ("Recession", "Launch"), range-frame x-axis, YoY via opacity (current solid, prior 30%) |
| Small multiples | Same scale ALL panels, shared axis labels (x on bottom row, y on left column), no panel borders |
| Sparklines | ~80x20px, no axes/labels/gridlines, min/max dots r=1.5, embed inline in text or table cells |
| Data tables | No zebra striping, whitespace + thin rules every 3–5 rows, right-align numbers, font-feature-settings: 'onum' 1 |
| Slopegraph | Before/after categories, label both endpoints (value + name), gray default + highlight key slopes |
| Area | Prefer lines. If area: fillOpacity 0.03–0.08, no gradient, direct labels at endpoints |
| Stacked bar | Avoid — use small multiples instead. If forced: sort by total, direct labels per segment, max 4 segments |
| Heatmap | Sequential or diverging palette only, value labels in cells, companion data table for accessibility |
For small multiples, sparklines, and slopegraph implementation patterns, see rules/small-multiples-sparklines.md.
Color quick reference
| Token | Light | Dark |
|---|---|---|
| Background (exported PDF/SVG/PNG) | transparent (none) | transparent (none) |
| Background (standalone full-page web canvas only) | #fffff8 | #151515 |
| Text | #111 | #ddd |
| Text secondary | #666 | #999 |
| Axis/rule | #ccc | #444 |
| Grid (if used) | #eee (8-12% opacity) | #333 |
| Default series | #666 | #999 |
| Highlight | #e41a1c | #fc8d62 |
Categorical (max 4): #4e79a7 steel blue · #f28e2b tangerine · #e15759 coral · #76b7b2 sage
Font stacks in rule 10. For full palettes (sequential, diverging), font loading, and old-style figures, see rules/typography-and-color.md.
Anti-pattern detection
When reviewing existing chart code, check for: legends (→ direct labels), pie charts (→ horizontal bars), 3D effects (→ flat 2D), dual y-axes (→ small multiples), heavy gridlines (→ remove or 0.1 opacity), rainbow palettes (→ gray + accent), gauge widgets (→ number + sparkline), gradient fills (→ solid color), rotated labels (→ flip axes or abbreviate), pure white/black backgrounds (→ #fffff8/#151515), hover-only information (→ tap/focus fallback), missing text alternatives (→ aria-label), color-only encoding (→ add shape/pattern).
For the full table with per-library detection patterns and one-liner fixes, see rules/anti-patterns.md.
Validation checklist
Before presenting any chart, verify:
- No top or right borders/spines
- No Legend component — series labeled directly on the chart
- Gridlines removed or horizontal-only at opacity <= 0.12
- Aspect ratio approximately 1.5:1
- Background is transparent (
transparent=True/none) for exported PDF/SVG/PNG figures (only use#fffff8if the entire surrounding HTML page is#fffff8) - Serif font for data labels and titles
- Default series color is gray (
#666); color used only for emphasis - No 3D effects, no pie chart (unless explicitly requested)
- Axis lines span only the data range (range-frame)
- Notable data features annotated directly on chart
- Comparison context present (reference line, band, or second series)
- Tooltips are plain text with no decorative styling
- Interactive elements have tap/click/focus alternatives (no hover-only)
- Contrast ratios meet 3:1 (elements) / 4.5:1 (text) minimums
- Chart has a text alternative (aria-label, description, or data table)
- Animations respect
prefers-reduced-motion - Charts render usably at 320px and 1440px+ widths
- Title states the finding, not the axis description
- Numbers are formatted for readability (abbreviations, separators, consistent precision)
- A chart is warranted — the data couldn't be communicated as a sentence or table
Additional resources
Library rules (read ONE per task): rules/recharts.md, rules/echarts.md, rules/chartjs.md, rules/matplotlib.md, rules/plotly.md, rules/svg-html.md — complete code examples, helpers, and theme registrations.
Cross-cutting (read when specifically needed):
rules/interactive-and-accessible.md— progressive disclosure, WCAG, responsive, animation, dark moderules/typography-and-color.md— font loading, full palette tables, old-style figuresrules/anti-patterns.md— per-library detection heuristics and fixesrules/small-multiples-sparklines.md— layout patterns for small multiples, sparklines, slopegraphs
Working examples in examples/ — one per library, plus an inline SVG sparkline.
Biomechanical Manuscript Figure Taxonomy & Exemplars (ISR Lab Standard)
When generating figures for scientific manuscripts (templates/myst-paper, ISR single-column & IEEE two-column builds), adhere to these mandatory rules and reference exemplars:
- Export
.svgas the Primary Format for MyST Markdown Alongside.pdf&.png(transparent=True):- Always save programmatic Python/Matplotlib/vector figures as
.svg(the primary format referenced in MyST.mdfiles viaso MyST Studio and HTML builds render crisp vector graphics natively) alongside.pdfand.png. - MyST (
myst build --tex/myst build --pdf) automatically converts.svg -> .pdfviainkscape(or our zero-GUIrsvg-convert -f pdfCLI shim managed byscripts/ensure_inkscape_and_check_fonts.py). - Never use cream/off-yellow (
#fbfaf5) backgrounds for manuscript figures (off-yellow only works in books matching that color). Usetransparent=Trueon allfig.savefig(...)calls for.svg,.pdf, and.pngmanuscript figures so they blend seamlessly into both ISR Preprint and IEEE two-column pages.
- Always save programmatic Python/Matplotlib/vector figures as
- Prevent IEEE-Rejected Type 3 Fonts (
pdf.fonttype = 42,ps.fonttype = 42,svg.fonttype = "none",mathtext.fontset = "stix"):- IEEE Xplore and PDF eXpress strictly reject PDFs containing Type 3 bitmap fonts or unembedded fonts (
emb = noinpdffonts). Matplotlib's defaultpdf.fonttype = 3and Computer Modern (cm) mathtext emit Type 3 fonts. - Always configure
TUFTE_RCwith"pdf.fonttype": 42,"ps.fonttype": 42,"svg.fonttype": "none","mathtext.fontset": "stix","text.usetex": False, and"font.serif": ["Times New Roman", "Liberation Serif", "DejaVu Serif", "Palatino"]. - When
svg.fonttype = "none"andmathtext.fontset = "stix"are used,scripts/ensure_inkscape_and_check_fonts.pyregisters Matplotlib's bundledSTIXGeneral*.ttfandSTIXNonUnicode*.ttfinto~/.local/share/fonts/matplotlibsoinkscape/rsvg-convertembeds 100% TrueType (Type 42/CID TrueType) outlines with zero Type 3 fonts.
- IEEE Xplore and PDF eXpress strictly reject PDFs containing Type 3 bitmap fonts or unembedded fonts (
- Keep Existing TikZ Diagrams Untouched as
.tikz.tex/.pdf(Do Not Convert TikZ to.svg):- Empirical verification shows that converting standalone TikZ PDFs to SVG via
pdftocairo -svgconverts all TeX Type 1C fonts into raw<symbol><path/></symbol>outlines without<text>nodes orToUnicodetables, stripping font metadata (0 fontsinpdffonts) and destroying PDF text searchability/selectability (pdftotextreturns empty). - Therefore, keep existing standalone/inline TikZ diagrams untouched as
.tikz.tex/.pdffor LaTeX builds, and use.svgas the primary format for all programmatic Python/Matplotlib/vector figures.
- Empirical verification shows that converting standalone TikZ PDFs to SVG via
- Every Figure Panel Must State Its Claim in the 2-Line Header:
- Line 1 (Bold Serif):
A. Direct Empirical Claim or Mechanism(e.g.,B. Bayesian Information Criterion (BIC) Parsimony). - Line 2 (Italic Muted
#555555):Metric, units, sample size (N subjects / n trials), and statistical test(e.g.,Lower ΔBIC = better parsimony (penalizes k · ln n)).
- Line 1 (Bold Serif):
- Three Canonical Figure Archetypes (Flexible — Use When Aligned with the Paper's Main Points):
- Archetype 1 — TikZ / Vector Method & Pathway Overview (
fig1_cambr_architecture_pathways.png,bga_fig1_architecture_overview.png):- Prefer standalone TikZ (
.tikz.tex$\to$.pdf, keeping existing TikZ diagrams untouched as.tikz.tex/.pdf) or clean programmatic vector schematics saved as.svg+.pdf+.pngwith inline mathematical symbols ($\mathcal{L}_{\text{NLL}}$, $\boldsymbol{\theta}$, $\boldsymbol{\beta}$) and explicit visual separation between the Primary Best Pathway and Ablation Pathways. Verify zero text clipping or arrow overlap at 300 DPI before embedding.
- Prefer standalone TikZ (
- Archetype 2 — Camera-Distortion-Corrected Qualitative Biomechanical Overlays + Time-Locked Waveforms (
tremor_multiview_finger_overlay_keyframes.png,hop_test_triple_hop_skeleton_image_overlay_keyframes.png,fig_two_person_cse_mamma_contact.png,fig_fascia_flow_architecture_and_samples.png):- Project 3D meshes and skeletons using full radial/tangential camera distortion via
multi_cameraJAX projection (and/or undistorted video images). - Render
MHRskin as a semi-transparent pearl envelope (alpha = 0.28–0.32) over rigid.stlbones colored by our Bilateral Bone Hue Standard: Left (_l) = Steel-Blue#7FA8D4, Right (_r) = Warm Coral-Red#D98880, Axial/Midline = Warm Ivory#E8E2D5, with CT-calibrated skull alignment (skull_offset_from_mhr). - Pair camera overlays with high-magnification insets (e.g., $2.6\times$ hand/finger or ribcage zoom) and synchronized bottom/side waveform panels (ensuring y-limits have $\ge 10%$ headroom so peaks never crop).
- Project 3D meshes and skeletons using full radial/tangential camera distortion via
- Archetype 3 — High-Density Tufte Quantitative & Occam's Razor Complexity Figures (
fig_ssrp_occams_razor_ablation.png,fig_real_stroke_vs_nonstroke_ssrp.png,fig1_clinical_gaitlab_emg_grf.png):- Show individual subject trajectories or paired spaghetti lines (
Nsubjects) alongside group medians/IQRs to prove zero pseudoreplication. - For model ablations, plot the Occam's Razor Complexity Ladder (in-sample vs. held-out CV performance + BIC/parsimony curve across model degrees of freedom $k$) with a direct callout box highlighting the simplest model that captures the effect.
- Show individual subject trajectories or paired spaghetti lines (
- Archetype 1 — TikZ / Vector Method & Pathway Overview (
Commands 1
Use the rae:bead-driven-development skill to orchestrate planning and execution with persistent beads tracking.
This command is for multi-task development workflows where you want:
- Persistent task tracking across sessions (via beads)
- Automatic skill orchestration (writing-plans, executing-plans, investigation)
- Two-stage code review integration
- Unified workspace organization in
scratch/{date}-plan-{topic}/
Prerequisites: beads CLI installed (bd), beads plugin active, superpowers plugin active.
Hooks 1
Sponsors
Details
- Marketplace
- peabody124/reproducible_agent_environment
- Category
- Community
