
mjswan
Skills for mjswan: build, port and ship MuJoCo scenes as browser apps.
- GitHub stars
- 366
Install
/plugin install mjswan@ttktjmtRun it inside Claude Code after adding the marketplace with /plugin marketplace add ttktjmt/mjswan.
Contents
Discovered from sourceSkills 1
Port one mjlab task from a target repo into a browser app built by mjswan. The target is any repo that registers mjlab tasks, given as a local path or a GitHub URL.
Scope. Inside the target repo everything you write stays on the Python side: no TypeScript (ts_src), no ViewerConfig. mjswan itself is edited in exactly one case: a missing capability that is generic rather than task-specific, which step 8 turns into a pull request.
Ground rules
- mjswan reimplements none of mjlab's term functions: a task's real observation / termination / event / command functions are traced to ONNX at build time and run in the browser beside the policy. A fix therefore means making mjlab's own function traceable, never rewriting its math.
- When something fails, read the exception in full, then the module it came from in the installed package:
python -c "import mjswan, pathlib; print(pathlib.Path(mjswan.__file__).parent)". That source is the only current truth. - Two fix attempts per term, then record it as skipped and move on. A partial port that builds and passes parity is the deliverable; all-or-nothing is not.
1. Acquire the target
A GitHub URL → git clone <url> ./<repo-name>, then work inside it. A local path → use it as is.
Then make mjswan importable from the same interpreter that can import the target's task registrations. What this pipeline needs is the mjlab extra (mjlab + torch: add_scene_mjlab, term tracing, .pt conversion) plus onnxruntime, which step 7 runs the exported graphs under:
- Repo already has an environment → add to it:
uv pip install "mjswan[mjlab]" onnxruntime. - Fresh clone with nothing →
uv venv && uv pip install -e . && uv pip install "mjswan[mjlab]" onnxruntime. - No
uvavailable → install into the interpreter that already runs the target, against its ownsys.prefix.
Every source is its own extra and import mjswan touches none of them, so install only the one the checkpoints come from: wandb for a W&B run path, hf for anything fetched from the Hub, e.g. "mjswan[mjlab,wandb]".
mjswan pins mujoco exactly, its mjlab extra pins mjlab exactly, and the mjlab extra also brings mjlab's own bounded requires-python with it, so the interpreter mjswan alone accepts is not always the one this pipeline accepts. A target that pins either package differently will not co-resolve. If the install fails on one, stop and report the resolver's output verbatim: resolving it is the user's call, not yours.
2. Find the task ids
Registration happens as an import side effect, and every repo does it differently: mjlab.tasks, a <pkg>/tasks/__init__.py, a src/tasks/ on sys.path, or a bootstrap_*() function that must be called, sometimes only after an env var or a data file is in place.
Look at the repo's own README run instructions first, then grep for modules touching mjlab.tasks.registry or calling register. Confirm each candidate by diffing the registry around the import:
python - <<'PY'
from mjlab.tasks.registry import list_tasks
before = set(list_tasks())
import your_repo.tasks # or: from ... import bootstrap_fn; bootstrap_fn()
after = set(list_tasks())
print(f"{len(before)} already registered, {len(after - before)} new")
print(sorted(after - before))
PY
An empty diff means the candidate registers nothing of its own: wrong candidate, or its registration needs a call and arguments you have not supplied yet.
Show the discovered task ids to the user and let them pick one. Ask here, not later, for anything the registration itself needs: env vars, clip or dataset paths, gated-dataset access, credentials.
3. Pre-flight, seconds before any build
load_env_cfg only reads config, so use it to find in seconds what a full build would take minutes to reach:
python - <<'PY'
from mjlab.tasks.registry import load_env_cfg
from mjswan.mjlab import adapt_actions
import your_repo.tasks
cfg = load_env_cfg("Mjlab-Velocity-Flat-Unitree-G1", play=True)
for name, term in (adapt_actions(cfg.actions) or {}).items():
print("action", name, type(term).__name__, term.unsupported_reason)
PY
A task's own action class, one mjswan has no class of its own for, is traced (ADR 0009): adapt_actions returns a TracedActionCfg for it, and the build traces its process_actions and reset against the trace env and checks both graphs against the term's own code. Its apply_actions runs natively in the browser. A class that inherits it from mjlab's position, relative, velocity or effort action needs nothing more; one that overrides it is refused until the port names what it does, reading the options off the task's cfg rather than restating them:
mjswan.register_action(
"JointPositionDeltaActionCfg",
mjswan.ActionBinding(
apply="joint_position", interpolate=lambda cfg: cfg.interpolate_decimation
),
)
ActionBinding covers a position, relative, velocity or torque target, a ramp across the substeps (interpolate) and a hold of the measured pose after a reset (hold_s). An apply outside that, a trace the build refuses (its message names the cause, such as a Python counter or a sim write), or a term with a non-None unsupported_reason makes the task unportable as configured. A refused body can sometimes port through ActionBinding(trace_override=...), which swaps a trace-friendly body onto the built term; otherwise report the reason verbatim, then go to step 8, which decides whether mjswan should learn it. Never hardcode which action classes those are; the build's messages and the unsupported_reason field are the source of truth.
A warning that a term sets fields mjswan's class does not have, or that an actuator sets what the browser does not run (anything past PD within the effort limit and the command delay: a torque-speed curve, friction), names settings the browser drops while the term itself ports. Each is a behaviour gap: report it verbatim beside the skipped terms. A generic one goes to step 8 like a skipped term; one the model can carry instead, such as an actuator's friction as the joint's frictionloss, moves there. A lag an event sets through Actuator.set_lags (older mjlab's sync_actuator_delays) does not trace: drop the event, and give the actuator cfg the lag range itself when the delay matters.
One shape still ports by hand: a term that takes no column of the policy's output (action_dim == 0) and writes joint targets of its own, such as an arm-motion generator beside a locomotion policy. Drop it, and the terms that read it, from the env config. Then pass a CommandJointPositionActionCfg for its joints, and the command it reads, to add_policy(actions=..., commands=...), never to the env config, whose terms mjlab builds itself. The command can be a ui_command of sliders, or a traced command that reproduces the dropped term's motion.
4. Get the policy into ONNX
- W&B run path → nothing to do here.
add_policy_wandb(run_path)converts everymodel_*.ptitself and passes the metadata below on its own. Needs thewandbandmjlabextras. - Hugging Face repo →
add_policy_hf(repo_id). The fetch needs thehfextra and neither mjlab nor torch: a Hub repo holds the published artifact, so this path downloads the.onnxand reads what mjlab baked into it. The scene and its traced terms still need mjlab. - Local
model_*.pt→ copyexport_policy.py(next to this file) into<repo>/mjswan_app/and run it. It writes one<stem>.onnxper checkpoint pluspolicy_meta.json. - Pre-exported
.onnxon disk → usable directly, but plainadd_policyreads nothing out of the file; see below.
policy_meta.json carries policy_joint_names, default_joint_pos and encoder_bias, which add_policy needs to resolve action scales and the browser's external PD gains. A port that omits them builds fine and moves wrongly. An mjlab export does carry the first two in its metadata_props, rounded to three decimals, but only add_policy_hf reads that block, and encoder_bias is never in the file. So always pass all three on the local path, and say so explicitly if a pre-exported .onnx left you without them.
5. Generate the port
In the target repo:
mjswan_app/
main.py builder wiring; the only entry point
terms.py upstream MDP terms, re-implemented traceably (only what step 6 needs)
export_policy.py the copied converter (local .pt only)
model_*.onnx converted checkpoints (local .pt only)
policy_meta.json checkpoint order + metadata (local .pt only)
README.md source URL, prerequisites, run command
The directory is mjswan_app/, not mjswan/: a mjswan/ in the repo root shadows the installed package for every python invoked from there.
terms.py is the one place an MDP term gets rewritten. When a term will not trace, its re-implementation in a form torch.onnx.export accepts, and its register_* call, both go here, whichever kind it is: observation, termination, event, command, or an action term's trace_override body (step 3). Such a re-implementation restates the upstream function's math and changes none of it, as the ground rules demand; what it drops is only what the exporter cannot follow, such as data-dependent control flow, .item(), Python RNG and per-env indexing. Write only the terms that actually failed: every function here is a second copy of upstream code that can silently drift from it, which is why a term mjlab already exports cleanly stays untouched and why step 7 checks each rewrite numerically.
main.py, the shortest thing that works. your_repo.tasks stands for the registration module found in step 2, and TASK_ID is a real constant: substitute the value, never the name.
"""Browser app for one mjlab task, built with mjswan."""
from __future__ import annotations
import json
import pathlib
import your_repo.tasks # noqa: F401 - populates the mjlab task registry
import onnx
import mjswan
HERE = pathlib.Path(__file__).parent
TASK_ID = "Mjlab-Velocity-Flat-Unitree-G1" # the task chosen in step 2
def setup_builder() -> mjswan.Builder:
builder = mjswan.Builder()
project = builder.add_project(name="G1 Velocity")
scene = project.add_scene_mjlab(TASK_ID)
meta = json.loads((HERE / "policy_meta.json").read_text())
mdp = mjswan.MdpConfig() # one MDP, traced once, shared by every checkpoint
for name in meta.pop("checkpoints"):
scene.add_policy(
pathlib.Path(name).stem, onnx.load(HERE / name), mdp=mdp, **meta
)
return builder
def main() -> None:
app = setup_builder().build()
app.launch()
if __name__ == "__main__":
main()
Canonical run command, from the repo root: python -m mjswan_app.main.
build() writes mjswan_app/dist/: the engine, plus the simulation document it serves: manifest.json, and one <project-id>/<scene-id>/ per scene holding scene.mjz, mdp/<mdp-id>/{obs,term,command,event,action}/*.onnx and policy/<policy-id>.onnx. app.save_document() packs that document alone as dist.swn, engine excluded, and mjswan serve / info / publish each take either form. Keep the app the build returns rather than chaining off it, so the document is one call away.
- W&B or the Hub instead of local checkpoints: replace the loop with
scene.add_policy_wandb("entity/project/run_id")orscene.add_policy_hf("owner/repo")and drop the meta plumbing. When that is the whole app,builder.add_project_mjlab(TASK_ID, run_path=..., hf_repo_id=...)is the same project-plus-scene-plus-policies wiring in one call. add_policyaccepts the five MDP term sets (observations=/actions=/terminations=/commands=/events=) or onemdp=mjswan.MdpConfig(...)carrying all five, shared by every policy handed the same object. Leave them all out:add_scene_mjlabsuppliescontrol_dtand the trace env, each term set already defaults to the scene's env config (events included), andadd_policy_wandbbuilds oneMdpConfigper call so a run's checkpoints share one traced MDP. When looping over local checkpoints as above, build oneMdpConfig()before the loop and passmdp=to eachadd_policyfor the same effect. To change a term set, mutateenv_cfginstead (step 6) and passenv_cfg=toadd_scene_mjlabbefore any policy is added, because a scene built first never sees the mutation.- A single-input network, which every mjlab export is, needs no
in_keys: its one observation group lands under the default slot,actor. A network with several inputs declaresin_keys=onadd_policy, and the build refuses one that does not.out_keys=is the same table for the outputs, and a network with several of them needs it whenever the action is not the first: the runtime drives the actuators fromout_keys'action, defaulting to output 0. The build warns; a checkpoint whose exporter sidecar listsout_keysis telling you what to pass. - When
terms.pyexists, import it inmain.pyfor its side effects:from mjswan_app import terms # noqa: F401. - No
ViewerConfig. The defaultOriginType.AUTOtracks the first non-world body.
6. Build, then fix what fails
python -m mjswan_app.main
The build fails loudly by design and names the ways out. For each failure: read the exception, read the module it came from, then classify.
-
Missing env-derived params: mjlab's function needs a constant the config does not carry (terrain limits, a clip path, a threshold). Load
env_cfgyourself, write the value intoenv_cfg.<terms>[name].params, and passenv_cfg=toadd_scene_mjlab. This is the most common fix and needs noterms.pyat all. -
Untraceable body: the function draws with an untraceable RNG, indexes per-env tensors, or reads state the browser has no equivalent for, so
torch.onnx.exportrefuses it. Re-implement it traceably interms.pyand register it:# register_termination / register_event take the same two arguments. mjswan.register_observation("mjlab_func_name", my_traceable_func)Resolution goes by the mjlab function's
__name__first, then the term's dict key, so a closure can only be reached by its key. The annotations onregister_event/register_terminationsay*Binding, but the adapter accepts a plain traceable callable, which is what you want here; a*Bindingmeans TypeScript and is out of scope. -
Unbound command cfg class:
mjswan.register_command("CfgClassName", CommandBinding(...)), keyed by the class name, not a function name. Readmanagers/command_manager.pyin the installed package for which shape applies (tracedstate_fields/command_field, versus nativets_name+serializer). mjswan binds only the classes mjlab's task families share (UniformVelocityCommandCfg,MotionCommandCfg); a class only one task uses is the port's to register, asexamples/demo/main.pydoes forLiftingCommandCfg. -
Missing asset or credential: stop and hand the user the exact command (
wandb login,hf auth login, a licence to accept, an env var to set).
Rebuild after each fix. A term skipped because mjswan cannot express it, rather than because the task is unusual, goes to step 8.
When it finally builds, read the document it wrote before spending minutes on parity:
mjswan info mjswan_app/dist
One line per project, scene, MDP and policy. Three things to check, none of which a successful build says out loud: one MDP per shared MdpConfig (one per add_policy_wandb or add_policy_hf call, one for the whole local-checkpoint loop; more than that means a policy was handed its own config and the same terms were traced once per checkpoint); a non-zero graph count on each MDP (zero means every term ended up native or skipped); and every checkpoint present as a policy. The same command reads a dist.swn, so it is also how you inspect a document someone hands you.
7. Parity gate
A successful build only proves every term traced. It does not prove the graph computes what mjlab computes. Run the numeric gate, the same harness mjswan's own CI runs over its reference tasks:
MUJOCO_GL=disable python - <<'PY'
from mjlab.tasks.registry import load_env_cfg
from mjswan.mjlab import resolve_runner_defaults
from mjswan.compile import run_parity
from mjswan.mjlab.env import build_mjlab_env
import your_repo.tasks
import mjswan_app.terms # only if terms.py exists
TASK_ID = "Mjlab-Velocity-Flat-Unitree-G1"
cfg = load_env_cfg(TASK_ID, play=True)
cfg.scene.num_envs = 1
obs_groups = resolve_runner_defaults(TASK_ID).obs_groups or {}
groups = obs_groups.get("actor")
report = run_parity(
build_mjlab_env(cfg),
obs_group=groups[0] if groups else "actor",
n_steps=16,
)
print(report.summary())
raise SystemExit(0 if report.passed else 1)
PY
A failing term is a real defect: the exported graph disagrees numerically with mjlab's own function, or the build would refuse the term. Treat it exactly like a build failure. A term reported OK as native is not a failure: it has no graph by design, or a registered TS class stands in for it. The report lists the task's own action terms too, traced as the build traces them.
8. When the gap is mjswan's, open a PR
Steps 3, 6 and 7 each end in a block the target repo cannot route around. Some of those blocks are mjswan's gap rather than the task's, and then the fix is a pull request against mjswan.
Only generic capabilities go in
A capability earns a place in mjswan only if another task would hit the same gap. Decide it mechanically by finding where the thing that failed is defined:
python -c "
import your_repo.tasks
from mjlab.tasks.registry import load_env_cfg
cfg = load_env_cfg('Mjlab-Velocity-Flat-Unitree-G1', play=True)
print(type(cfg.actions['TERM_NAME']).__module__)
"
- Defined in
mjlab,myosuite, or another published package → generic. Open the PR. - Defined in the target repo → task-specific. It does not go into mjswan. Use the author-side escape hatch:
terms.pyfor an observation, termination, event or command, anActionBinding(step 3) for an action term; if that cannot work either, report it and stop.
Take the configuration from mjlab
A feature that makes the author restate what mjlab already declares is the wrong design. mjlab's config is the source; mjswan reads it.
- A new mjlab action class: name the mjswan cfg class exactly as mjlab names its own, or add the mapping to
_ALIASESinenvs/mdp/actions/__init__.py._adapt_action_cfgthen copies every matching dataclass field by itself and the port needs no extra argument. A task's own class needs neither: it is traced. - Anything the task's env config or runner config already carries (
env_cfg,resolve_runner_defaults) is read from there, never restated at the call site. - A keyword argument the author has to pass by hand is the last resort, not the first.
Where the change goes
Find the neighbouring case in the mjswan tree first: each already carries a worked example of the thing being added. An action apply ActionBinding cannot name is the one exception to the no-TypeScript rule at the top of this file, because apply runs natively every physics substep (ADR 0009) and needs an engine-side branch in applyAction.ts as well as a binding field or cfg class; that prohibition covers per-port ts_src, not mjswan's own engine.
Change discipline
- Follow mjswan's
AGENTS.md. - One generic unit test at
tests/test_<feature>.py;tests/test_muscle_action.pyis the precedent. Test the feature directly. Do not add a demo scenario, a golden file, or the target's task toexamples/. make format && make type && uv run pytest -m "not slow"before pushing.
Open it
gh repo fork ttktjmt/mjswan --clone /tmp/mjswan-pr
cd /tmp/mjswan-pr && git checkout -b <branch>
# implement, test
gh pr create --repo ttktjmt/mjswan --title "<title>" --body "<body>"
The body names the task and term that motivated the change, states which package defines that term (the generality evidence), and says what the change does not cover.
The port stays blocked until the PR lands. Finish every other part of it, and report this one as waiting on the PR, with its URL.
9. Report
Write mjswan_app/README.md: the target's source URL, the task id, prerequisites (credentials, env vars, data files), the run command, and how the build is handed on: app.save_document() writes mjswan_app/dist.swn, which mjswan serve, mjswan info and mjswan publish each accept in place of the directory.
Then tell the user, in this order:
- the
mjswan infotree, then terms traced and parity result (report.summary()verbatim if anything failed); - terms skipped and settings dropped, each with what unblocks it: an author-side
terms.pyreplacement, anActionBinding, ats_srcTypeScript class (out of scope here), or the mjswan PR you opened in step 8, with its URL; - anything you stopped on: dependency conflicts, missing credentials, task-specific gaps that mjswan must not absorb;
- that browser behaviour is unverified by this pipeline: parity proves the graph matches mjlab, but mjlab integrates with
mujoco_warpwhile the browser runs MuJoCo's WASM build, and a policy can behave differently under the two.
Sponsors
Details
- Marketplace
- ttktjmt/mjswan
- Category
- Community
- License
- Apache-2.0