Skip to content

Commit 8909d25

Browse files
AIFlowMLclaude
andcommitted
Phase 0: Port MJX foundation to MLX (types, math, dataclasses, io)
First 4 files of the JAX->MLX translation for MuJoCo XLA physics engine. Verified: imports work, quat_mul correct, normalize correct on MLX. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 88ba8ff commit 8909d25

8 files changed

Lines changed: 2831 additions & 0 deletions

File tree

.claude/settings.json

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
{
2+
"permissions": {
3+
"allow": [
4+
"Bash(*)",
5+
"Read(*)",
6+
"Write(*)",
7+
"Edit(*)",
8+
"Glob(*)",
9+
"Grep(*)",
10+
"Agent(*)",
11+
"WebFetch(*)",
12+
"WebSearch(*)",
13+
"NotebookEdit(*)",
14+
"Skill(*)",
15+
"ToolSearch(*)"
16+
],
17+
"deny": [
18+
"Bash(rm -rf /)",
19+
"Bash(sudo rm -rf /)",
20+
"Write(/etc/*)",
21+
"Write(/usr/*)",
22+
"Write(/System/*)",
23+
"Write(/Library/*)"
24+
]
25+
}
26+
}

TODO.md

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# MJX → MLX Port TODO
2+
3+
Port MuJoCo XLA (MJX) from JAX to Apple MLX. This creates the first MLX-native physics engine.
4+
5+
## Scope
6+
7+
MJX is a pure-Python GPU physics engine for MuJoCo. It uses JAX for GPU acceleration.
8+
We replace JAX with MLX to run on Apple Silicon Metal GPUs.
9+
10+
**Source:** `mjx/mujoco/mjx/_src/` — 19,393 lines across ~30 core files
11+
**Target:** `mjx/mujoco/mjx_mlx/_src/` — same API, MLX backend
12+
13+
## Translation Rules
14+
15+
| JAX | MLX |
16+
|-----|-----|
17+
| `import jax` | `import mlx.core as mx` |
18+
| `import jax.numpy as jnp` | `import mlx.core as mx` |
19+
| `jnp.array(...)` | `mx.array(...)` |
20+
| `jnp.zeros(...)` | `mx.zeros(...)` |
21+
| `jnp.ones(...)` | `mx.ones(...)` |
22+
| `jnp.concatenate(...)` | `mx.concatenate(...)` |
23+
| `jnp.stack(...)` | `mx.stack(...)` |
24+
| `jnp.where(...)` | `mx.where(...)` |
25+
| `jnp.dot(...)` | `mx.matmul(...)` or `mx.inner(...)` |
26+
| `jnp.einsum(...)` | Manual implementation or `mx.einsum` if available |
27+
| `jnp.linalg.norm(...)` | `mx.sqrt(mx.sum(mx.square(...)))` |
28+
| `jnp.float32` | `mx.float32` |
29+
| `jax.jit(fn)` | `mx.compile(fn)` |
30+
| `jax.vmap(fn)` | Loop or manual batching (MLX has limited vmap) |
31+
| `jax.lax.scan(...)` | Python loop with `mx.eval()` |
32+
| `jax.lax.cond(...)` | `mx.where(...)` or Python if |
33+
| `jax.lax.switch(...)` | Python if/elif chain |
34+
| `jax.lax.fori_loop(...)` | Python for loop |
35+
| `jax.lax.while_loop(...)` | Python while loop |
36+
| `jax.lax.dynamic_slice(...)` | Array slicing |
37+
| `jax.tree_util.tree_map(...)` | Manual or helper function |
38+
| `@jax.custom_vjp` | Not needed (MLX has autograd) |
39+
| `jax.random.PRNGKey(...)` | `mx.random.key(...)` |
40+
| `jax.random.split(...)` | Manual key management |
41+
| `functools.partial(jax.jit, ...)` | `mx.compile(...)` |
42+
| `chex` assertions | Plain Python assertions |
43+
| `flax.struct.dataclass` | `@dataclass` or custom |
44+
45+
## Key Differences to Handle
46+
47+
1. **vmap**: MLX doesn't have full vmap. Batch dimensions must be explicit.
48+
2. **scan**: MLX doesn't have lax.scan. Use Python loops.
49+
3. **Custom VJP**: MLX autograd handles most cases. Remove custom_vjp decorators.
50+
4. **Tree utilities**: Replace jax.tree_util with manual traversal or a simple helper.
51+
5. **Dynamic shapes**: MLX is less flexible with dynamic shapes. May need padding.
52+
6. **In-place ops**: MLX arrays are immutable like JAX. No issue.
53+
7. **Device placement**: MLX auto-places on GPU. Remove jax.devices() calls.
54+
55+
## Phase 0: Foundation
56+
- [x] Fork repo, set remotes
57+
- [x] Push to RobotFlow-Labs/Mujoco-mlx
58+
- [ ] Create `mjx/mujoco/mjx_mlx/` package structure
59+
- [ ] Port `_src/types.py` (data structures — no compute, just dataclasses)
60+
- [ ] Port `_src/math.py` (linear algebra primitives)
61+
- [ ] Port `_src/dataclasses.py` (MJX dataclass utilities)
62+
- [ ] Port `_src/io.py` (model loading — bridges C MuJoCo to Python)
63+
- [ ] Validate: types + math + io import and pass basic tests
64+
65+
## Phase 1: Core Dynamics
66+
- [ ] Port `_src/support.py` (helper functions used everywhere)
67+
- [ ] Port `_src/smooth.py` (smooth dynamics — kinematics, mass matrix)
68+
- [ ] Port `_src/passive.py` (passive forces — gravity, springs)
69+
- [ ] Port `_src/constraint.py` (constraint computation)
70+
- [ ] Port `_src/solver.py` (constraint solver — CG, Newton)
71+
- [ ] Port `_src/forward.py` (forward dynamics — the main loop)
72+
- [ ] Port `_src/inverse.py` (inverse dynamics)
73+
- [ ] Port `_src/derivative.py` (finite-difference derivatives)
74+
- [ ] Validate: forward step on cartpole model matches C MuJoCo
75+
76+
## Phase 2: Collision
77+
- [ ] Port `_src/collision_types.py` (collision data structures)
78+
- [ ] Port `_src/collision_primitive.py` (sphere, capsule, box)
79+
- [ ] Port `_src/collision_convex.py` (convex mesh collision)
80+
- [ ] Port `_src/collision_sdf.py` (signed distance fields)
81+
- [ ] Port `_src/collision_driver.py` (collision dispatch)
82+
- [ ] Port `_src/bvh.py` (bounding volume hierarchy)
83+
- [ ] Validate: collision detection matches C MuJoCo on test scenes
84+
85+
## Phase 3: Sensors and Rendering
86+
- [ ] Port `_src/sensor.py` (sensor simulation)
87+
- [ ] Port `_src/ray.py` (ray casting)
88+
- [ ] Port `_src/mesh.py` (mesh utilities)
89+
- [ ] Port `_src/render.py` (basic rendering)
90+
- [ ] Port `_src/render_util.py` (rendering helpers)
91+
- [ ] Port `_src/scan.py` (parallel scan operations)
92+
- [ ] Validate: sensor outputs match C MuJoCo
93+
94+
## Phase 4: Tests and Benchmarks
95+
- [ ] Port all `*_test.py` files from JAX assertions to MLX
96+
- [ ] Run numerical comparison: MJX-MLX vs MJX-JAX vs C MuJoCo
97+
- [ ] Benchmark: steps/second on Apple Silicon vs JAX on GPU
98+
- [ ] Integration tests: cartpole, humanoid, ant
99+
100+
## Phase 5: Integration with IsaacLab-MLX
101+
- [ ] Wire MJX-MLX as physics backend for IsaacLab mac-sim
102+
- [ ] Replace analytical dynamics with MJX-MLX forward step
103+
- [ ] Validate: IsaacLab cartpole task runs on MJX-MLX physics
104+
- [ ] Performance comparison: analytical mac-sim vs MJX-MLX
105+
106+
## File-by-File Porting Checklist
107+
108+
| File | Lines | Phase | Status |
109+
|------|-------|-------|--------|
110+
| `types.py` | 685 | 0 | TODO |
111+
| `math.py` | 430 | 0 | TODO |
112+
| `dataclasses.py` | 280 | 0 | TODO |
113+
| `io.py` | 890 | 0 | TODO |
114+
| `support.py` | 1,200 | 1 | TODO |
115+
| `smooth.py` | 2,100 | 1 | TODO |
116+
| `passive.py` | 350 | 1 | TODO |
117+
| `constraint.py` | 1,800 | 1 | TODO |
118+
| `solver.py` | 1,500 | 1 | TODO |
119+
| `forward.py` | 850 | 1 | TODO |
120+
| `inverse.py` | 400 | 1 | TODO |
121+
| `derivative.py` | 300 | 1 | TODO |
122+
| `collision_types.py` | 200 | 2 | TODO |
123+
| `collision_primitive.py` | 1,500 | 2 | TODO |
124+
| `collision_convex.py` | 1,200 | 2 | TODO |
125+
| `collision_sdf.py` | 800 | 2 | TODO |
126+
| `collision_driver.py` | 600 | 2 | TODO |
127+
| `bvh.py` | 500 | 2 | TODO |
128+
| `sensor.py` | 900 | 3 | TODO |
129+
| `ray.py` | 600 | 3 | TODO |
130+
| `mesh.py` | 500 | 3 | TODO |
131+
| `render.py` | 400 | 3 | TODO |
132+
| `render_util.py` | 300 | 3 | TODO |
133+
| `scan.py` | 200 | 3 | TODO |
134+
| **TOTAL** | **~18,000** | | |

mjx/mujoco/mjx_mlx/__init__.py

Whitespace-only changes.

mjx/mujoco/mjx_mlx/_src/__init__.py

Whitespace-only changes.
Lines changed: 226 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,226 @@
1+
# Copyright 2023 DeepMind Technologies Limited
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
# ==============================================================================
15+
"""MLX dataclass utilities for MJX.
16+
17+
Ported from JAX pytree dataclasses to plain Python dataclasses with MLX array
18+
support. Instead of JAX pytree registration, we provide a `tree_map` helper
19+
that applies a function to all mlx.core.array fields in a dataclass.
20+
"""
21+
22+
import copy
23+
import dataclasses
24+
from typing import Any, Callable, Dict, Optional, Sequence, Tuple, TypeVar
25+
26+
import mlx.core as mx
27+
import numpy as np
28+
29+
_T = TypeVar('_T')
30+
31+
32+
def _is_array_field(value: Any) -> bool:
33+
"""Check if a value is an MLX array or a nested dataclass containing arrays."""
34+
if isinstance(value, mx.array):
35+
return True
36+
if isinstance(value, np.ndarray):
37+
return False # numpy arrays are treated as metadata, not mapped over
38+
if dataclasses.is_dataclass(value) and not isinstance(value, type):
39+
return True
40+
if isinstance(value, (list, tuple)):
41+
return any(_is_array_field(v) for v in value)
42+
if isinstance(value, dict):
43+
return any(_is_array_field(v) for v in value.values())
44+
return False
45+
46+
47+
def _apply_to_arrays(fn: Callable, value: Any) -> Any:
48+
"""Recursively apply fn to all mx.array leaves in a nested structure."""
49+
if isinstance(value, mx.array):
50+
return fn(value)
51+
if dataclasses.is_dataclass(value) and not isinstance(value, type):
52+
return tree_map(fn, value)
53+
if isinstance(value, tuple):
54+
mapped = tuple(_apply_to_arrays(fn, v) for v in value)
55+
# Preserve namedtuple types
56+
if hasattr(type(value), '_fields'):
57+
return type(value)(*mapped)
58+
return mapped
59+
if isinstance(value, list):
60+
return [_apply_to_arrays(fn, v) for v in value]
61+
if isinstance(value, dict):
62+
return {k: _apply_to_arrays(fn, v) for k, v in value.items()}
63+
return value
64+
65+
66+
def tree_map(fn: Callable, node: _T, *rest: Any) -> _T:
67+
"""Apply fn to all mx.array fields in a dataclass, returning a new instance.
68+
69+
This is the MLX replacement for jax.tree_util.tree_map. It recursively
70+
traverses dataclass fields and applies fn to every mx.array leaf.
71+
72+
Args:
73+
fn: function to apply to each mx.array leaf.
74+
node: a dataclass instance to map over.
75+
*rest: additional dataclass instances to map over in parallel (like
76+
jax.tree_util.tree_map with multiple trees).
77+
78+
Returns:
79+
A new dataclass instance with fn applied to all array fields.
80+
"""
81+
if not dataclasses.is_dataclass(node) or isinstance(node, type):
82+
raise TypeError(f'tree_map expects a dataclass instance, got {type(node)}')
83+
84+
if rest:
85+
# Multi-tree map: apply fn(leaf_from_node, leaf_from_rest0, ...)
86+
return _tree_map_multi(fn, node, *rest)
87+
88+
updates = {}
89+
for field in dataclasses.fields(node):
90+
value = getattr(node, field.name)
91+
mapped = _apply_to_arrays(fn, value)
92+
if mapped is not value:
93+
updates[field.name] = mapped
94+
95+
if updates:
96+
return dataclasses.replace(node, **updates)
97+
return node
98+
99+
100+
def _apply_to_arrays_multi(fn: Callable, *values: Any) -> Any:
101+
"""Multi-tree version of _apply_to_arrays."""
102+
first = values[0]
103+
if isinstance(first, mx.array):
104+
return fn(*values)
105+
if dataclasses.is_dataclass(first) and not isinstance(first, type):
106+
return _tree_map_multi(fn, *values)
107+
if isinstance(first, tuple):
108+
mapped = tuple(
109+
_apply_to_arrays_multi(fn, *(v[i] for v in values))
110+
for i in range(len(first))
111+
)
112+
if hasattr(type(first), '_fields'):
113+
return type(first)(*mapped)
114+
return mapped
115+
if isinstance(first, list):
116+
return [
117+
_apply_to_arrays_multi(fn, *(v[i] for v in values))
118+
for i in range(len(first))
119+
]
120+
if isinstance(first, dict):
121+
return {
122+
k: _apply_to_arrays_multi(fn, *(v[k] for v in values))
123+
for k in first
124+
}
125+
return first
126+
127+
128+
def _tree_map_multi(fn: Callable, node: _T, *rest: Any) -> _T:
129+
"""Multi-tree map implementation."""
130+
updates = {}
131+
for field in dataclasses.fields(node):
132+
values = [getattr(node, field.name)] + [
133+
getattr(r, field.name) for r in rest
134+
]
135+
mapped = _apply_to_arrays_multi(fn, *values)
136+
if mapped is not values[0]:
137+
updates[field.name] = mapped
138+
139+
if updates:
140+
return dataclasses.replace(node, **updates)
141+
return node
142+
143+
144+
def dataclass(clz: _T) -> _T:
145+
"""Create a frozen dataclass with a replace() method.
146+
147+
MLX port: no pytree registration needed. We just create a standard frozen
148+
dataclass and attach `replace` as a convenience method.
149+
150+
Args:
151+
clz: the class to register as a dataclass.
152+
153+
Returns:
154+
The resulting frozen dataclass.
155+
"""
156+
data_clz = dataclasses.dataclass(frozen=True)(clz)
157+
data_clz.replace = dataclasses.replace
158+
return data_clz
159+
160+
161+
TNode = TypeVar('TNode', bound='PyTreeNode')
162+
163+
164+
class PyTreeNode:
165+
"""Base class for dataclasses that should act like array-carrying nodes.
166+
167+
MLX port of the JAX PyTreeNode. Instead of registering with JAX's pytree
168+
system, this simply creates a frozen dataclass with convenience methods.
169+
The `tree_map` function can be used to map over array fields.
170+
"""
171+
172+
def __init_subclass__(cls, **kwargs):
173+
# Pop register_as_pytree if passed (for compatibility), but ignore it
174+
kwargs.pop('register_as_pytree', None)
175+
super().__init_subclass__(**kwargs)
176+
dataclass(cls)
177+
178+
def __init__(self, *args, **kwargs):
179+
# stub for type checkers
180+
raise NotImplementedError
181+
182+
def replace(self: TNode, **overrides) -> TNode:
183+
# stub for type checkers
184+
raise NotImplementedError
185+
186+
@classmethod
187+
def fields(cls) -> Tuple[dataclasses.Field, ...]: # pylint: disable=g-bare-generic
188+
return dataclasses.fields(cls)
189+
190+
def tree_replace(
191+
self, params: Dict[str, Optional[mx.array]]
192+
) -> 'PyTreeNode':
193+
"""Replace nested fields using dot-separated keys."""
194+
new = self
195+
for k, v in params.items():
196+
new = _tree_replace(new, k.split('.'), v)
197+
return new
198+
199+
200+
def _tree_replace(
201+
base: PyTreeNode,
202+
attr: Sequence[str],
203+
val: Optional[mx.array],
204+
) -> PyTreeNode:
205+
"""Sets attributes in a dataclass with values."""
206+
if not attr:
207+
return base
208+
209+
# special case for List attribute
210+
if len(attr) > 1 and isinstance(getattr(base, attr[0]), list):
211+
lst = copy.deepcopy(getattr(base, attr[0]))
212+
213+
for i, g in enumerate(lst):
214+
if not hasattr(g, attr[1]):
215+
continue
216+
v = val if not hasattr(val, '__iter__') else val[i]
217+
lst[i] = _tree_replace(g, attr[1:], v)
218+
219+
return base.replace(**{attr[0]: lst})
220+
221+
if len(attr) == 1:
222+
return base.replace(**{attr[0]: val})
223+
224+
return base.replace(
225+
**{attr[0]: _tree_replace(getattr(base, attr[0]), attr[1:], val)}
226+
)

0 commit comments

Comments
 (0)