[Bug]: One environment's constraint NaN kills the whole batched scene — no per-env fault isolation or failing-env index

[Enoch208]

Bug Description

With scene.build(n_envs=N), when any single environment's rigid solver produces invalid constraint forces, scene.step() raises GenesisException: Invalid constraint forces causing 'nan' for the entire scene. Two problems compound:

1. No quarantine — at n_envs=4096, one bad contact configuration destroys 4,095 healthy rollouts mid-step.
2. No environment index — the exception carries no indication of which environment went non-finite, so the failing rollout cannot be excluded, reset, or even analysed.

We hit this repeatedly in a contact-rich manipulation batch (16-link articulated cable + Franka, 16–64 envs, gs.amdgpu backend): one environment's gripper/floor/cable interaction goes NaN and the full batch dies.

Steps to Reproduce

The failure is contact-dependent and, on this stack, not deterministic — from bit-identical resets we measured contact rollouts diverging up to 256 mm in final cable position, so the repro is statistical rather than a fixed script. The reliable recipe:

1. Build a batched scene (16+ envs) with an articulated chain (~16 links) and a Franka, scene.build(n_envs=N).
2. Run a force-controlled grasp-and-drag policy against the chain so fingers, chain, and floor stay in sustained contact.
3. Within a few hundred steps, some run of the batch raises GenesisException from scene.step() for all environments.

Happy to provide our full scene setup and driver if useful (public at Enoch208/Crux, src/crux/simulation/batchscene.py).

Expected Behavior

Per-environment fault isolation — quarantine the offending environment (freeze it, flag it, or reset it) and let the remaining N−1 continue; at minimum, include the failing environment index (or a per-env validity mask) in the exception so callers can salvage and analyse.

Relevant log output

gs.GenesisException: Invalid constraint forces causing 'nan'.

(raised by scene.step(); our batch runner catches it, records every in-flight episode as failed-unstable, and salvages already-finished episodes — but the healthy rollouts are unrecoverable)

Environment

• OS: Ubuntu 24.04.4 LTS
• GPU/CPU: AMD Radeon PRO W7900 (gfx1100), 48 GB
• GPU-driver version: amdgpu, ROCm 7.2.1 (torch 2.13.0+rocm7.2)
• CUDA / CUDA-toolkit version: N/A (ROCm)

Release version or Commit ID

1.3.1

Additional Context

Large-batch evaluation is the headline use-case for GPU simulation, and blast-radius-of-one-env makes long unattended sweeps fragile: our 4,096-env throughput runs were only completable because episodes that finished before the explosion could be salvaged. Related in spirit to #2652 (NaN crash in the manipulation example), but this report is about the batching API contract rather than any specific NaN source.

[duburcqa]

You should improve this, especially the documentation. This is how env failure should be handled:

genesis-world/examples/locomotion/go2_env.py

Line 193 in 89beb5c

self._reset_idx(self.reset_buf)