NovaSky Releases IsoExec to Fix AI Training Math Bugs

During reinforcement learning post-training, the rollout engine and the training engine often disagree on token log-probabilities due to non-associative floating-point arithmetic. NovaSky has addressed this with IsoExec, an open-source abstraction that enforces a machine-checkable execution contract across engines. This contract pins down kernels, data types, and reduction orders, verifying them via SHA-256 identity digests. By ensuring bitwise identical log-probabilities, IsoExec prevents the silent corruption of policy gradient signals that can ruin training runs.

The consequences of these rounding discrepancies are severe. In a GLM-5.2 run with a train-inference KL divergence of approximately 0.013, clipping discarded about 45% of tokens, causing the reward to collapse around step 20. A bitwise-aligned run, by contrast, experienced zero clipped tokens and remained stable. To achieve this alignment, IsoExec introduces parallelism-invariant kernels that maintain consistency across tensor, expert, and sequence parallel layouts. In an 8xH100 run training the Qwen3.5-35B-A3B model with synchronous DAPO, IsoExec reduced the mean rollout-versus-training logprob difference from 1.6e-2 to 6.7e-7, while the maximum per-step difference dropped from 5.073.

To align training, prefill, and decode without the massive slowdowns of running recurrent algorithms everywhere, IsoExec uses chunkwise-parallel recurrent (CPR) Gated DeltaNet. On an H100, CPR trainer forward and backward passes for 10240 tokens take 7.386 ms (a 1.43x overhead compared to 5.177 ms for native mixed, but far better than the 22.863 ms or 4.42x of recurrent-everywhere). Rollout prefill for 5x2048 tokens takes 1.412 ms (1.67x overhead vs 0.844 ms native), and rollout decode for 256x1 tokens takes 0.0846 ms (1.38x overhead vs 0.0612 ms native). Overall, a full RL step on Qwen3.5-35B-A3B saw its time increase from 1224.6 to 1534.0 seconds, representing a 25.3% overhead, with generation rising 31.3% (591.3s to 776.6s) and policy training rising 18.6% (498.6s to 591.3s).

For AI practitioners, IsoExec preserves vLLM's scheduler, paged KV cache, and CUDA graphs while integrating directly into existing SkyRL, vLLM, and Megatron stacks. This allows engineering teams to confidently attribute training divergence to actual algorithmic or environmental flaws rather than hunting down elusive discrepancies hidden within kernel reduction orders.