The Echelon Optimizer
Muon’s update at the cost of a low-rank one: up to 10× faster optimizer steps, and proof checked in Lean.
Each pause on a foothold is one optimizer step: Muon’s takes 10.5× as long as Echelon’s (70B-class layer shapes, batched, rank 64).
What we found (tl;dr)
- Muon flattens every singular value, at a cost that grows with the cube of the layer width. Echelon flattens only the top and scales the rest down, at cost.
- On one GPU, the optimizer step for 70B-class layers drops from 33.9 s to 3.23 s. At 1.3B the optimizer is 2.5× faster and each training step 14% shorter, with validation loss within seed noise of Muon’s over 3 seeds each.
- The gains grow on big MoE models. On DeepSeek V4.1 Flash, each training step takes about 35% less time on 16 B300s (about 31% on a more realistic 24), measured on B300 at rank 32, communication not counted.
- We proved that Echelon’s update stays bounded and that it converges.
Performance at a glance
What it saves: training time and cost
A faster optimizer step only saves the share of the training step that the optimizer takes. If Muon’s optimizer is a fraction of each step and Echelon makes it times faster, the time saved per step is
GPU time is what you pay for, so the same fraction comes off the compute bill. Here is what we measured, first in our two training runs, then on the real layer shapes of five current open models:
| Model | Share f | Speedup s | Saved |
|---|---|---|---|
| Trained end to end, one GPU, 32k tokens per step; speedup measured in training | |||
| GPT-2 small | 10.9% | 1.1× | 1.0% |
| 1.3B dense | 23.5% | 2.5× | 14.2% |
| Timed on each model’s real layer shapes, rank 32, 256k tokens per step, each on the fewest GPUs that hold it | |||
| Qwen3.8-27B (dense), 3 RTX PRO | 6.8% | 17.6× | 6.4% |
| Nemotron 3.5 Lightning 30B-A3B, 3 RTX PRO | 21.6% | 11.5× | 19.7% |
| Mistral Small 4 119B-A6.5B, 10 RTX PRO | 31.5% | 10.4× | 28.5% |
| Reflection Beam 501B-A23B, 43 RTX PRO | 28.5% | 11.7× | ≈26.1% |
| DeepSeek V4.1 Flash 552B-A16B, 16 B300 | 47.3% | 3.7× | 34.6% |
Each lower row puts the model on the fewest GPUs that hold its training state (bf16 weights and gradients, fp32 momentum) and splits the optimizer’s work across them: DeepSeek V4.1 Flash on 16 B300s, timed on a B300 (about 31% on a more realistic 24), and the others on RTX PRO 6000s, timed there: Beam on 43, Mistral Small 4 on 10, Nemotron 3.5 Lightning and Qwen3.8-27B on 3 each. Communication is not counted. The speedup depends on the hardware: on DeepSeek V4.1 Flash’s layer shapes at rank 32 the optimizer step is 10.7× faster on the RTX PRO 6000 but 3.7× on a B300, because Muon’s matrix products run 3.4× faster there.
Optimizer speedup on the whole DeepSeek V4.1 Flash model, Echelon vs Muon
| Rank | B300 | RTX PRO 6000 |
|---|---|---|
| 32 | 3.7× | 10.7× |
| 64 | 3.4× | 7.3× |
| 128 | 2.8× | 3.0× |
| 256 | 2.0× | 2.3× |
| 320 (width/16) | 1.7× | 2.0× |
| 512 | 1.2× | 1.4× |
Best Echelon implementation per GPU (fused kernel or batched); whole-model optimizer step on one GPU.
And here is what the same formula gives for other optimizer shares and the speedups we measured at each width (batched implementation at rank 64 on one RTX PRO 6000, against the faster of per-layer and batched Muon). Wider layers raise both numbers, and mixture-of-experts models raise further, since the optimizer updates every expert while each token uses only a few (see below). These cells are projections, not measurements.
| Share f | s = 1.4× (d768) | 3.8× (d2048) | 5.9× (d4096) | 10.5× (d8192) |
|---|---|---|---|---|
| 10% | 3% | 7% | 8% | 9% |
| 25% | 7% | 18% | 21% | 23% |
| 50% | 14% | 37% | 42% | 45% |
To put it in money, plug in your own run
On DeepSeek V4.1 Flash, Echelon takes about 35% off each training step on 16 B300s (about 31% on a more realistic 24), measured on B300 at rank 32, communication not counted. A frontier lab can absorb that overhead. A university group, a startup or an open-source project often can’t: for them, 100,000 versus about 65,500 GPU-hours (on 16 B300s) can decide whether a run happens at all, or how many ablations fit in the budget. Echelon’s update is a few thin matrix products whose cost grows with the rank, not the cube of the width, so the saving is largest on big mixture-of-experts models like DeepSeek V4.1 Flash. Our 1.3B result was trained on a single GPU. The code (including the fused kernel) and a package will be released on GitHub soon, to put strong training within reach of more people.
Muon is one of the strongest optimizers for the hidden layers of transformers. Its update is easy to state: it takes the momentum matrix, keeps its singular vectors, and sets every singular value to one. The hard part is doing it fast. Muon approximates it with five Newton–Schulz iterations. For a square layer of width they cost floating-point operations per step, while everything else in the optimizer costs .
At frontier widths that cubic term is no longer small. On one Blackwell GPU, the Newton–Schulz step over every hidden matrix of a model with 70B-class layer shapes takes 33.9 seconds. Real training shards that work across GPUs, but it still grows with the cube of the width.
This post introduces the Echelon optimizer, a cheaper way to get most of what Muon does. For an matrix and a rank you choose, it costs . It has a clear meaning in terms of singular values, and it is provably stable even when its internal estimates are wrong. On the same GPU it cuts the 33.9-second step to 3.23 seconds. On DeepSeek V4.1 Flash it takes about 35% off each training step on 16 B300s (about 31% on a more realistic 24), measured on B300 at rank 32, communication not counted. In our 1.3B runs (3 seeds per optimizer) its validation loss matched Muon’s within seed noise, while each training step took 14% less time.
We also wrote the convergence argument in Lean, a proof assistant, and had the computer check every step. That is still rare for optimizer work.
The math, explained
What Muon actually computes
We write the momentum of one weight matrix as its singular value decomposition, . Muon wants its polar factor, : the same directions, with every singular value replaced by one.
The polar factor is the linear-minimization oracle of the spectral-norm ball. We follow Bernstein and Newhouse’s anthology and Pethick et al.’s LMO view.
There is a good reason for that choice. Among all matrices with spectral norm at most one, the polar factor is the one that lines up best with :
So it is steepest descent when we measure step size in the spectral norm, which is a natural way to measure how much a linear layer can change its outputs.
What each matrix does to the unit circle. Drag the two columns of M.
The same candidate, as stretches along M’s singular directions. Drag O.
- alignment ⟨M, O⟩
- –
- best possible, σ₁ + σ₂
- –
- ∥O∥₂
- –
Try
Computing exactly needs an SVD, which GPUs do badly. Muon instead normalizes by its Frobenius norm and applies an odd quintic polynomial five times, with . Only matrix multiplies, each of cost proportional to . The polynomial is tuned for speed rather than accuracy. It does not send singular values to exactly one; it shoves them all, large and small, into a band around .
Across: a singular value before the step. Up: the same value after it. Drag the largest value of M in the row under the axis.
Try
the same values, step by step
- inside the band now
- –
- smallest, largest
- –
largest value: –
Splitting the spectrum
Gradients and momenta of trained networks are far from full rank: a few directions carry most of the mass and the rest decay. That suggests treating the top of the spectrum and the rest differently. Echelon keeps one extra piece of state per matrix, an orthonormal frame that tracks the top- left singular directions of the momentum from one step to the next. Each step:
- Refresh. We refresh the frame with one power step on the current momentum: .O(mnr)
- Head. We project, , a small matrix, and take its polar factor . It is small, so this is cheap: exact on CPU, five Newton–Schulz steps on the small matrix on GPU.O(mnr)
- Tail. The rest of the momentum is . We never store it, and estimate its largest singular value with two power iterations, times a 1.05 margin.O(mn)
- Output. We return .O(mnr)
w = Xz − U(Hz), z = Xᵀw − Hᵀ(Uᵀw)
τ = 1.05 · ∥w∥
- Flops per step
- Echelon 6slr + 6(s + l)r² + 20r³ + 10sl · Muon 5(4s²l + 2s³)
- Memory traffic
- first version: about 20 passes, 78 bytes per element · fused kernel: 6 passes (A–F), 20 bytes per element
In total that is flops against Muon’s : linear in the width for a fixed rank, where Muon is cubic. When the tail is empty and the output is exactly Muon’s ideal, the polar factor.
Warm-started subspace tracking is the trick at the heart of Dion. What differs here is what happens to the head and the tail.
Here is the part we like. We assume the frame is exactly right, so spans , and . Then the head contributes and the tail contributes . Together they give:
Echelon is a spectral clip. The top directions get equalized, as in Muon. The rest keep their shape, rescaled so the largest of them is exactly one. The threshold is not a tuned constant: it is set by the rank you choose and moves with the scale of the momentum. At it is close to normalized gradient descent; at full rank it is Muon.
Output singular value, direction by direction. Drag the blue line to set the rank r.
The map each method applies to a singular value.
Try
score = ⟨M, O⟩ / (∥M∥∗ ∥O∥₂); the exact polar factor scores 1. flatness = stable rank of O ÷ width.
The clip picture needs an exact frame. Ours is refreshed by a single power step per training step, so it is always a little stale. The second property is what makes that safe. For any orthonormal frame, however wrong, the head lives inside the span of and the tail lives outside it, so they cannot add up in the same direction:
As long as the head’s factor has norm at most one and τ is at least the tail’s largest singular value, a stale frame never costs stability. What it costs is alignment per unit of norm: the raw alignment can even rise slightly at small angles, because a longer update collects more of it, but alignment divided by the spectral norm, the score, falls from the first degree. In our training runs these conditions usually don’t hold exactly: the estimate of the tail’s norm runs about 5–8% low and the head’s factor reaches norm 1.2, so the bound is a guide rather than a guarantee. In practice the update’s norm stays near 1.2 and exceeds √2 mostly in the first 50 steps.
Turn the frame. Drag the blue ray away from the true top-r subspace. Distance from the corner is the value at that angle.
Why it cannot pass √2. One dot per singular direction of O.
- ∥O∥₂ at this angle
- –
- ceiling √2
- 1.414
- largest over all angles
- –
- alignment ⟨M, O⟩ / ∥M∥∗
- –
- with an exact frame
- –
- score = alignment ÷ norm
- –
To bring the norm back down to one, we apply a single cubic Newton–Schulz step, . Its scalar map sends all of into , so it fixes the norm even when underestimates the tail by up to . We call this polishing.
Across: a singular value of the update before polishing. Up: the same value after. Drag the handle to make τ too small.
Try
- τ is
- –
The alignment Echelon gives up has a short formula. Since and ,
The loss is zero when the tail is flat and at most the tail’s nuclear norm otherwise. Throwing the tail away loses all of it, which is exactly why “head only” was the worst variant in every sweep we ran.
Where it fits best: wide layers and mixture-of-experts
An optimizer speedup only matters in proportion to how much of the training step the optimizer takes. Two things decide that share: how wide each matrix is, and how many tokens each matrix sees per step.
Why wider layers
For a square layer trained on tokens per step, the forward and backward matmuls cost about flops, while Muon’s Newton–Schulz costs . Their ratio is
So Muon’s overhead grows linearly with width at a fixed batch. Echelon replaces with about , and the rank it needs grows much more slowly than the width: in our measurements on real momentum it was a quarter of the width at 256 and between a sixteenth and an eighth at 1024. Both effects point the same way, which is why Figure 1 climbs from 2.0× (1.4× against batched Muon) to 10.5× as the width goes from 768 to 8192.
Why mixture-of-experts
A mixture-of-experts (MoE) model replaces each feed-forward block with many expert feed-forward blocks and routes each token to only a few of them. That is what makes MoEs attractive: the model holds many parameters but each token only pays for the few experts it visits, so you get the capacity of a large model at the per-token compute of a small one.
The optimizer does not get that discount. Every expert’s weight matrices are updated every step, so the optimizer’s cost scales with the total parameter count, while the training compute scales with the active count. With experts and active per token, each expert sees only about tokens per step, and the ratio above becomes roughly
With 256 experts and 8 active, as in several current open models, : the orthogonalization share is tens of times larger than in a dense model of the same width. Expert matrices in today’s MoEs are also wide in both directions, for example 5120×2560 in a model that holds 21× more total than active parameters. That is where a faster optimizer step turns into end-to-end time.
Three more properties line up with Echelon specifically:
- Low-rank gradients. A layer’s gradient is a sum of one outer product per token it saw, so an expert that sees fewer tokens per micro-batch has lower-rank gradients, and a small head rank captures more of its momentum.
- Many identical shapes. Hundreds of experts with the same shape are exactly what the batched implementation wants: one call per shape group, so launch latency disappears.
- Bounded even when stale. Rarely routed experts update their tracked frame less usefully, and the bound means a stale frame can cost score but never stability.
One training step, as a share of Muon’s step time.
- saved = f (1 − 1/s)
- –
- ceiling if Echelon were free
- –
Measured: share of the step saved, as tokens per step grow. Pick a run to highlight it and load its numbers above.
Trained end to end, one GPU, 32k tokens/step
Timed shapes, each on the fewest GPUs that hold it
Tokens per step
Measured on real model shapes
To check the argument on models people train today, we took four mixture-of-experts models and one dense model and timed both optimizers over every matrix Muon would update, on one NVIDIA RTX PRO 6000 Blackwell, and DeepSeek V4.1 Flash also on one B300. We also timed one real layer’s forward and backward pass in bf16 at several batch sizes, and combined the parts into an estimated step on the fewest GPUs that hold each model, with the optimizer’s work and the tokens split across them. Echelon runs at rank 32 (our fused kernel on the RTX PRO 6000; on the B300 the faster of the fused kernel and the batched implementation); at rank 64 the DeepSeek V4.1 Flash row saves 33.5% instead of 34.6%.
| Model | GPUs | Muon opt. | Echelon opt. | 128k | 256k | 1M | 4M |
|---|---|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash 552B-A16B | 16 B300 | 2.13 s | 0.57 s | 40.1% | 34.6% | ≈15.5% | ≈4.6% |
| Reflection Beam 501B-A23B | 43 RTX PRO | 2.67 s | 0.23 s | ≈31.3% | ≈26.1% | 13.0% | 4.4% |
| Mistral Small 4 119B-A6.5B | 10 RTX PRO | 2.34 s | 0.23 s | ≈39.6% | 28.5% | 10.9% | ≈3.0% |
| Nemotron 3.5 Lightning 30B-A3B | 3 RTX PRO | 2.28 s | 0.20 s | 31.7% | 19.7% | ≈6.0% | ≈1.6% |
| Qwen3.8-27B (dense) | 3 RTX PRO | 3.03 s | 0.17 s | 11.9% | 6.4% | ≈1.7% | ≈0.4% |
Each model on the fewest GPUs that hold its training state, timed on that GPU type. Optimizer columns: seconds per GPU per step, whole model split over those GPUs. The other columns: share of step time saved at that many tokens per optimizer step, summed over all GPUs. ≈: the forward and backward time is computed outside the measured tokens per GPU (extended below the smallest measured batch, or accumulated above the largest).
Which runs benefit
- Good fit: fine-tuning and RL policy updates of large mixture-of-experts models, at roughly 128k to 1M tokens per optimizer step. That is where the optimizer is a large share of the step: 31–40% saved at 128k, 13–15% at 1M for DeepSeek (16 B300s) and Beam (43 RTX PRO 6000s), each on the fewest GPUs that hold it, partly computed outside the measured tokens per GPU.
- Small dense models at small batches: the 1.3B run saved 14% at 32k tokens per step.
How much of Muon it keeps
We trained a GPT-2-small-shaped model (width 768, 12 layers; 163M parameters with an untied output head) on FineWeb for 1,500 steps of 32k tokens.
| Optimizer | Val loss | Share of Muon’s gain over AdamW |
|---|---|---|
| Muon (3 seeds) | 4.6975 ± 0.0024 | 100% |
| Echelon, r=96 (3 seeds) | 4.7624 ± 0.0027 | 87% |
| Echelon, r=32 + one polish step (1 seed) | 4.801 | 81% |
| AdamW (best of two learning rates, 1 seed) | 5.199 | 0% |
At rank 96, one eighth of the width, Echelon keeps 87% of Muon’s improvement over AdamW. On a much smaller 4-layer character model of width 256, averaged over three seeds, rank 64 keeps about two thirds (Muon 1.561 ± 0.012, Echelon 1.587 ± 0.007, AdamW 1.639 ± 0.009). The gap is smaller in our larger runs, though those runs also differ in depth, rank, length and number of seeds.
One reason to expect it to shrink is that the rank you need grows much more slowly than the width. We saved real momentum from Muon training runs of a small model at widths 256, 512 and 1024 and scored each method against an exact SVD. Echelon matches Muon’s score at rank 64 for widths 256 and 512, a quarter and an eighth of the width, and needs between 64 and 128 at width 1024, a sixteenth to an eighth. Over the same range the momentum’s effective rank grew only about 1.2× per doubling of width.
At 1.3B parameters the gap was within seed noise. We trained a GPT with width 2048 and 24 layers on FineWeb for 2,000 steps of 32k tokens (about 65M tokens, three seeds per optimizer and one for AdamW, one Blackwell GPU with gradient accumulation; mean ± s.d. over seeds):
| Optimizer | Val loss | Optimizer ms/step | Full step ms | Tokens/s |
|---|---|---|---|---|
| Muon | 4.3757 ± 0.0109 | 273 | 1,162 | 28,205 |
| Echelon, r=128 | 4.3751 ± 0.0115 | 111 | 997 | 32,860 |
| AdamW (one learning rate, one seed) | 5.184 | 28 | 915 | 35,796 |
Rank 128 is one sixteenth of the width. Over three seeds, Echelon’s final loss is 0.0006 ± 0.0017 below Muon’s (paired by seed), while the spread from seed to seed is about 0.011, so the two are level within seed noise. Averaged over the three seeds it reaches AdamW’s final loss at about step 670, after about 11 minutes of training against about 30 for an untuned AdamW (one seed): roughly 2.7× less wall-clock time and GPU cost (Figures 14 and 15 show seed 0).
Steps are not what you pay for. Plotted against wall-clock time, Echelon’s cheaper step moves its whole curve to the left: at 1.3B it finishes the same 2,000 steps 5.5 minutes sooner than Muon at the same loss. On one GPU, cost is wall-clock time multiplied by the hourly price, so the same picture is also a cost curve. The price field above the plot rescales the top axis.
Estimated, 2,000 steps on 16 B300 GPUs: Echelon 1.6 h · $79, Muon 2.5 h · $120.
In GPU-hours the 2,000 steps took 0.555 for Echelon and 0.646 for Muon (2,000 times the mean step time on one GPU, without evaluation).
Using Echelon
- Where
- The matrices Muon handles: attention and MLP or expert weights. Embeddings, the output head and norms stay on AdamW.
- Settings
- Muon’s carry over unchanged: learning rate 0.02, Nesterov momentum 0.95, each update scaled by √max(1, rows/cols), no weight decay. Same warmup and cosine schedule.
- Rank
- What worked: a quarter of the width at 256 (r=64), an eighth at 768 (r=96) and a sixteenth at 2048 (r=128). Our suggestion is to start at width/16, and not below 64.
- Tail
- Two power iterations and a 1.05 margin on τ; no polishing in the quality runs.
- Memory
- The frame Q and the power-iteration vector add m·r + n numbers per m×n matrix (m the short side): 1.3% more than the weights for a 2304×5120 expert at r=64, 2.5% at r=128.
- Implementation
- We stack same-shaped matrices and process each stack in one call, use bf16 for the large thin products and fp32 for everything small, and orthonormalize with Cholesky-QR plus a tiny diagonal shift, as two half-steps.
- Sharding
- A sharded version runs on the DTensor layouts that FSDP2, tensor and expert parallelism produce: rows, columns, 2-D blocks and expert stacks. On 4 processes it matches single-process Echelon to within 2.1×10−6 relative with fp32 products, over three steps and eleven layouts, and to 0.5% with bf16 products, where the reference rounds its partial products to bf16.
The core of one batched step, for a stack X of same-shaped momenta with the short side first:
def echelon_step(X, state, r, tau_iters=2, margin=1.05):
# X: (B, s, l) momenta, s <= l. state keeps U (B, s, r) and z (B, l, 1).
Xf, Xl = X.float(), X.bfloat16()
U = state["U"] # warm-started frame
V = cholqr(Xl.mT @ U.bfloat16()) # right half-step
U = cholqr(Xl @ V.bfloat16()) # left half-step
H = U.mT @ Xf # head, (B, r, l)
P = newton_schulz5(H) # polar factor of the small head
z = state["z"]
for _ in range(tau_iters): # top singular value of the tail,
z = z / z.norm(dim=1, keepdim=True) # without forming T = X - U H
w = Xf @ z - U @ (H @ z)
tau = w.norm(dim=1, keepdim=True)
z = Xf.mT @ w - H.mT @ (U.mT @ w)
state["U"], state["z"] = U, z
s_ = margin * tau
return Xf / s_ + U @ (P - H / s_) # = U P + (X - U H) / s_
cholqr is two passes of Cholesky-QR with a diagonal shift of 1e-6 times the mean diagonal; newton_schulz5 is Muon’s quintic. The first call starts U from a random frame refined by three subspace steps. The full code (including the fused kernel) and a package will be released on GitHub soon.
What we tried that didn’t work
Not every variant helped. These are the ones worth knowing about, with the numbers that ruled them out. The first two items are from the 4-layer character model (width 256, 1,000 steps, seed 0), where Muon ends at 1.548.
- Dropping the tail. Keeping only the head’s polar factor, at rank 16, ended at 1.788, against 1.629 for Echelon at the same rank and 1.629 for AdamW. The tail carries signal, as the alignment formula above says it should.
- Cheaper updates with no head at all. Dividing the momentum by its spectral norm, which is Echelon at rank 0, reached 1.729. Alternating row and column normalization, a SinkGD-style update, reached 1.846. Dion at rank 16 reached 1.715 (1.781 at twice the learning rate).
- A higher-order Echelon (HO-4). It keeps Echelon’s tracked head, gives it an exact limiter, and runs a polar iteration in Gram space on the deflated tail. On the character model it beat Muon on all six seeds, by 0.005 on average. On GPT-2-small it lost on all three seeds, by 0.0435 ± 0.0054, and its fp32 small-matrix iteration made the optimizer 2.3× slower than bf16 Muon (31.3 against 13.8 ms per step).
What is new here
Echelon is a new combination of mostly published parts, and several close relatives appeared within weeks of this work. Dion tracks a warm-started subspace but sends the tail to error feedback. He, Deng and Lu take the exact polar factor of a sketched head and discard the tail. Spectral clipping methods (Yukhimchuk et al., MuCon, and the concurrent Spectrally Targeted Muon) clip at a tuned threshold. The concurrent PMRC adds the residual back scaled by a fixed constant, where Echelon divides it by its own spectral norm. Peng, Wang and Yu prove, on paper, a convergence theorem close to our certificate theorem and the same 1.25 bound.
What we think is new: the rank-anchored, scale-free clip at with the guarantee under a stale frame, the polishing step, the spiked-spectrum separation, a batched GPU implementation that makes the saving real at small widths, the flatness observation (an open finding, not a solved problem), and machine-checked proofs of the bounds, the rate and the separation. As far as we can find, this is the first formal convergence proof for Muon or any steepest-descent method in the spectral norm.
Methods, full results and limitations are in the technical report (PDF), and the proofs are in the Lean repository on GitHub.
Why we’re doing this
We are optimizing the infra stack for LLM pre-training and post-training to democratize frontier AI R&D. Echelon is one part of that work: a cheaper optimizer step means a cheaper training step, and the rest of the stack gets the same treatment. We want a university group, a startup or an open-source project to be able to run the experiments it cares about on the budget it has.
Acknowledgements
This research was done on compute provided through AWS credits for startups and Daytona credits. We thank AWS and Daytona for compute credits.
@misc{jain2026echelon,
author = {Jain, Tushar},
title = {The Echelon Optimizer: Rank-Anchored Orthogonalization for Muon in O(mnr) with Machine-Checked Convergence},
year = {2026},
month = nov,
url = {https://evolvingmachines.ai/blog/echelon},
howpublished = {\url{https://evolvingmachines.ai/blog/echelon}},
note = {Evolving Machines blog}
}