The Echelon Optimizer

Muon’s update at the cost of a low-rank one: up to 10× faster optimizer steps, and proof checked in Lean.

Each pause on a foothold is one optimizer step: Muon’s takes 10.5× as long as Echelon’s (70B-class layer shapes, batched, rank 64).

What we found (tl;dr)

  • Muon flattens every singular value, at a cost that grows with the cube of the layer width. Echelon flattens only the top rr and scales the rest down, at O(mnr)O(mnr) cost.
  • On one GPU, the optimizer step for 70B-class layers drops from 33.9 s to 3.23 s. At 1.3B the optimizer is 2.5× faster and each training step 14% shorter, with validation loss within seed noise of Muon’s over 3 seeds each.
  • The gains grow on big MoE models. On DeepSeek V4.1 Flash, each training step takes about 35% less time on 16 B300s (about 31% on a more realistic 24), measured on B300 at rank 32, communication not counted.
  • We proved that Echelon’s update stays bounded and that it converges.

Performance at a glance

10.5×faster optimizer step at 70B-class layer shapes, one RTX PRO 6000 (batched, rank 64)
2.5×faster optimizer step inside a real 1.3B training run
−35%less time per training step on DeepSeek V4.1 Flash on 16 B300s (about 31% on 24), measured on B300, rank 32, 256k tokens per step, communication not counted
4.375validation loss at 1.3B, mean of 3 seeds, within seed noise of Muon (4.3751 ± 0.0115 vs 4.3757 ± 0.0109; AdamW 5.184, one seed)
1×5×10×1× = same speed as MuonGPT-2 smalld = 7682.0×1.3B-classd = 20483.8×8B-classd = 40965.9×70B-classd = 819210.5×
Figure 1. How many times faster the optimizer step is (batched implementation, rank 64, one RTX PRO 6000 Blackwell, against per-layer Muon).
0%50%100%DeepSeek V4.1 Flash552B total, 16B active16 B300−34.6%Reflection Beam501B total, 23B active43 RTX PRO−26.1%Mistral Small 4119B total, 6.5B active10 RTX PRO−28.5%Nemotron 3.5 Lightning30B total, 3B active3 RTX PRO−19.7%Qwen3.8-27Bdense3 RTX PRO−6.4%forward + backwardMuon optimizerEchelon optimizersaved
Figure 2. What it does to a training step on today’s models (estimated, 256k tokens per step, each model on the fewest GPUs that hold it: DeepSeek V4.1 Flash on 16 B300s, the others on RTX PRO 6000s).
0 ms400 ms800 ms1200 msOptimizer step2.5× fasterMuon 273 msEchelon 111 msFull training step1.17× faster, −14%Muon 1,162 msEchelon 997 ms
Figure 3. End-to-end, the gain is smaller (1.3B, mean of 3 seeds).

What it saves: training time and cost

A faster optimizer step only saves the share of the training step that the optimizer takes. If Muon’s optimizer is a fraction ff of each step and Echelon makes it ss times faster, the time saved per step is

saving=f(1−1s).\text{saving} = f\Big(1 - \frac{1}{s}\Big).

GPU time is what you pay for, so the same fraction comes off the compute bill. Here is what we measured, first in our two training runs, then on the real layer shapes of five current open models:

ModelShare fSpeedup sSaved
Trained end to end, one GPU, 32k tokens per step; speedup measured in training
GPT-2 small10.9%1.1×1.0%
1.3B dense23.5%2.5×14.2%
Timed on each model’s real layer shapes, rank 32, 256k tokens per step, each on the fewest GPUs that hold it
Qwen3.8-27B (dense), 3 RTX PRO6.8%17.6×6.4%
Nemotron 3.5 Lightning 30B-A3B, 3 RTX PRO21.6%11.5×19.7%
Mistral Small 4 119B-A6.5B, 10 RTX PRO31.5%10.4×28.5%
Reflection Beam 501B-A23B, 43 RTX PRO28.5%11.7×≈26.1%
DeepSeek V4.1 Flash 552B-A16B, 16 B30047.3%3.7×34.6%

Each lower row puts the model on the fewest GPUs that hold its training state (bf16 weights and gradients, fp32 momentum) and splits the optimizer’s work across them: DeepSeek V4.1 Flash on 16 B300s, timed on a B300 (about 31% on a more realistic 24), and the others on RTX PRO 6000s, timed there: Beam on 43, Mistral Small 4 on 10, Nemotron 3.5 Lightning and Qwen3.8-27B on 3 each. Communication is not counted. The speedup depends on the hardware: on DeepSeek V4.1 Flash’s layer shapes at rank 32 the optimizer step is 10.7× faster on the RTX PRO 6000 but 3.7× on a B300, because Muon’s matrix products run 3.4× faster there.

Optimizer speedup on the whole DeepSeek V4.1 Flash model, Echelon vs Muon

RankB300RTX PRO 6000
323.7×10.7×
643.4×7.3×
1282.8×3.0×
2562.0×2.3×
320 (width/16)1.7×2.0×
5121.2×1.4×

Best Echelon implementation per GPU (fused kernel or batched); whole-model optimizer step on one GPU.

And here is what the same formula gives for other optimizer shares and the speedups we measured at each width (batched implementation at rank 64 on one RTX PRO 6000, against the faster of per-layer and batched Muon). Wider layers raise both numbers, and mixture-of-experts models raise ff further, since the optimizer updates every expert while each token uses only a few (see below). These cells are projections, not measurements.

Share fs = 1.4× (d768)3.8× (d2048)5.9× (d4096)10.5× (d8192)
10%3%7%8%9%
25%7%18%21%23%
50%14%37%42%45%

To put it in money, plug in your own run

34.5%of time and cost saved
34,516GPU-hours saved
$103,549dollars saved

On DeepSeek V4.1 Flash, Echelon takes about 35% off each training step on 16 B300s (about 31% on a more realistic 24), measured on B300 at rank 32, communication not counted. A frontier lab can absorb that overhead. A university group, a startup or an open-source project often can’t: for them, 100,000 versus about 65,500 GPU-hours (on 16 B300s) can decide whether a run happens at all, or how many ablations fit in the budget. Echelon’s update is a few thin matrix products whose cost grows with the rank, not the cube of the width, so the saving is largest on big mixture-of-experts models like DeepSeek V4.1 Flash. Our 1.3B result was trained on a single GPU. The code (including the fused kernel) and a package will be released on GitHub soon, to put strong training within reach of more people.

Muon is one of the strongest optimizers for the hidden layers of transformers. Its update is easy to state: it takes the momentum matrix, keeps its singular vectors, and sets every singular value to one. The hard part is doing it fast. Muon approximates it with five Newton–Schulz iterations. For a square layer of width nn they cost 30n330n^3 floating-point operations per step, while everything else in the optimizer costs O(n2)O(n^2).

At frontier widths that cubic term is no longer small. On one Blackwell GPU, the Newton–Schulz step over every hidden matrix of a model with 70B-class layer shapes takes 33.9 seconds. Real training shards that work across GPUs, but it still grows with the cube of the width.

This post introduces the Echelon optimizer, a cheaper way to get most of what Muon does. For an m×nm \times n matrix and a rank rr you choose, it costs O(mnr)O(mnr). It has a clear meaning in terms of singular values, and it is provably stable even when its internal estimates are wrong. On the same GPU it cuts the 33.9-second step to 3.23 seconds. On DeepSeek V4.1 Flash it takes about 35% off each training step on 16 B300s (about 31% on a more realistic 24), measured on B300 at rank 32, communication not counted. In our 1.3B runs (3 seeds per optimizer) its validation loss matched Muon’s within seed noise, while each training step took 14% less time.

We also wrote the convergence argument in Lean, a proof assistant, and had the computer check every step. That is still rare for optimizer work.

The math, explained

What Muon actually computes

We write the momentum of one weight matrix as its singular value decomposition, M=UΣV⊤M = U\Sigma V^\top. Muon wants its polar factor, polar⁡(M)=UV⊤\operatorname{polar}(M) = UV^\top: the same directions, with every singular value replaced by one.

The polar factor is the linear-minimization oracle of the spectral-norm ball. We follow Bernstein and Newhouse’s anthology and Pethick et al.’s LMO view.

There is a good reason for that choice. Among all matrices with spectral norm at most one, the polar factor is the one that lines up best with MM:

polar⁡(M)=arg⁡max⁡∥O∥2≤1 ⟨M,O⟩⏟alignment,⟨M,polar⁡(M)⟩=∥M∥∗=∑iσi.\operatorname{polar}(M) = \arg\max_{\lVert O\rVert_2\le 1}\ \underbrace{\langle M, O\rangle}_{\text{alignment}},\qquad \langle M,\operatorname{polar}(M)\rangle = \lVert M\rVert_* = \textstyle\sum_i \sigma_i .

So it is steepest descent when we measure step size in the spectral norm, which is a natural way to measure how much a linear layer can change its outputs.

What each matrix does to the unit circle. Drag the two columns of M.

The same candidate, as stretches along M’s singular directions. Drag O.

M (momentum) candidate update O polar(M) updates that beat your O unit circle
M
polar(M)
your O
alignment ⟨M, O⟩
–
best possible, σ₁ + σ₂
–
∥O∥₂
–
what M/∥M∥F gets

Try

Figure 4. The best update inside the spectral-norm ball is the polar factor.

Computing UV⊤UV^\top exactly needs an SVD, which GPUs do badly. Muon instead normalizes MM by its Frobenius norm and applies an odd quintic polynomial five times, X←aX+b(XX⊤)X+c(XX⊤)2XX \leftarrow aX + b(XX^\top)X + c(XX^\top)^2X with (a,b,c)=(3.4445,−4.7750,2.0315)(a,b,c) = (3.4445, -4.7750, 2.0315). Only matrix multiplies, each of cost proportional to m2nm^2 n. The polynomial is tuned for speed rather than accuracy. It does not send singular values to exactly one; it shoves them all, large and small, into a band around [0.7,1.2][0.7, 1.2].

Across: a singular value before the step. Up: the same value after it. Drag the largest value of M in the row under the axis.

one step, p(x) no change, y = x band 0.68 to 1.20 a singular value

Try

the same values, step by step

inside the band now
–
smallest, largest
–

largest value: –

Figure 5. What five Newton–Schulz steps do to singular values.

Splitting the spectrum

Gradients and momenta of trained networks are far from full rank: a few directions carry most of the mass and the rest decay. That suggests treating the top of the spectrum and the rest differently. Echelon keeps one extra piece of state per matrix, an orthonormal frame Q∈Rm×rQ\in\mathbb{R}^{m\times r} that tracks the top-rr left singular directions of the momentum from one step to the next. Each step:

  1. Refresh. We refresh the frame with one power step on the current momentum: Q←orth⁡(MM⊤Q)Q \leftarrow \operatorname{orth}(M M^\top Q).O(mnr)
  2. Head. We project, H=Q⊤MH = Q^\top M, a small r×nr\times n matrix, and take its polar factor PP. It is small, so this is cheap: exact on CPU, five Newton–Schulz steps on the small matrix on GPU.O(mnr)
  3. Tail. The rest of the momentum is T=M−QHT = M - QH. We never store it, and estimate its largest singular value τ\tau with two power iterations, times a 1.05 margin.O(mn)
  4. Output. We return O=QP+T/τO = QP + T/\tau.O(mnr)
inMomentum X
s × l, short side first (s ≤ l)
fp32+ bf16 copy
stateWarm frame U, vector z
U: s × r, z: l × 1, kept from the last step
fp32
1Right half-step
V = orth(Xᵀ U), l × r
bf16 productfp32 Cholesky-QRO(slr)pass A
2Left half-step
U = orth(X V), s × r
bf16 productfp32 Cholesky-QRO(slr)pass B
head: the top r directions
3Project
H = Uᵀ X, a small r × l matrix
fp32O(slr)pass C
4Polar factor of the head
P = NS5(H), five Newton–Schulz steps on r × l only
fp32O(r²l)small op
tail: everything else, never formed
5Largest tail singular value
power iteration on T = X − U H:
w = Xz − U(Hz), z = Xᵀw − Hᵀ(Uᵀw)
τ = 1.05 · ∥w∥
fp32O(sl) per iterationpasses C–E
6Output
O = X/τ + U(P − H/τ) = U P + T/τ
fp32O(slr)pass F
Flops per step
Echelon 6slr + 6(s + l)r² + 20r³ + 10sl · Muon 5(4s²l + 2s³)
Memory traffic
first version: about 20 passes, 78 bytes per element · fused kernel: 6 passes (A–F), 20 bytes per element
Figure 6. One Echelon step.

In total that is 6mnr+6(m+n)r2+20r3+10mn6mnr + 6(m+n)r^2 + 20r^3 + 10mn flops against Muon’s 5(4m2n+2m3)5(4m^2n + 2m^3): linear in the width for a fixed rank, where Muon is cubic. When r=mr = m the tail is empty and the output is exactly Muon’s ideal, the polar factor.

Warm-started subspace tracking is the trick at the heart of Dion. What differs here is what happens to the head and the tail.

Here is the part we like. We assume the frame is exactly right, so QQ spans u1,…,uru_1,\dots,u_r, and τ=σr+1\tau = \sigma_{r+1}. Then the head contributes ∑i≤ruivi⊤\sum_{i\le r} u_iv_i^\top and the tail contributes ∑i>r(σi/σr+1) uivi⊤\sum_{i>r} (\sigma_i/\sigma_{r+1})\,u_iv_i^\top. Together they give:

O=∑imin⁡ ⁣(1, σiσr+1)⏟spectral clip uivi⊤.O = \sum_i \underbrace{\min\!\Big(1,\ \frac{\sigma_i}{\sigma_{r+1}}\Big)}_{\text{spectral clip}}\,u_i v_i^\top .

Echelon is a spectral clip. The top rr directions get equalized, as in Muon. The rest keep their shape, rescaled so the largest of them is exactly one. The threshold is not a tuned constant: it is set by the rank you choose and moves with the scale of the momentum. At r=1r = 1 it is close to normalized gradient descent; at full rank it is Muon.

Output singular value, direction by direction. Drag the blue line to set the rank r.

The map each method applies to a singular value.

input σᵢ/σ₁ (left alone) Muon, five Newton–Schulz steps Echelon, exact frame head, the top r directions

Try

scoreflatness Muon–– Echelon––

score = ⟨M, O⟩ / (∥M∥∗ ∥O∥₂); the exact polar factor scores 1. flatness = stable rank of O ÷ width.

Figure 7. Echelon is a spectral clip at a rank you choose.

The clip picture needs an exact frame. Ours is refreshed by a single power step per training step, so it is always a little stale. The second property is what makes that safe. For any orthonormal frame, however wrong, the head lives inside the span of QQ and the tail lives outside it, so they cannot add up in the same direction:

∥Ox∥2=∥QPx∥2⏟head+∥Tx∥2/τ2⏟tail≤2∥x∥2⟹∥O∥2≤2.\lVert Ox\rVert^2 = \underbrace{\lVert QPx\rVert^2}_{\text{head}} + \underbrace{\lVert Tx\rVert^2/\tau^2}_{\text{tail}} \le 2\lVert x\rVert^2 \quad\Longrightarrow\quad \lVert O\rVert_2 \le \sqrt{2}.

As long as the head’s factor has norm at most one and τ is at least the tail’s largest singular value, a stale frame never costs stability. What it costs is alignment per unit of norm: the raw alignment can even rise slightly at small angles, because a longer update collects more of it, but alignment divided by the spectral norm, the score, falls from the first degree. In our training runs these conditions usually don’t hold exactly: the estimate of the tail’s norm runs about 5–8% low and the head’s factor reaches norm 1.2, so the bound is a guide rather than a guarantee. In practice the update’s norm stays near 1.2 and exceeds √2 mostly in the first 50 steps.

Turn the frame. Drag the blue ray away from the true top-r subspace. Distance from the corner is the value at that angle.

Why it cannot pass √2. One dot per singular direction of O.

∥O∥₂, the update’s spectral norm alignment ⟨M, O⟩ / ∥M∥∗ √2 1
∥O∥₂ at this angle
–
ceiling √2
1.414
largest over all angles
–
alignment ⟨M, O⟩ / ∥M∥∗
–
with an exact frame
–
score = alignment ÷ norm
–
Figure 8. A wrong frame makes the update longer, never by more than a factor of √2.

To bring the norm back down to one, we apply a single cubic Newton–Schulz step, O←32O−12OO⊤OO \leftarrow \tfrac32 O - \tfrac12 OO^\top O. Its scalar map 32x−12x3\tfrac32x - \tfrac12x^3 sends all of [0,3][0,\sqrt3] into [0,1][0,1], so it fixes the norm even when τ\tau underestimates the tail by up to 3\sqrt3. We call this polishing.

Across: a singular value of the update before polishing. Up: the same value after. Drag the handle to make τ too small.

one polish step, 1.5x − 0.5x³ no polish, y = x a singular value of O, before and after largest possible with any frame [0, √3] lands in [0, 1]

Try

τ is
–
∥O∥₂beforeafter exact frame–– any frame, at most––

Figure 9. One cubic step pulls the norm back to one.

The alignment Echelon gives up has a short formula. Since ⟨M,QP⟩=∥H∥∗\langle M, QP\rangle = \lVert H\rVert_* and ⟨M,T⟩=∥T∥F2\langle M, T\rangle = \lVert T\rVert_F^2,

⟨M,O⟩  ≥  ∥M∥∗−(∥T∥∗−∥T∥F2τ)⏟alignment given up.\langle M, O\rangle \;\ge\; \lVert M\rVert_* - \underbrace{\Big(\lVert T\rVert_* - \frac{\lVert T\rVert_F^2}{\tau}\Big)}_{\text{alignment given up}}.

The loss is zero when the tail is flat and at most the tail’s nuclear norm otherwise. Throwing the tail away loses all of it, which is exactly why “head only” was the worst variant in every sweep we ran.

Where it fits best: wide layers and mixture-of-experts

An optimizer speedup only matters in proportion to how much of the training step the optimizer takes. Two things decide that share: how wide each matrix is, and how many tokens each matrix sees per step.

Why wider layers

For a square d×dd\times d layer trained on BB tokens per step, the forward and backward matmuls cost about 6Bd26Bd^2 flops, while Muon’s Newton–Schulz costs 30d330d^3. Their ratio is

orthogonalizationtraining compute≈30d36Bd2=5dB.\frac{\text{orthogonalization}}{\text{training compute}} \approx \frac{30d^3}{6Bd^2} = \frac{5d}{B}.

So Muon’s overhead grows linearly with width at a fixed batch. Echelon replaces 30d330d^3 with about 6d2r6d^2r, and the rank it needs grows much more slowly than the width: in our measurements on real momentum it was a quarter of the width at 256 and between a sixteenth and an eighth at 1024. Both effects point the same way, which is why Figure 1 climbs from 2.0× (1.4× against batched Muon) to 10.5× as the width goes from 768 to 8192.

Why mixture-of-experts

A mixture-of-experts (MoE) model replaces each feed-forward block with many expert feed-forward blocks and routes each token to only a few of them. That is what makes MoEs attractive: the model holds many parameters but each token only pays for the few experts it visits, so you get the capacity of a large model at the per-token compute of a small one.

The optimizer does not get that discount. Every expert’s weight matrices are updated every step, so the optimizer’s cost scales with the total parameter count, while the training compute scales with the active count. With EE experts and kk active per token, each expert sees only about kB/EkB/E tokens per step, and the ratio above becomes roughly

orthogonalizationtraining compute  ∝  Ek⋅min⁡(d,dff)B.\frac{\text{orthogonalization}}{\text{training compute}} \;\propto\; \frac{E}{k}\cdot\frac{\min(d, d_{\text{ff}})}{B}.

With 256 experts and 8 active, as in several current open models, E/k=32E/k = 32: the orthogonalization share is tens of times larger than in a dense model of the same width. Expert matrices in today’s MoEs are also wide in both directions, for example 5120×2560 in a model that holds 21× more total than active parameters. That is where a faster optimizer step turns into end-to-end time.

Three more properties line up with Echelon specifically:

  • Low-rank gradients. A layer’s gradient is a sum of one outer product per token it saw, so an expert that sees fewer tokens per micro-batch has lower-rank gradients, and a small head rank captures more of its momentum.
  • Many identical shapes. Hundreds of experts with the same shape are exactly what the batched implementation wants: one call per shape group, so launch latency disappears.
  • Bounded even when stale. Rarely routed experts update their tracked frame less usefully, and the 2\sqrt2 bound means a stale frame can cost score but never stability.

One training step, as a share of Muon’s step time.

forward + backward Muon’s optimizer step, f Echelon’s optimizer step, f/s saved, f(1 − 1/s)
saved = f (1 − 1/s)
–
ceiling if Echelon were free
–

Measured: share of the step saved, as tokens per step grow. Pick a run to highlight it and load its numbers above.

Trained end to end, one GPU, 32k tokens/step

Timed shapes, each on the fewest GPUs that hold it

Tokens per step

Figure 10. A faster optimizer saves only the optimizer’s share.

Measured on real model shapes

To check the argument on models people train today, we took four mixture-of-experts models and one dense model and timed both optimizers over every matrix Muon would update, on one NVIDIA RTX PRO 6000 Blackwell, and DeepSeek V4.1 Flash also on one B300. We also timed one real layer’s forward and backward pass in bf16 at several batch sizes, and combined the parts into an estimated step on the fewest GPUs that hold each model, with the optimizer’s work and the tokens split across them. Echelon runs at rank 32 (our fused kernel on the RTX PRO 6000; on the B300 the faster of the fused kernel and the batched implementation); at rank 64 the DeepSeek V4.1 Flash row saves 33.5% instead of 34.6%.

ModelGPUsMuon opt.Echelon opt.128k256k1M4M
DeepSeek-V4.1-Flash 552B-A16B16 B3002.13 s0.57 s40.1%34.6%≈15.5%≈4.6%
Reflection Beam 501B-A23B43 RTX PRO2.67 s0.23 s≈31.3%≈26.1%13.0%4.4%
Mistral Small 4 119B-A6.5B10 RTX PRO2.34 s0.23 s≈39.6%28.5%10.9%≈3.0%
Nemotron 3.5 Lightning 30B-A3B3 RTX PRO2.28 s0.20 s31.7%19.7%≈6.0%≈1.6%
Qwen3.8-27B (dense)3 RTX PRO3.03 s0.17 s11.9%6.4%≈1.7%≈0.4%

Each model on the fewest GPUs that hold its training state, timed on that GPU type. Optimizer columns: seconds per GPU per step, whole model split over those GPUs. The other columns: share of step time saved at that many tokens per optimizer step, summed over all GPUs. ≈: the forward and backward time is computed outside the measured tokens per GPU (extended below the smallest measured batch, or accumulated above the largest).

forward + backwardMuon optimizerEchelon optimizer (r=32)
DeepSeek V4.1 Flash552B total, 16B active · 16 B300
Muon
4.50 s
Echelon
2.94 s −34.6%
Reflection Beam501B total, 23B active · 43 RTX PRO
Muon
9.37 s
Echelon
6.93 s ≈−26.1%
Mistral Small 4119B total, 6.5B active · 10 RTX PRO
Muon
7.44 s
Echelon
5.32 s −28.5%
Nemotron 3.5 Lightning30B total, 3B active · 3 RTX PRO
Muon
10.55 s
Echelon
8.47 s −19.7%
Qwen3.8-27Bdense · 3 RTX PRO
Muon
44.50 s
Echelon
41.65 s −6.4%
Figure 11. Where the time goes, by batch size (estimated, seconds per GPU, each model on the fewest GPUs that hold it; each model drawn to its own scale; ≈ computed outside the measured tokens per GPU).
0%20%40%60%128k256k512k1M4Mtokens per optimizer step (all GPUs)DeepSeek-V4.1 40%Beam 501B 31%Mistral Small 4 40%Lightning 30B 32%Qwen3.8-27B 12%
Figure 12. The saving is largest for big MoE models at small batches, and falls as the batch grows. Each model on the fewest GPUs that hold it (DeepSeek V4.1 Flash on 16 B300s, the others on RTX PRO 6000s); points outside the measured tokens per GPU are computed.

Which runs benefit

  • Good fit: fine-tuning and RL policy updates of large mixture-of-experts models, at roughly 128k to 1M tokens per optimizer step. That is where the optimizer is a large share of the step: 31–40% saved at 128k, 13–15% at 1M for DeepSeek (16 B300s) and Beam (43 RTX PRO 6000s), each on the fewest GPUs that hold it, partly computed outside the measured tokens per GPU.
  • Small dense models at small batches: the 1.3B run saved 14% at 32k tokens per step.

How much of Muon it keeps

We trained a GPT-2-small-shaped model (width 768, 12 layers; 163M parameters with an untied output head) on FineWeb for 1,500 steps of 32k tokens.

OptimizerVal lossShare of Muon’s gain over AdamW
Muon (3 seeds)4.6975 ± 0.0024100%
Echelon, r=96 (3 seeds)4.7624 ± 0.002787%
Echelon, r=32 + one polish step (1 seed)4.80181%
AdamW (best of two learning rates, 1 seed)5.1990%

At rank 96, one eighth of the width, Echelon keeps 87% of Muon’s improvement over AdamW. On a much smaller 4-layer character model of width 256, averaged over three seeds, rank 64 keeps about two thirds (Muon 1.561 ± 0.012, Echelon 1.587 ± 0.007, AdamW 1.639 ± 0.009). The gap is smaller in our larger runs, though those runs also differ in depth, rank, length and number of seeds.

One reason to expect it to shrink is that the rank you need grows much more slowly than the width. We saved real momentum from Muon training runs of a small model at widths 256, 512 and 1024 and scored each method against an exact SVD. Echelon matches Muon’s score at rank 64 for widths 256 and 512, a quarter and an eighth of the width, and needs between 64 and 128 at width 1024, a sixteenth to an eighth. Over the same range the momentum’s effective rank grew only about 1.2× per doubling of width.

At 1.3B parameters the gap was within seed noise. We trained a GPT with width 2048 and 24 layers on FineWeb for 2,000 steps of 32k tokens (about 65M tokens, three seeds per optimizer and one for AdamW, one Blackwell GPU with gradient accumulation; mean ± s.d. over seeds):

OptimizerVal lossOptimizer ms/stepFull step msTokens/s
Muon4.3757 ± 0.01092731,16228,205
Echelon, r=1284.3751 ± 0.011511199732,860
AdamW (one learning rate, one seed)5.1842891535,796

Rank 128 is one sixteenth of the width. Over three seeds, Echelon’s final loss is 0.0006 ± 0.0017 below Muon’s (paired by seed), while the spread from seed to seed is about 0.011, so the two are level within seed noise. Averaged over the three seeds it reaches AdamW’s final loss at about step 670, after about 11 minutes of training against about 30 for an untuned AdamW (one seed): roughly 2.7× less wall-clock time and GPU cost (Figures 14 and 15 show seed 0).

1.3B (width 2048)4.55.05.56.06.5500100015002000AdamWMuonEchelon GPT-2-small (width 768)4.55.05.56.050010001500r=32+polishEchelon r=96Muon
Figure 13. Validation loss during training (seed 0)

Steps are not what you pay for. Plotted against wall-clock time, Echelon’s cheaper step moves its whole curve to the left: at 1.3B it finishes the same 2,000 steps 5.5 minutes sooner than Muon at the same loss. On one GPU, cost is wall-clock time multiplied by the hourly price, so the same picture is also a cost curve. The price field above the plot rescales the top axis.

DeepSeek V4.1 Flash estimated4.55.05.56.06.50$0.001$482$963 h$144GPU cost, 16 B300 GPUsestimated: 1.3B loss per step× this model’s step time;not this model’s lossMuonEchelon

Estimated, 2,000 steps on 16 B300 GPUs: Echelon 1.6 h · $79, Muon 2.5 h · $120.

measured
estimated
Figure 14. The same runs against wall-clock time (1.3B and GPT-2 small: seed 0)
To reach AdamW’s final lossEchelon11.3 min · $0.57Muon13.0 min · $0.65AdamW30.5 min · $1.53Full 2,000-step runEchelon33.3 min · $1.66Muon38.8 min · $1.94AdamW30.5 min · $1.53
Figure 15. Time and cost at 1.3B (seed 0).

In GPU-hours the 2,000 steps took 0.555 for Echelon and 0.646 for Muon (2,000 times the mean step time on one GPU, without evaluation).

Using Echelon

Where
The matrices Muon handles: attention and MLP or expert weights. Embeddings, the output head and norms stay on AdamW.
Settings
Muon’s carry over unchanged: learning rate 0.02, Nesterov momentum 0.95, each update scaled by √max(1, rows/cols), no weight decay. Same warmup and cosine schedule.
Rank
What worked: a quarter of the width at 256 (r=64), an eighth at 768 (r=96) and a sixteenth at 2048 (r=128). Our suggestion is to start at width/16, and not below 64.
Tail
Two power iterations and a 1.05 margin on τ; no polishing in the quality runs.
Memory
The frame Q and the power-iteration vector add m·r + n numbers per m×n matrix (m the short side): 1.3% more than the weights for a 2304×5120 expert at r=64, 2.5% at r=128.
Implementation
We stack same-shaped matrices and process each stack in one call, use bf16 for the large thin products and fp32 for everything small, and orthonormalize with Cholesky-QR plus a tiny diagonal shift, as two half-steps.
Sharding
A sharded version runs on the DTensor layouts that FSDP2, tensor and expert parallelism produce: rows, columns, 2-D blocks and expert stacks. On 4 processes it matches single-process Echelon to within 2.1×10−6 relative with fp32 products, over three steps and eleven layouts, and to 0.5% with bf16 products, where the reference rounds its partial products to bf16.

The core of one batched step, for a stack X of same-shaped momenta with the short side first:

def echelon_step(X, state, r, tau_iters=2, margin=1.05):
    # X: (B, s, l) momenta, s <= l. state keeps U (B, s, r) and z (B, l, 1).
    Xf, Xl = X.float(), X.bfloat16()
    U = state["U"]                                   # warm-started frame
    V = cholqr(Xl.mT @ U.bfloat16())                 # right half-step
    U = cholqr(Xl @ V.bfloat16())                    # left half-step
    H = U.mT @ Xf                                    # head, (B, r, l)
    P = newton_schulz5(H)                            # polar factor of the small head
    z = state["z"]
    for _ in range(tau_iters):                       # top singular value of the tail,
        z = z / z.norm(dim=1, keepdim=True)          # without forming T = X - U H
        w = Xf @ z - U @ (H @ z)
        tau = w.norm(dim=1, keepdim=True)
        z = Xf.mT @ w - H.mT @ (U.mT @ w)
    state["U"], state["z"] = U, z
    s_ = margin * tau
    return Xf / s_ + U @ (P - H / s_)                # = U P + (X - U H) / s_

cholqr is two passes of Cholesky-QR with a diagonal shift of 1e-6 times the mean diagonal; newton_schulz5 is Muon’s quintic. The first call starts U from a random frame refined by three subspace steps. The full code (including the fused kernel) and a package will be released on GitHub soon.

What we tried that didn’t work

Not every variant helped. These are the ones worth knowing about, with the numbers that ruled them out. The first two items are from the 4-layer character model (width 256, 1,000 steps, seed 0), where Muon ends at 1.548.

  • Dropping the tail. Keeping only the head’s polar factor, at rank 16, ended at 1.788, against 1.629 for Echelon at the same rank and 1.629 for AdamW. The tail carries signal, as the alignment formula above says it should.
  • Cheaper updates with no head at all. Dividing the momentum by its spectral norm, which is Echelon at rank 0, reached 1.729. Alternating row and column normalization, a SinkGD-style update, reached 1.846. Dion at rank 16 reached 1.715 (1.781 at twice the learning rate).
  • A higher-order Echelon (HO-4). It keeps Echelon’s tracked head, gives it an exact limiter, and runs a polar iteration in Gram space on the deflated tail. On the character model it beat Muon on all six seeds, by 0.005 on average. On GPT-2-small it lost on all three seeds, by 0.0435 ± 0.0054, and its fp32 small-matrix iteration made the optimizer 2.3× slower than bf16 Muon (31.3 against 13.8 ms per step).

What is new here

Echelon is a new combination of mostly published parts, and several close relatives appeared within weeks of this work. Dion tracks a warm-started subspace but sends the tail to error feedback. He, Deng and Lu take the exact polar factor of a sketched head and discard the tail. Spectral clipping methods (Yukhimchuk et al., MuCon, and the concurrent Spectrally Targeted Muon) clip at a tuned threshold. The concurrent PMRC adds the residual back scaled by a fixed constant, where Echelon divides it by its own spectral norm. Peng, Wang and Yu prove, on paper, a convergence theorem close to our certificate theorem and the same 1.25 bound.

What we think is new: the rank-anchored, scale-free clip at O(mnr)O(mnr) with the 2\sqrt2 guarantee under a stale frame, the polishing step, the spiked-spectrum separation, a batched GPU implementation that makes the saving real at small widths, the flatness observation (an open finding, not a solved problem), and machine-checked proofs of the bounds, the rate and the separation. As far as we can find, this is the first formal convergence proof for Muon or any steepest-descent method in the spectral norm.

Methods, full results and limitations are in the technical report (PDF), and the proofs are in the Lean repository on GitHub.

Why we’re doing this

We are optimizing the infra stack for LLM pre-training and post-training to democratize frontier AI R&D. Echelon is one part of that work: a cheaper optimizer step means a cheaper training step, and the rest of the stack gets the same treatment. We want a university group, a startup or an open-source project to be able to run the experiments it cares about on the budget it has.

Acknowledgements

This research was done on compute provided through AWS credits for startups and Daytona credits. We thank AWS and Daytona for compute credits.

Cite this post
@misc{jain2026echelon,
  author = {Jain, Tushar},
  title = {The Echelon Optimizer: Rank-Anchored Orthogonalization for Muon in O(mnr) with Machine-Checked Convergence},
  year = {2026},
  month = nov,
  url = {https://evolvingmachines.ai/blog/echelon},
  howpublished = {\url{https://evolvingmachines.ai/blog/echelon}},
  note = {Evolving Machines blog}
}