sm-sae is a benchmark fixture for evaluating techniques
that extract simpler, structured representations from sparse autoencoders.
The Standard Model gives us exact ground truth at multiple granularities
(per-charge, per-color, per-generation, per-particle, per-origin), so any
candidate technique — polygram is the worked example below, but other
methods could be evaluated against the same fixture — can be scored
quantitatively rather than judged on vibes. This walkthrough reports the
current scores candidly: some axes work, some don't, and the gaps are the
point.
(a) How the physics is encoded
Background — a 60-second primer on world models (neural networks)
Every neural network in this report shares the same big-picture shape: it
is a
world
model. There are three pieces:
State — whatever describes the world right now.
For a
large language model (LLM) the
state is the sequence of tokens generated so far. For a
diffusion model it's a noisy image. For the
Standard Model cascade we'll meet just below, it's a multiset of
particles.
Transition function — the actual neural network.
It takes the current state and produces the next state (possibly
stochastically). Its behaviour is determined by a set of numbers called
parameters (or weights). An LLM has on the order of
1010 of them; the Standard Model has 19. They stop changing
once the model is "ready to use".
Loop — apply the transition function repeatedly,
feeding the new state back in as input. A 100-word LLM completion is
100 transitions. A diffusion sample is dozens of denoising steps. An
SM cascade is however many decay steps until everything is stable.
While one transition step runs, the transition function computes
intermediate values internally; these are called
activations. Same input state → same activations; different
input state → different activations.
Sparse autoencoders (SAEs) — see also the
Anthropic "Towards Monosemanticity"
report for the interpretability use — are a separate, smaller
network trained to take those activations and decompose them into a larger
but sparser set of human-interpretable features.
Aside — other neural-network topologies.
Not every neural network is a world model: image classifiers,
sentence-embedding encoders, and encoder-only models like
BERT run end-to-end once with no loop. Inside the
transition function (or inside a non-looping network), the math is the
classic layer-stack: matrix multiply, add bias, apply a nonlinearity, repeat
— the
multilayer perceptron picture. Modern systems use richer
layer types (transformer blocks for LLMs,
U-Nets for diffusion). What varies is the layer type and whether there's
an outer loop; everything in this report has the loop.
Background — a 60-second primer on the Standard Model
The Standard Model (SM) of particle physics is the
agreed-upon catalogue of fundamental particles and the forces between them. Three things
to know to read the diagrams below:
Forces are described by
gauge theory. Each force corresponds to a
symmetry group: electromagnetism is U(1), the
weak force is
SU(2), and the strong force is
SU(3). The mathematical fingerprint
of each group is its set of
structure constants.
The Standard Model has just 19 free numerical parameters
(plus 7 more once neutrino masses are included). Every other number in the
tables below is computed from those parameters and the choice of
symmetry groups; the structure is fixed, not learned.
How the Standard Model fits the world-model picture
The Standard Model is a world model in exactly the sense above:
State: a multiset of particles, e.g. {H: 1}
(one Higgs boson).
Transition function: enumerate the allowed decays of
particles in the current state (from the vertex catalogue), weight each
by its physical rate (computed from
Yukawa couplings,
CKM matrix, gauge couplings,
masses), sample one, mutate the state.
Loop: repeat until no remaining particle has a
kinematically allowed decay.
What's unusual about this world model: its 19 parameters were
measured by experimental particle physics over the 20th
century, not learned by gradient descent. But the role is identical to a
trained LLM's weights — they are the knobs that define the transition
function, frozen at "ready to use" time.
So why bother packaging the SM as an nn.Module?
Same tooling that inspects an LLM (state-dict navigation, parameter
counts, activation hooks,
safetensors loaders) now works on physics.
The submodules below (gauge → reps →
fields → vertices → params) are
an organizational hierarchy by mathematical dependency, not a
stack of layers. The "layer stack" of the SM world model lives in the
cascade engine and shows up later in the "Running the model" section.
The SM as a layered nn.Module
The Standard Model is packaged here as a nn.Module with five
layered submodules. Every "weight" is either a frozen buffer
(the architecture: Lie-algebra
structure constants,
irreducible representations,
particle vectors, vertex incidence) or one of the
26 trainable scalars — the only learnable
knobs the SM has. That ratio (≈26 learnable vs 13,594 frozen) is
the entire point: physics is mostly architecture, almost nothing is fit.
kind
name
shape
dtype
contents — meaning & values
gauge — architecture: Lie algebra structure constants and fundamental generators
buffer
gauge.su2_d
3 × 3 × 3
float64
SU(2) symmetric structure constants dabc. Identically zero for SU(2); the buffer is kept for structural parallelism with SU(3).
inspect values (27 entries)
[0, …] (3 × 3)
0
1
2
0
1
2
[1, …] (3 × 3)
0
1
2
0
1
2
[2, …] (3 × 3)
0
1
2
0
1
2
buffer
gauge.su2_f
3 × 3 × 3
float64
SU(2) antisymmetric structure constants fabc. They define the Lie bracket [Ta, Tb] = i fabc Tc; for SU(2) this is just εabc.
inspect values (27 entries)
[0, …] (3 × 3)
0
1
2
0
1
1
2
-1
[1, …] (3 × 3)
0
1
2
0
-1
1
2
1
[2, …] (3 × 3)
0
1
2
0
1
1
-1
2
buffer
gauge.su2_generators
3 × 2 × 2
complex128
The three Pauli matrices σa/2 — generators of SU(2) in its fundamental representation.
inspect values (12 entries)
real part
[0, …] (2 × 2)
0
1
0
0.5
1
0.5
[1, …] (2 × 2)
0
1
0
1
[2, …] (2 × 2)
0
1
0
0.5
1
-0.5
imaginary part (max |Im| = 0.5)
[0, …] (2 × 2)
0
1
0
1
[1, …] (2 × 2)
0
1
0
-0.5
1
0.5
[2, …] (2 × 2)
0
1
0
1
buffer
gauge.su3_d
8 × 8 × 8
float64
SU(3) symmetric structure constants dabc. Non-trivial (unlike SU(2)); these enter anomaly calculations and the chiral Lagrangian.
inspect values (512 entries)
[0, …] (8 × 8)
0
1
2
3
4
5
6
7
0
0.5774
1
2
3
0.5
4
0.5
5
0.5
6
0.5
7
0.5774
[1, …] (8 × 8)
0
1
2
3
4
5
6
7
0
1
0.5774
2
3
-0.5
4
0.5
5
0.5
6
-0.5
7
0.5774
[2, …] (8 × 8)
0
1
2
3
4
5
6
7
0
1
2
0.5774
3
0.5
4
0.5
5
-0.5
6
-0.5
7
0.5774
[3, …] (8 × 8)
0
1
2
3
4
5
6
7
0
0.5
1
-0.5
2
0.5
3
0.5
-0.2887
4
5
0.5
6
-0.5
7
-0.2887
[4, …] (8 × 8)
0
1
2
3
4
5
6
7
0
0.5
1
0.5
2
0.5
3
4
0.5
-0.2887
5
0.5
6
0.5
7
-0.2887
[5, …] (8 × 8)
0
1
2
3
4
5
6
7
0
0.5
1
0.5
2
-0.5
3
0.5
4
0.5
5
-0.5
-0.2887
6
7
-0.2887
[6, …] (8 × 8)
0
1
2
3
4
5
6
7
0
0.5
1
-0.5
2
-0.5
3
-0.5
4
0.5
5
6
-0.5
-0.2887
7
-0.2887
[7, …] (8 × 8)
0
1
2
3
4
5
6
7
0
0.5774
1
0.5774
2
0.5774
3
-0.2887
4
-0.2887
5
-0.2887
6
-0.2887
7
-0.5774
buffer
gauge.su3_f
8 × 8 × 8
float64
SU(3) antisymmetric structure constants — the canonical fingerprint of QCD's color algebra. Most entries are 0; the non-zero pattern is what makes gluon self-interaction possible.
inspect values (512 entries)
[0, …] (8 × 8)
0
1
2
3
4
5
6
7
0
1
1
2
-1
3
0.5
4
-0.5
5
0.5
6
-0.5
7
[1, …] (8 × 8)
0
1
2
3
4
5
6
7
0
-1
1
2
1
3
0.5
4
0.5
5
-0.5
6
-0.5
7
[2, …] (8 × 8)
0
1
2
3
4
5
6
7
0
1
1
-1
2
3
0.5
4
-0.5
5
-0.5
6
0.5
7
[3, …] (8 × 8)
0
1
2
3
4
5
6
7
0
-0.5
1
-0.5
2
-0.5
3
4
0.5
0.866
5
0.5
6
0.5
7
-0.866
[4, …] (8 × 8)
0
1
2
3
4
5
6
7
0
0.5
1
-0.5
2
0.5
3
-0.5
-0.866
4
5
-0.5
6
0.5
7
0.866
[5, …] (8 × 8)
0
1
2
3
4
5
6
7
0
-0.5
1
0.5
2
0.5
3
-0.5
4
0.5
5
6
-0.5
0.866
7
-0.866
[6, …] (8 × 8)
0
1
2
3
4
5
6
7
0
0.5
1
0.5
2
-0.5
3
-0.5
4
-0.5
5
0.5
-0.866
6
7
0.866
[7, …] (8 × 8)
0
1
2
3
4
5
6
7
0
1
2
3
0.866
4
-0.866
5
0.866
6
-0.866
7
buffer
gauge.su3_generators
8 × 3 × 3
complex128
The eight Gell-Mann matrices λa/2 — generators of SU(3) in its fundamental (3) representation.
inspect values (72 entries)
real part
[0, …] (3 × 3)
0
1
2
0
0.5
1
0.5
2
[1, …] (3 × 3)
0
1
2
0
1
2
[2, …] (3 × 3)
0
1
2
0
0.5
1
-0.5
2
[3, …] (3 × 3)
0
1
2
0
0.5
1
2
0.5
[4, …] (3 × 3)
0
1
2
0
1
2
[5, …] (3 × 3)
0
1
2
0
1
0.5
2
0.5
[6, …] (3 × 3)
0
1
2
0
1
2
[7, …] (3 × 3)
0
1
2
0
0.2887
1
0.2887
2
-0.5774
imaginary part (max |Im| = 0.5)
[0, …] (3 × 3)
0
1
2
0
1
2
[1, …] (3 × 3)
0
1
2
0
-0.5
1
0.5
2
[2, …] (3 × 3)
0
1
2
0
1
2
[3, …] (3 × 3)
0
1
2
0
1
2
[4, …] (3 × 3)
0
1
2
0
-0.5
1
2
0.5
[5, …] (3 × 3)
0
1
2
0
1
2
[6, …] (3 × 3)
0
1
2
0
1
-0.5
2
0.5
[7, …] (3 × 3)
0
1
2
0
1
2
reps — architecture: higher irreducible representations (computed from gauge)
buffer
reps.su2_j0
3 × 1 × 1
complex128
SU(2) representation, spin j = 0 (trivial). Three matrices (one per SU(2) generator) acting on the 1-dim rep space.
inspect values (3 entries)
real part
[0, …] (1 × 1)
0
0
[1, …] (1 × 1)
0
0
[2, …] (1 × 1)
0
0
imaginary part (max |Im| = 0)
[0, …] (1 × 1)
0
0
[1, …] (1 × 1)
0
0
[2, …] (1 × 1)
0
0
buffer
reps.su2_j0_5
3 × 2 × 2
complex128
SU(2) representation, spin j = 1/2 (fundamental). Three matrices (one per SU(2) generator) acting on the 2-dim rep space.
inspect values (12 entries)
real part
[0, …] (2 × 2)
0
1
0
0.5
1
0.5
[1, …] (2 × 2)
0
1
0
1
[2, …] (2 × 2)
0
1
0
0.5
1
-0.5
imaginary part (max |Im| = 0.5)
[0, …] (2 × 2)
0
1
0
1
[1, …] (2 × 2)
0
1
0
-0.5
1
0.5
[2, …] (2 × 2)
0
1
0
1
buffer
reps.su2_j1
3 × 3 × 3
complex128
SU(2) representation, spin j = 1 (vector). Three matrices (one per SU(2) generator) acting on the 3-dim rep space.
inspect values (27 entries)
real part
[0, …] (3 × 3)
0
1
2
0
0.7071
1
0.7071
0.7071
2
0.7071
[1, …] (3 × 3)
0
1
2
0
1
2
[2, …] (3 × 3)
0
1
2
0
1
1
2
-1
imaginary part (max |Im| = 0.707)
[0, …] (3 × 3)
0
1
2
0
1
2
[1, …] (3 × 3)
0
1
2
0
-0.7071
1
0.7071
-0.7071
2
0.7071
[2, …] (3 × 3)
0
1
2
0
1
2
buffer
reps.su2_j1_5
3 × 4 × 4
complex128
SU(2) representation, spin j = 3/2. Three matrices (one per SU(2) generator) acting on the 4-dim rep space.
inspect values (48 entries)
real part
[0, …] (4 × 4)
0
1
2
3
0
0.866
1
0.866
1
2
1
0.866
3
0.866
[1, …] (4 × 4)
0
1
2
3
0
1
2
3
[2, …] (4 × 4)
0
1
2
3
0
1.5
1
0.5
2
-0.5
3
-1.5
imaginary part (max |Im| = 1)
[0, …] (4 × 4)
0
1
2
3
0
1
2
3
[1, …] (4 × 4)
0
1
2
3
0
-0.866
1
0.866
-1
2
1
-0.866
3
0.866
[2, …] (4 × 4)
0
1
2
3
0
1
2
3
buffer
reps.su2_j2
3 × 5 × 5
complex128
SU(2) representation, spin j = 2. Three matrices (one per SU(2) generator) acting on the 5-dim rep space.
inspect values (75 entries)
real part
[0, …] (5 × 5)
0
1
2
3
4
0
1
1
1
1.225
2
1.225
1.225
3
1.225
1
4
1
[1, …] (5 × 5)
0
1
2
3
4
0
1
2
3
4
[2, …] (5 × 5)
0
1
2
3
4
0
2
1
1
2
3
-1
4
-2
imaginary part (max |Im| = 1.22)
[0, …] (5 × 5)
0
1
2
3
4
0
1
2
3
4
[1, …] (5 × 5)
0
1
2
3
4
0
-1
1
1
-1.225
2
1.225
-1.225
3
1.225
-1
4
1
[2, …] (5 × 5)
0
1
2
3
4
0
1
2
3
4
buffer
reps.su3_1
8 × 1 × 1
complex128
SU(3) irreducible representation 1. Eight matrices (one per SU(3) generator) acting on the 1-dim rep space.
inspect values (8 entries)
real part
[0, …] (1 × 1)
0
0
[1, …] (1 × 1)
0
0
[2, …] (1 × 1)
0
0
[3, …] (1 × 1)
0
0
[4, …] (1 × 1)
0
0
[5, …] (1 × 1)
0
0
[6, …] (1 × 1)
0
0
[7, …] (1 × 1)
0
0
imaginary part (max |Im| = 0)
[0, …] (1 × 1)
0
0
[1, …] (1 × 1)
0
0
[2, …] (1 × 1)
0
0
[3, …] (1 × 1)
0
0
[4, …] (1 × 1)
0
0
[5, …] (1 × 1)
0
0
[6, …] (1 × 1)
0
0
[7, …] (1 × 1)
0
0
buffer
reps.su3_10
8 × 10 × 10
complex128
SU(3) irreducible representation 10. Eight matrices (one per SU(3) generator) acting on the 10-dim rep space.
inspect values (800 entries)
real part
[0, …] (10 × 10)
0
1
2
3
4
5
6
7
8
9
0
0.866
1
0.866
1
2
0.7071
3
1
0.866
4
0.7071
0.7071
5
0.5
6
0.866
7
0.7071
8
0.5
9
[1, …] (10 × 10)
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
[2, …] (10 × 10)
0
1
2
3
4
5
6
7
8
9
0
1.5
1
0.5
2
1
3
-0.5
4
5
0.5
6
-1.5
7
-1
8
-0.5
9
[3, …] (10 × 10)
0
1
2
3
4
5
6
7
8
9
0
0.866
1
0.7071
2
0.866
1
3
0.5
4
0.7071
0.7071
5
1
0.866
6
7
0.5
8
0.7071
9
0.866
[4, …] (10 × 10)
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
[5, …] (10 × 10)
0
1
2
3
4
5
6
7
8
9
0
1
0.5
2
0.5
3
0.7071
4
0.7071
0.7071
5
0.7071
6
0.866
7
0.866
1
8
1
0.866
9
0.866
[6, …] (10 × 10)
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
[7, …] (10 × 10)
0
1
2
3
4
5
6
7
8
9
0
0.866
1
0.866
2
3
0.866
4
5
-0.866
6
0.866
7
8
-0.866
9
-1.732
imaginary part (max |Im| = 1)
[0, …] (10 × 10)
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
[1, …] (10 × 10)
0
1
2
3
4
5
6
7
8
9
0
-0.866
1
0.866
-1
2
-0.7071
3
1
-0.866
4
0.7071
-0.7071
5
-0.5
6
0.866
7
0.7071
8
0.5
9
[2, …] (10 × 10)
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
[3, …] (10 × 10)
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
[4, …] (10 × 10)
0
1
2
3
4
5
6
7
8
9
0
-0.866
1
-0.7071
2
0.866
-1
3
-0.5
4
0.7071
-0.7071
5
1
-0.866
6
7
0.5
8
0.7071
9
0.866
[5, …] (10 × 10)
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
[6, …] (10 × 10)
0
1
2
3
4
5
6
7
8
9
0
1
-0.5
2
0.5
3
-0.7071
4
0.7071
-0.7071
5
0.7071
6
-0.866
7
0.866
-1
8
1
-0.866
9
0.866
[7, …] (10 × 10)
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
buffer
reps.su3_3
8 × 3 × 3
complex128
SU(3) irreducible representation 3. Eight matrices (one per SU(3) generator) acting on the 3-dim rep space.
inspect values (72 entries)
real part
[0, …] (3 × 3)
0
1
2
0
0.5
1
0.5
2
[1, …] (3 × 3)
0
1
2
0
1
2
[2, …] (3 × 3)
0
1
2
0
0.5
1
-0.5
2
[3, …] (3 × 3)
0
1
2
0
0.5
1
2
0.5
[4, …] (3 × 3)
0
1
2
0
1
2
[5, …] (3 × 3)
0
1
2
0
1
0.5
2
0.5
[6, …] (3 × 3)
0
1
2
0
1
2
[7, …] (3 × 3)
0
1
2
0
0.2887
1
0.2887
2
-0.5774
imaginary part (max |Im| = 0.5)
[0, …] (3 × 3)
0
1
2
0
1
2
[1, …] (3 × 3)
0
1
2
0
-0.5
1
0.5
2
[2, …] (3 × 3)
0
1
2
0
1
2
[3, …] (3 × 3)
0
1
2
0
1
2
[4, …] (3 × 3)
0
1
2
0
-0.5
1
2
0.5
[5, …] (3 × 3)
0
1
2
0
1
2
[6, …] (3 × 3)
0
1
2
0
1
-0.5
2
0.5
[7, …] (3 × 3)
0
1
2
0
1
2
buffer
reps.su3_3bar
8 × 3 × 3
complex128
SU(3) irreducible representation 3bar. Eight matrices (one per SU(3) generator) acting on the 3-dim rep space.
inspect values (72 entries)
real part
[0, …] (3 × 3)
0
1
2
0
-0.5
1
-0.5
2
[1, …] (3 × 3)
0
1
2
0
1
2
[2, …] (3 × 3)
0
1
2
0
-0.5
1
0.5
2
[3, …] (3 × 3)
0
1
2
0
-0.5
1
2
-0.5
[4, …] (3 × 3)
0
1
2
0
1
2
[5, …] (3 × 3)
0
1
2
0
1
-0.5
2
-0.5
[6, …] (3 × 3)
0
1
2
0
1
2
[7, …] (3 × 3)
0
1
2
0
-0.2887
1
-0.2887
2
0.5774
imaginary part (max |Im| = 0.5)
[0, …] (3 × 3)
0
1
2
0
1
2
[1, …] (3 × 3)
0
1
2
0
-0.5
1
0.5
2
[2, …] (3 × 3)
0
1
2
0
1
2
[3, …] (3 × 3)
0
1
2
0
1
2
[4, …] (3 × 3)
0
1
2
0
-0.5
1
2
0.5
[5, …] (3 × 3)
0
1
2
0
1
2
[6, …] (3 × 3)
0
1
2
0
1
-0.5
2
0.5
[7, …] (3 × 3)
0
1
2
0
1
2
buffer
reps.su3_6
8 × 6 × 6
complex128
SU(3) irreducible representation 6. Eight matrices (one per SU(3) generator) acting on the 6-dim rep space.
inspect values (288 entries)
real part
[0, …] (6 × 6)
0
1
2
3
4
5
0
0.7071
1
0.7071
0.7071
2
0.5
3
0.7071
4
0.5
5
[1, …] (6 × 6)
0
1
2
3
4
5
0
1
2
3
4
5
[2, …] (6 × 6)
0
1
2
3
4
5
0
1
1
2
0.5
3
-1
4
-0.5
5
[3, …] (6 × 6)
0
1
2
3
4
5
0
0.7071
1
0.5
2
0.7071
0.7071
3
4
0.5
5
0.7071
[4, …] (6 × 6)
0
1
2
3
4
5
0
1
2
3
4
5
[5, …] (6 × 6)
0
1
2
3
4
5
0
1
0.5
2
0.5
3
0.7071
4
0.7071
0.7071
5
0.7071
[6, …] (6 × 6)
0
1
2
3
4
5
0
1
2
3
4
5
[7, …] (6 × 6)
0
1
2
3
4
5
0
0.5774
1
0.5774
2
-0.2887
3
0.5774
4
-0.2887
5
-1.155
imaginary part (max |Im| = 0.707)
[0, …] (6 × 6)
0
1
2
3
4
5
0
1
2
3
4
5
[1, …] (6 × 6)
0
1
2
3
4
5
0
-0.7071
1
0.7071
-0.7071
2
-0.5
3
0.7071
4
0.5
5
[2, …] (6 × 6)
0
1
2
3
4
5
0
1
2
3
4
5
[3, …] (6 × 6)
0
1
2
3
4
5
0
1
2
3
4
5
[4, …] (6 × 6)
0
1
2
3
4
5
0
-0.7071
1
-0.5
2
0.7071
-0.7071
3
4
0.5
5
0.7071
[5, …] (6 × 6)
0
1
2
3
4
5
0
1
2
3
4
5
[6, …] (6 × 6)
0
1
2
3
4
5
0
1
-0.5
2
0.5
3
-0.7071
4
0.7071
-0.7071
5
0.7071
[7, …] (6 × 6)
0
1
2
3
4
5
0
1
2
3
4
5
buffer
reps.su3_8
8 × 8 × 8
complex128
SU(3) irreducible representation 8. Eight matrices (one per SU(3) generator) acting on the 8-dim rep space.
inspect values (512 entries)
real part
[0, …] (8 × 8)
0
1
2
3
4
5
6
7
0
1
2
3
4
5
6
7
[1, …] (8 × 8)
0
1
2
3
4
5
6
7
0
1
2
3
4
5
6
7
[2, …] (8 × 8)
0
1
2
3
4
5
6
7
0
1
2
3
4
5
6
7
[3, …] (8 × 8)
0
1
2
3
4
5
6
7
0
1
2
3
4
5
6
7
[4, …] (8 × 8)
0
1
2
3
4
5
6
7
0
1
2
3
4
5
6
7
[5, …] (8 × 8)
0
1
2
3
4
5
6
7
0
1
2
3
4
5
6
7
[6, …] (8 × 8)
0
1
2
3
4
5
6
7
0
1
2
3
4
5
6
7
[7, …] (8 × 8)
0
1
2
3
4
5
6
7
0
1
2
3
4
5
6
7
imaginary part (max |Im| = 1)
[0, …] (8 × 8)
0
1
2
3
4
5
6
7
0
1
-1
2
1
3
-0.5
4
0.5
5
-0.5
6
0.5
7
[1, …] (8 × 8)
0
1
2
3
4
5
6
7
0
1
1
2
-1
3
-0.5
4
-0.5
5
0.5
6
0.5
7
[2, …] (8 × 8)
0
1
2
3
4
5
6
7
0
-1
1
1
2
3
-0.5
4
0.5
5
0.5
6
-0.5
7
[3, …] (8 × 8)
0
1
2
3
4
5
6
7
0
0.5
1
0.5
2
0.5
3
4
-0.5
-0.866
5
-0.5
6
-0.5
7
0.866
[4, …] (8 × 8)
0
1
2
3
4
5
6
7
0
-0.5
1
0.5
2
-0.5
3
0.5
0.866
4
5
0.5
6
-0.5
7
-0.866
[5, …] (8 × 8)
0
1
2
3
4
5
6
7
0
0.5
1
-0.5
2
-0.5
3
0.5
4
-0.5
5
6
0.5
-0.866
7
0.866
[6, …] (8 × 8)
0
1
2
3
4
5
6
7
0
-0.5
1
-0.5
2
0.5
3
0.5
4
0.5
5
-0.5
0.866
6
7
-0.866
[7, …] (8 × 8)
0
1
2
3
4
5
6
7
0
1
2
3
-0.866
4
0.866
5
-0.866
6
0.866
7
fields — architecture: particle content and quantum-number table
buffer
fields.field_table
9 × 6
float64
Per-field quantum-number table: color multiplicity, weak isospin multiplicity, hypercharge Y, generation index, chirality, and Lorentz type (scalar / spinor / vector).
inspect values (54 entries)
n_color
n_weak
Y
n_gen
chirality_enc
lorentz_enc
Q_L
3
2
0.3333
3
1
1
u_R
3
1
1.333
3
-1
1
d_R
3
1
-0.6667
3
-1
1
L_L
1
2
-1
3
1
1
e_R
1
1
-2
3
-1
1
H
1
2
1
1
G
8
1
1
2
W
1
3
1
2
B
1
1
1
2
buffer
fields.particle_vectors
61 × 9
float64
All 61 Standard-Model particles as 9-vectors. Columns: electric charge Q, baryon number B, lepton numbers Le/Lμ/Lτ, the two diagonal color generators C3/C8, spin, and mass in GeV.
inspect values (549 entries)
Q
B
L_e
L_mu
L_tau
C3
C8
spin
mass_GeV
H
125.1
W+
1
1
80.38
W-
-1
1
80.38
Z
1
91.19
b_b
-0.3333
0.3333
-0.5774
0.5
4.18
b_g
-0.3333
0.3333
-0.5
0.2887
0.5
4.18
b_r
-0.3333
0.3333
0.5
0.2887
0.5
4.18
c_b
0.6667
0.3333
-0.5774
0.5
1.27
c_g
0.6667
0.3333
-0.5
0.2887
0.5
1.27
c_r
0.6667
0.3333
0.5
0.2887
0.5
1.27
d_b
-0.3333
0.3333
-0.5774
0.5
0.0047
d_g
-0.3333
0.3333
-0.5
0.2887
0.5
0.0047
d_r
-0.3333
0.3333
0.5
0.2887
0.5
0.0047
e+
1
-1
0.5
0.000511
e-
-1
1
0.5
0.000511
g_bgbar
0.5
-0.866
1
g_brbar
-0.5
-0.866
1
g_diag3
1
g_diag8
1
g_gbbar
-0.5
0.866
1
g_grbar
-1
1
g_rbbar
0.5
0.866
1
g_rgbar
1
1
mu+
1
-1
0.5
0.1057
mu-
-1
1
0.5
0.1057
nu_e
1
0.5
nu_mu
1
0.5
nu_tau
1
0.5
photon
1
s_b
-0.3333
0.3333
-0.5774
0.5
0.095
s_g
-0.3333
0.3333
-0.5
0.2887
0.5
0.095
s_r
-0.3333
0.3333
0.5
0.2887
0.5
0.095
t_b
0.6667
0.3333
-0.5774
0.5
172.8
t_g
0.6667
0.3333
-0.5
0.2887
0.5
172.8
t_r
0.6667
0.3333
0.5
0.2887
0.5
172.8
tau+
1
-1
0.5
1.777
tau-
-1
1
0.5
1.777
u_b
0.6667
0.3333
-0.5774
0.5
0.0022
u_g
0.6667
0.3333
-0.5
0.2887
0.5
0.0022
u_r
0.6667
0.3333
0.5
0.2887
0.5
0.0022
~b_b
0.3333
-0.3333
0.5774
0.5
4.18
~b_g
0.3333
-0.3333
0.5
-0.2887
0.5
4.18
~b_r
0.3333
-0.3333
-0.5
-0.2887
0.5
4.18
~c_b
-0.6667
-0.3333
0.5774
0.5
1.27
~c_g
-0.6667
-0.3333
0.5
-0.2887
0.5
1.27
~c_r
-0.6667
-0.3333
-0.5
-0.2887
0.5
1.27
~d_b
0.3333
-0.3333
0.5774
0.5
0.0047
~d_g
0.3333
-0.3333
0.5
-0.2887
0.5
0.0047
~d_r
0.3333
-0.3333
-0.5
-0.2887
0.5
0.0047
~nu_e
-1
0.5
~nu_mu
-1
0.5
~nu_tau
-1
0.5
~s_b
0.3333
-0.3333
0.5774
0.5
0.095
~s_g
0.3333
-0.3333
0.5
-0.2887
0.5
0.095
~s_r
0.3333
-0.3333
-0.5
-0.2887
0.5
0.095
~t_b
-0.6667
-0.3333
0.5774
0.5
172.8
~t_g
-0.6667
-0.3333
0.5
-0.2887
0.5
172.8
~t_r
-0.6667
-0.3333
-0.5
-0.2887
0.5
172.8
~u_b
-0.6667
-0.3333
0.5774
0.5
0.0022
~u_g
-0.6667
-0.3333
0.5
-0.2887
0.5
0.0022
~u_r
-0.6667
-0.3333
-0.5
-0.2887
0.5
0.0022
vertices — architecture: signed vertex-particle incidence matrix
buffer
vertices.incidence
168 × 59
float64
Signed vertex–particle incidence: B[v,p] = +1 if p is incoming to vertex v, −1 if outgoing, 0 otherwise. 168 SM vertices × 59 particles. The 7-dimensional nullspace is the conservation algebra.
inspect values (9,912 entries)
H
W+
W-
Z
b_b
b_g
b_r
c_b
c_g
c_r
d_b
d_g
d_r
e+
e-
g_bgbar
g_brbar
g_gbbar
g_grbar
g_rbbar
g_rgbar
mu+
mu-
nu_e
nu_mu
nu_tau
photon
s_b
s_g
s_r
t_b
t_g
t_r
tau+
tau-
u_b
u_g
u_r
~b_b
~b_g
~b_r
~c_b
~c_g
~c_r
~d_b
~d_g
~d_r
~nu_e
~nu_mu
~nu_tau
~s_b
~s_g
~s_r
~t_b
~t_g
~t_r
~u_b
~u_g
~u_r
QED e- gamma
-1
QED e+ gamma
-1
QED mu- gamma
-1
QED mu+ gamma
-1
QED tau- gamma
-1
QED tau+ gamma
-1
QED u_r gamma
-1
QED ~u_r gamma
-1
QED u_g gamma
-1
QED ~u_g gamma
-1
QED u_b gamma
-1
QED ~u_b gamma
-1
QED d_r gamma
-1
QED ~d_r gamma
-1
QED d_g gamma
-1
QED ~d_g gamma
-1
QED d_b gamma
-1
QED ~d_b gamma
-1
QED c_r gamma
-1
QED ~c_r gamma
-1
QED c_g gamma
-1
QED ~c_g gamma
-1
QED c_b gamma
-1
QED ~c_b gamma
-1
QED s_r gamma
-1
QED ~s_r gamma
-1
QED s_g gamma
-1
QED ~s_g gamma
-1
QED s_b gamma
-1
QED ~s_b gamma
-1
QED t_r gamma
-1
QED ~t_r gamma
-1
QED t_g gamma
-1
QED ~t_g gamma
-1
QED t_b gamma
-1
QED ~t_b gamma
-1
QED b_r gamma
-1
QED ~b_r gamma
-1
QED b_g gamma
-1
QED ~b_g gamma
-1
QED b_b gamma
-1
QED ~b_b gamma
-1
QCD u_r -> u_g g_rgbar
-1
-1
1
QCD u_g -> u_b g_gbbar
-1
-1
1
QCD u_r -> u_b g_rbbar
-1
-1
1
QCD u_g -> u_r g_grbar
-1
1
-1
QCD u_b -> u_g g_bgbar
-1
1
-1
QCD u_b -> u_r g_brbar
-1
1
-1
QCD d_r -> d_g g_rgbar
-1
1
-1
QCD d_g -> d_b g_gbbar
-1
1
-1
QCD d_r -> d_b g_rbbar
-1
1
-1
QCD d_g -> d_r g_grbar
1
-1
-1
QCD d_b -> d_g g_bgbar
1
-1
-1
QCD d_b -> d_r g_brbar
1
-1
-1
QCD c_r -> c_g g_rgbar
-1
1
-1
QCD c_g -> c_b g_gbbar
-1
1
-1
QCD c_r -> c_b g_rbbar
-1
1
-1
QCD c_g -> c_r g_grbar
1
-1
-1
QCD c_b -> c_g g_bgbar
1
-1
-1
QCD c_b -> c_r g_brbar
1
-1
-1
QCD s_r -> s_g g_rgbar
-1
-1
1
QCD s_g -> s_b g_gbbar
-1
-1
1
QCD s_r -> s_b g_rbbar
-1
-1
1
QCD s_g -> s_r g_grbar
-1
1
-1
QCD s_b -> s_g g_bgbar
-1
1
-1
QCD s_b -> s_r g_brbar
-1
1
-1
QCD t_r -> t_g g_rgbar
-1
-1
1
QCD t_g -> t_b g_gbbar
-1
-1
1
QCD t_r -> t_b g_rbbar
-1
-1
1
QCD t_g -> t_r g_grbar
-1
1
-1
QCD t_b -> t_g g_bgbar
-1
1
-1
QCD t_b -> t_r g_brbar
-1
1
-1
QCD b_r -> b_g g_rgbar
-1
1
-1
QCD b_g -> b_b g_gbbar
-1
1
-1
QCD b_r -> b_b g_rbbar
-1
1
-1
QCD b_g -> b_r g_grbar
1
-1
-1
QCD b_b -> b_g g_bgbar
1
-1
-1
QCD b_b -> b_r g_brbar
1
-1
-1
QCD 3g g_rgbar g_gbbar g_brbar
-1
-1
-1
QCD 3g g_grbar g_rbbar g_bgbar
-1
-1
-1
NC Z -> e- e+
1
-1
-1
NC Z -> mu- mu+
1
-1
-1
NC Z -> tau- tau+
1
-1
-1
NC Z -> nu_e ~nu_e
1
-1
-1
NC Z -> nu_mu ~nu_mu
1
-1
-1
NC Z -> nu_tau ~nu_tau
1
-1
-1
NC Z -> u_r ~u_r
1
-1
-1
NC Z -> u_g ~u_g
1
-1
-1
NC Z -> u_b ~u_b
1
-1
-1
NC Z -> d_r ~d_r
1
-1
-1
NC Z -> d_g ~d_g
1
-1
-1
NC Z -> d_b ~d_b
1
-1
-1
NC Z -> c_r ~c_r
1
-1
-1
NC Z -> c_g ~c_g
1
-1
-1
NC Z -> c_b ~c_b
1
-1
-1
NC Z -> s_r ~s_r
1
-1
-1
NC Z -> s_g ~s_g
1
-1
-1
NC Z -> s_b ~s_b
1
-1
-1
NC Z -> t_r ~t_r
1
-1
-1
NC Z -> t_g ~t_g
1
-1
-1
NC Z -> t_b ~t_b
1
-1
-1
NC Z -> b_r ~b_r
1
-1
-1
NC Z -> b_g ~b_g
1
-1
-1
NC Z -> b_b ~b_b
1
-1
-1
Yuk H -> e- e+
1
-1
-1
Yuk H -> mu- mu+
1
-1
-1
Yuk H -> tau- tau+
1
-1
-1
Yuk H -> u_r ~u_r
1
-1
-1
Yuk H -> u_g ~u_g
1
-1
-1
Yuk H -> u_b ~u_b
1
-1
-1
Yuk H -> d_r ~d_r
1
-1
-1
Yuk H -> d_g ~d_g
1
-1
-1
Yuk H -> d_b ~d_b
1
-1
-1
Yuk H -> c_r ~c_r
1
-1
-1
Yuk H -> c_g ~c_g
1
-1
-1
Yuk H -> c_b ~c_b
1
-1
-1
Yuk H -> s_r ~s_r
1
-1
-1
Yuk H -> s_g ~s_g
1
-1
-1
Yuk H -> s_b ~s_b
1
-1
-1
Yuk H -> t_r ~t_r
1
-1
-1
Yuk H -> t_g ~t_g
1
-1
-1
Yuk H -> t_b ~t_b
1
-1
-1
Yuk H -> b_r ~b_r
1
-1
-1
Yuk H -> b_g ~b_g
1
-1
-1
Yuk H -> b_b ~b_b
1
-1
-1
CC W+ -> e+ nu_e
1
-1
-1
CC W- -> e- ~nu_e
1
-1
-1
CC W+ -> mu+ nu_mu
1
-1
-1
CC W- -> mu- ~nu_mu
1
-1
-1
CC W+ -> tau+ nu_tau
1
-1
-1
CC W- -> tau- ~nu_tau
1
-1
-1
CC W+ -> u_r ~d_r
1
-1
-1
CC W+ -> u_g ~d_g
1
-1
-1
CC W+ -> u_b ~d_b
1
-1
-1
CC W+ -> u_r ~s_r
1
-1
-1
CC W+ -> u_g ~s_g
1
-1
-1
CC W+ -> u_b ~s_b
1
-1
-1
CC W+ -> u_r ~b_r
1
-1
-1
CC W+ -> u_g ~b_g
1
-1
-1
CC W+ -> u_b ~b_b
1
-1
-1
CC W+ -> c_r ~d_r
1
-1
-1
CC W+ -> c_g ~d_g
1
-1
-1
CC W+ -> c_b ~d_b
1
-1
-1
CC W+ -> c_r ~s_r
1
-1
-1
CC W+ -> c_g ~s_g
1
-1
-1
CC W+ -> c_b ~s_b
1
-1
-1
CC W+ -> c_r ~b_r
1
-1
-1
CC W+ -> c_g ~b_g
1
-1
-1
CC W+ -> c_b ~b_b
1
-1
-1
CC W+ -> t_r ~d_r
1
-1
-1
CC W+ -> t_g ~d_g
1
-1
-1
CC W+ -> t_b ~d_b
1
-1
-1
CC W+ -> t_r ~s_r
1
-1
-1
CC W+ -> t_g ~s_g
1
-1
-1
CC W+ -> t_b ~s_b
1
-1
-1
CC W+ -> t_r ~b_r
1
-1
-1
CC W+ -> t_g ~b_g
1
-1
-1
CC W+ -> t_b ~b_b
1
-1
-1
TGV Z -> W+ W-
-1
-1
1
TGV gamma -> W+ W-
-1
-1
1
QGV W+ W- -> Z Z
1
1
-2
QGV W+ W- -> gamma gamma
1
1
-2
QGV W+ W- -> Z gamma
1
1
-1
-1
HSC H -> H H (off-shell)
-1
HSC H H -> H H
HVV H -> Z Z
1
-2
HVV H -> W+ W-
1
-1
-1
HVV H H -> Z Z
2
-2
params — trainable: 19 free SM parameters (+7 BSM for neutrino sector)
param
params.ckm_wolfenstein
4
float64
CKM quark-mixing matrix in the Wolfenstein parameterization: (λ, A, ρ̄, η̄). η̄ is the source of CP violation in quark physics.
inspect values (4 entries)
lambda
A
rho_bar
eta_bar
0.225
0.826
0.159
0.348
param
params.gauge_couplings
3
float64
The three gauge coupling strengths at the MZ scale: gs (strong / SU(3)), g (weak / SU(2)), g' (hypercharge / U(1)).
inspect values (3 entries)
g_s
g
g_prime
1.217
0.6517
0.3574
param
params.higgs
2
float64
Higgs sector: v is the vacuum expectation value (≈246 GeV, sets all W/Z masses); λ is the Higgs self-coupling.
inspect values (2 entries)
v
lambda
246.2
0.1291
param
params.neutrino_dm2
3
float64
Neutrino mass-squared splittings (Δm²21, Δm²31, Δm²32) in eV². Sign of Δm²32 = neutrino mass ordering, still unknown.
inspect values (3 entries)
dm2_21
dm2_31
dm2_32
7.42e-05
0.002515
param
params.pmns_angles
4
float64
PMNS neutrino-mixing angles: (θ12, θ23, θ13, δCP). Beyond the original minimal SM.
inspect values (4 entries)
theta_12
theta_23
theta_13
delta_CP
0.5831
0.857
0.1491
3.438
param
params.theta_qcd
scalar
float64
The QCD vacuum angle θ̄. Measured to be ≈ 0; the unexplained smallness is the strong-CP problem.
inspect values (1 entries)
param
params.yukawa_down_eigvals
3
float64
Yukawa coupling eigenvalues for the down-type quarks (d, s, b).
inspect values (3 entries)
d
s
b
2.70e-05
0.0005457
0.02401
param
params.yukawa_lep_eigvals
3
float64
Yukawa coupling eigenvalues for the charged leptons (e, μ, τ).
inspect values (3 entries)
e
mu
tau
2.94e-06
0.0006069
0.01021
param
params.yukawa_up_eigvals
3
float64
Yukawa coupling eigenvalues for the up-type quarks (u, c, t). Mass = y · v / √2; the top's value is O(1).
inspect values (3 entries)
u
c
t
1.26e-05
0.007294
0.9923
Particle table — the ground-truth feature dictionary made tangible
Each row is one of the 61 Standard-Model
particles; each column is one of its labels. Reading
left to right: Q is
electric charge; B is
baryon number; L_e, L_μ, L_τ are
lepton numbers (one per generation); C3, C8
are two diagonal generators of
color; spin and mass_GeV close
things out. The factorization (charge ⊗ color ⊗ flavor ⊗ generation ⊗
chirality) is exactly what the SAE in section (b) will be asked to
recover.
Each column is normalized to its own absolute maximum (shown in axis label) so the structure of each label is legible. Without this, the mass column would wash out everything else.
Vertex incidence — symmetry as the matrix's nullspace
Every allowed interaction in the SM (e.g. an electron emitting a photon,
a top quark decaying via W) is one row of this matrix; columns are
particles. A conserved quantity is, mathematically, a
vector that lies in the nullspace of this matrix: combine particles
with those weights and every interaction leaves the total unchanged. There
are exactly seven independent such vectors — the seven
conservation laws (Q, B, L_e, L_μ, L_τ, C3, C8). The
anomaly cancellation property of the SM is also a
fact about this matrix.
B[v,p] = +1 incoming, −1 outgoing for vertex v and particle p. The 7-dimensional nullspace is the conservation algebra.
The free parameters — the SM's "model weights"
These are the only numbers the universe gets to choose. Click any term to
read more:
All 26 trainable scalars (19 minimal SM + 7 beyond-Standard-Model neutrino parameters). A typical large language model has ~10¹⁰ of these; physics has 26.Further reading (curated link directory)
Neural networks & interpretability
World models (Ha & Schmidhuber) — the canonical reference for the state / transition / loop framing
How the model is "run" — one transition step at a time
With the world-model framing in hand, "running" the Standard Model is
straightforward: loop the transition function. Each transition
step is one decay event — the engine looks at the current particle
multiset, queries the bundle's tensors to score every allowed decay, samples
one, and mutates the state. Repeat until nothing can decay further. This
section walks through that loop concretely.
One transition step, written out
Pseudocode for one step:
def step(state):
# 1. enumerate allowed decays (those whose parent is in the state)
candidates = [(d, w) for d in decay_catalog
if d.parent in state
for w in [decay_weight(d, params)]
if w > 0]
# 2. sample one channel proportional to its physical rate
chosen = sample_categorical(candidates)
# 3. mutate the state
new_state = state.copy()
new_state[chosen.parent] -= 1
for p in chosen.products:
new_state[p] += 1
return new_state
Every quantity on the right-hand side is something we've already met:
decay_catalog is enumerated from vertices.incidence;
decay_weight reads
params.yukawa_* / params.ckm_wolfenstein /
params.gauge_couplings depending on the channel. So the
transition function is just a deterministic recipe that maps state →
next state using the bundle as a parameter store.
Contrast with an LLM step: state = sequence of tokens,
transition = run all transformer layers on the state and sample the next
token from the output distribution. Same loop shape; different transition
internals.
Branching fractions for three heavy particles
Before running the cascade, here's what the bundle says about three
iconic decays. Each bar's length is computed from the
Yukawa eigenvalues and
CKM matrix stored in
model.params — no fitting, just direct calculation
from the 19 free parameters. The
Particle
Data Group (PDG) values shown in each subtitle are what the real
world measures.
Branching fractions aggregated over color and summed where the catalog distinguishes channels (e.g. each H → ff̄ split per color is summed into one ff̄ entry). Top is overwhelmingly bW because Vtb ≈ 1.
A concrete worked example: one Higgs decays
The trace below was generated by running
cascade(sm, {H: 1}, catalog, max_steps=30, rng=Random(7)).
Each row is one Monte-Carlo step. The "tensors consulted" column names
the submodule the simulator read to compute that step's decay weight
— for example, when the Higgs picks a Yukawa channel the engine
reads params.yukawa_*_eigvals; when a W picks a
quark-antiquark pair it reads params.ckm_wolfenstein.
Started with 1 particle (a Higgs, mass ≈
125.1 GeV). After 1 decay step(s) the state
contained 2 stable particles
(b_r + ~b_r). Different seeds produce
different trajectories — that randomness is the entire reason
feed_cascade can supply a non-trivial distribution to the
SAE in the next section.
So why does this matter for interpretability?
The cascade is what produces feed_cascade, the only
one of the three SAE feeds that has genuine distributional structure
(the other two are static lookup tables). Each row of feed_cascade is
one final state like the one above, encoded as a bag-of-particles count
vector. The SAE will see thousands of such vectors and is asked to
discover — among other things — that they came from one of
a small set of heavy parents. The fact that the SAE recovers
origin:t_b, origin:H, etc. at AUC = 1.0 is
the SAE inferring backward through the cascade tree from a
final-state bag to its starting heavy particle. That's the
interpretability claim made concrete.
(b) How SAEs are trained and scored
A sparse autoencoder's training data is whatever activations a model
produces while running. This project ships three feeds of
increasing difficulty, all derived from the Standard Model substrate.
raw = a sanity-check shape; embedded = artificially superposed (61 concepts squashed into 16 dims, mimicking the residual stream of a large language model); cascade = bag-of-particles from a Monte Carlo decay simulator — the only feed with genuine distributional structure.
Each trained sparse autoencoder (SAE) produces sparse latent codes. We
score each latent feature by the
Area Under the ROC Curve (AUC)
against every ground-truth label
(color:r, generation:2, origin:t_b, …):
one cell of the alignment matrix below. AUC = 1.0 means the SAE feature
fires on exactly the inputs that carry that label; AUC = 0.5 means no
better than chance.
Each cell is an AUC matrix for one (feed, variant). Rows reordered so a SAE feature's best GT match sits along the diagonal. Brighter = higher AUC.
Headline numbers per (feed, variant)
feed
variant
n_sae
active
cov≥0.95
cov≥0.90
mean_best_auc
monosemantic
raw
topk
32
32
56.4%
65.5%
0.898
100.0%
raw
l1
32
32
56.4%
66.4%
0.891
100.0%
raw
jumprelu
32
32
47.3%
58.2%
0.875
100.0%
embedded
topk
64
63
70.0%
72.7%
0.902
100.0%
embedded
l1
64
64
71.8%
72.7%
0.894
100.0%
embedded
jumprelu
64
64
69.1%
74.5%
0.917
100.0%
cascade
topk
128
113
20.7%
30.6%
0.705
48.7%
cascade
l1
128
128
28.9%
33.9%
0.726
42.2%
cascade
jumprelu
128
128
37.2%
45.5%
0.767
80.5%
Notice monosemanticity collapses on the cascade feed
while staying at 100% on the easy feeds — that's where the project starts
earning its keep as a benchmark.
(c) How polygram captures structure
polygram is the worked example of a compression / extraction technique in this report. This section walks through one MPSRung1 dictionary — how it's built, what the interference / cancellation experiments look like, and the Orca artifact polygram emits. The candid pass/fail across encoding configurations (which is where the actionable story is) lives on the scoreboard.
Dictionary: 8 features in 4 clusters
polygram encodes the 8 chosen particles as quantum states using a Matrix Product State (MPS) — specifically Rung 1 with bond dimension 2, which means the 8 concepts live in the Hilbert space of 3 qubits (2³ = 8 amplitudes per state). Each particle is a leaf in a 4-cluster hierarchy:
charged_lepton
e-
e+
up_quark
u_r
~u_r
down_quark
d_r
~d_r
boson
photon
W+
Interference sweep — phase as a knob
Rotating e⁺'s phase moves it around the Bloch sphere; the inner product with e⁻ modulates sinusoidally. The minimum is the structural floor — the level below which phase alone cannot push the overlap.
Gram matrix at the optimised dictionary
The 8×8 squared-inner-product matrix among the prepared concept states after the cancellation search. Off-diagonal cells should be near zero for unrelated concepts; cluster boxes should remain intra-bright if preserve_tiers worked. Computed analytically from the optimised prepare_concept circuit parameters — the deeper Orca state-machine artifact is a lower-level concern best inspected in the per-pair runs/polygram/cancellation_*/ directories.
Cells: |⟨ψ_i|ψ_j⟩|² for the 8 prepared concept states. White rectangles mark the four cluster boundaries (charged_lepton / up_quark / down_quark / boson).
The benchmark scoreboard
sm-sae is intentionally a benchmark fixture, not a
finished pipeline. Its job is to make compression and extraction techniques
(polygram is the worked example here, but any technique that takes an SAE
dictionary and proposes a simpler structured representation should fit) directly
comparable and measurable against exact ground truth. This
section reports the current scoreboard candidly — including, in
particular, where the polygram baseline fails.
Three axes of measurement
axis
what it measures
high score means…
A. reconstruction
does the SAE re-produce activations faithfully?
the SAE has enough capacity for the feed
B. ground-truth alignment
do SAE features map onto known physical labels?
features are interpretable, not polysemantic
C. compression extraction
can we re-encode the SAE dictionary into a simpler, structured form
(polygram, decision tree, sparse code, …) that still respects the
known feature factorization?
downstream tools (sae-forge, distillation) can consume it
Axes A and B above are pre-polygram
(encoding-independent). Their post-polygram variants
— the same scores re-computed on the kept feature subset, which
is encoding-dependent — live in the
Forge pipeline runs table further down, in the
post-A, Δ A, post-B, Δ B columns.
Axis A — Reconstruction quality
How faithfully does the trained SAE reproduce the feed activations? This
is the easiest axis: every variant clears 96% variance explained.
feed
variant
var explained
recon loss
L0
dead %
raw
topk
0.9999
0.0196
6.0
0.0%
raw
l1
1.0000
0.0028
12.1
0.0%
raw
jumprelu
1.0000
0.0005
14.3
0.0%
embedded
topk
0.9988
0.0001
6.0
0.0%
embedded
l1
0.9837
0.0010
5.2
0.0%
embedded
jumprelu
1.0000
0.0000
18.3
0.0%
cascade
topk
0.9934
0.0003
12.0
0.0%
cascade
l1
0.9608
0.0018
7.2
0.0%
cascade
jumprelu
0.9986
0.0001
48.2
0.0%
Axis B — Ground-truth feature alignment
Best result on the hardest feed (cascade): 37.2% of ground-truth features recovered at AUC ≥ 0.95. That is the gap the benchmark is asking the next technique to close.
feed
variant
cov ≥0.95
cov ≥0.90
mean best AUC
monosemanticity
raw
topk
56.4%
65.5%
0.898
100.0%
raw
l1
56.4%
66.4%
0.891
100.0%
raw
jumprelu
47.3%
58.2%
0.875
100.0%
embedded
topk
70.0%
72.7%
0.902
100.0%
embedded
l1
71.8%
72.7%
0.894
100.0%
embedded
jumprelu
69.1%
74.5%
0.917
100.0%
cascade
topk
20.7%
30.6%
0.705
48.7%
cascade
l1
28.9%
33.9%
0.726
42.2%
cascade
jumprelu
37.2%
45.5%
0.767
80.5%
Easy feeds (raw, embedded) are at
100% monosemanticity but cap around 70% coverage. The realistic feed
(cascade) drops to ~37% coverage and ~80% monosemanticity even
with the best variant. That gap is the open problem.
Axis C — Compression extraction (per polygram encoding)
1 of 2 displayed encoding configurations clear every cancellation pair. Cells show pass (✓ green) or fail (✗ red) with the achieved Δ in overlap. Hover for the exact value.
Hidden from this table: MPSRung1_phase (0 passes; recorded on disk under runs/polygram/sweep/ for the historical record).
Each row is one polygram encoding configuration tested
against the same 4 cancellation pairs. The dimension of the search and
the optimizer budget are the two knobs that vary; see the
recommended-defaults table below for
the actionable summary.
Worth flagging:Rung5(n_amp_qubits=2) and Rung4 are
mathematically identical encodings — Rung5's own
docstring states "Generalises Rung4 (the fixed n_amp_qubits=2
case)". They produce the same gram and the same optimum. But in
current polygram (v0.8.1) Rung5's code path is much faster: at the
same max_steps=10 budget Rung5 lands the same Δ in
~4 seconds per pair versus Rung4's ~240 seconds, an
observed ~60× gap. Filed upstream as
polygram#86; for now, prefer
Rung5(n_amp_qubits=2) over Rung4 for new
callers.
sae-forge as a candidate benchmark entry
sae-forge sits one stage downstream of polygram:
it consumes a polygram-compressed SAE checkpoint and projects a host
model's weights into the surviving feature basis, producing a small
transformer whose residual stream is the SAE feature space.
Its "wider loops" (per its README) are a three-loop continual-learning
structure on top of the single-shard pipeline:
stream — iterate over data shards
refine — per-shard convergence
basis — compress ↔ regrow refinement
of the feature set (the "wider" loop the README highlights)
Status: running end-to-end on sm-sae against sae-forge v0.5.1
sae-forge v0.5.1 ships everything sm-sae actually consumes:
FaithfulnessTarget (v0.4.0), the bundled
GroundTruthTarget (v0.5.0) used here as the Axis-C scorer,
and the WorldModel protocol (v0.5.1) that makes the
family registry pluggable. The pipeline runs end-to-end on every one
of the 9 (feed, variant) cells in the
Forge pipeline runs table; the headline
post-W_enc-fix result is cascade__jumprelu reaching 12
Compressor clusters from its full 128-feature SAE width, up from 1
cluster under the legacy rung3+head
defaults.
The host slot: cascade-host shim is wired and opt-in
sae-forge's run_synthetic path projects a host
transformer's weights into the feature basis. sm-sae ships
scripts/train_cascade_host.py, which trains a tiny
GPT-2 on next-state cascade transitions and saves the checkpoint
under runs/cascade_host/<n_embd>/host/. The forge
pipeline picks it up automatically (load-then-fallback), and the
host column in the runs table renders 🎓 for trained, 🎲
for the random-init fallback. The snapshot above has 6/9 rows
trained (embedded + cascade feeds, where the shim's
n_embd matches the SAE's input_dim); the
3 raw rows stay 🎲 because raw SAEs are 9-dimensional and the
cascade-transition vocabulary doesn't compress meaningfully into 9
hidden dims. Trained-vs-random faithfulness deltas on the 6 🎓
cells land at +0.001 to +0.036 — positive and directionally
consistent, but below the ≥ 0.05 gate originally proposed for
the cascade-host shim even at the post-W_enc-fix 12-cluster width.
The remaining gap is most plausibly the cascade-host's training
objective: pure next-state token prediction doesn't demand
that conservation / lineage information land in the residual stream
(see econ-sae's Phase 5+ findings for the same
lesson in the macro domain). An auxiliary-supervision head on the
cascade host is the natural follow-up.
Would the wider loops compensate for polygram weakness?
Two cases to keep apart:
Over-compression. If polygram drops features
that carry signal, sae-forge's basis loop (compress
↔ regrow) is designed to bring them back. This is a real
mitigation, untested on this fixture today.
Missing structure. If polygram's encoding cannot
express a needed factorization (the original
MPSRung1+phase-only failure mode), no amount of regrow helps:
you cannot regrow something polygram never had a way to find.
The right answer there is to fix the encoding upstream —
which is what the current Rung5 default delivers, with cap=128
matching the cascade SAE's full feature count.
So the conceptual answer to "can sae-forge's wider loops
compensate?" is: yes for one failure mode, no for the
other. The current encoding default removes the second class
of gap for this fixture; the first class is the natural next thing
to measure but isn't a sm-sae-side blocker.
Where the remaining gaps come from, by layer
layer
current gap
where to look
SAE training (Axis A)
none worth chasing on this fixture — ≥96% variance
explained across all 9 configurations
becomes interesting again at LLM scale
SAE coverage (Axis B)
~63% of GT features missed on the cascade feed
(best variant: JumpReLU at 37% coverage at AUC ≥ 0.95)
still the main open gap. The
econ-sae result that "substrate richness
beats SAE tuning" suggests the load-bearing knob is the
substrate (the cascade itself and its encoded
representation), not the SAE architecture.
polygram compression (Axis C)
resolved at default settings — Rung5 with
n_amp_qubits=4 passes the cascade SAE's full
128 features through to the Compressor end-to-end (PR #5)
fine as-is. The earlier
cap > sae.input_dim IndexError that
masqueraded as over-consolidation was sm-sae writing W_enc
with the wrong axis order; polygram's contract was correct
all along.
sae-forge faithfulness (downstream of C)
9/9 cells produce a faithfulness number; 6/9 use a
cascade-trained host. Trained-vs-random delta is positive
but below the ≥ 0.05 gate originally proposed.
the natural next experiment is supervising the cascade
host on conservation / lineage labels (the econ-sae
Phase 5+ recipe transposed). Separately, sae-forge's
basis-regrow loop on intentionally over-compressed configs
is the unstarted "earn the wider loops" experiment.
Honest read: with the W_enc fix and the
cascade-host shim both shipped, the wiring and architecture are no
longer the bottleneck. The unresolved question is whether the host's
training objective demands the cascade structure cleanly enough
for the SAE to see it post-projection — an objective-side
question, not an architecture-side one.
Forge pipeline runs
Per-SAE runs of the end-to-end pipeline. All 9 stages are wired
end-to-end: SAE → polygram Dictionary → ValidationReport → Compressor
→ compressed safetensors → sae-forge run_synthetic with
the bundled GroundTruthTarget as the faithfulness
scorer. The baseline column reports the raw SAE's coverage
at AUC ≥ 0.95 — the number the forged model has to beat (or
at least match) for the forge to be worth doing on this fixture.
run
encoding
selection
host
dict features
compress
baseline cov≥0.95
post-A
Δ A
post-B cov
Δ B
forge score
forge stage
cascade__jumprelu
mps_rung1
firing_rate
🎓 trained (loss=1.778)
8
clusters 2, kept 2, zeroed 6
37.2%
0.947
-0.052
35.5%
-1.7%
0.757
ok
cascade__jumprelu
rung3
firing_rate
🎓 trained (loss=1.778)
16
clusters 3, kept 3, zeroed 10
37.2%
0.927
-0.071
34.7%
-2.5%
0.755
ok
cascade__jumprelu
rung4
firing_rate
🎓 trained (loss=1.778)
32
clusters 2, kept 2, zeroed 24
37.2%
0.812
-0.187
34.7%
-2.5%
0.747
ok
cascade__jumprelu
rung5
firing_rate
🎓 trained (loss=1.778)
128
clusters 12, kept 12, zeroed 84
37.2%
0.399
-0.600
27.3%
-9.9%
0.726
ok
cascade__l1
rung5
firing_rate
🎓 trained (loss=1.778)
128
clusters 12, kept 12, zeroed 48
28.9%
—
—
—
—
0.733
ok
cascade__topk
rung5
firing_rate
🎓 trained (loss=1.778)
128
clusters 5, kept 5, zeroed 5
20.7%
—
—
—
—
0.746
ok
embedded__jumprelu
rung5
firing_rate
🎓 trained (loss=1.768)
64
clusters 9, kept 9, zeroed 13
69.1%
—
—
—
—
0.882
ok
embedded__l1
rung5
firing_rate
🎓 trained (loss=1.768)
64
clusters 5, kept 5, zeroed 5
71.8%
—
—
—
—
0.876
ok
embedded__topk
mps_rung1
firing_rate
🎓 trained (loss=1.768)
8
clusters 1, kept 1, zeroed 5
70.0%
0.911
-0.088
70.0%
+0.0%
0.892
ok
embedded__topk
rung3
firing_rate
🎓 trained (loss=1.768)
16
clusters 4, kept 4, zeroed 7
70.0%
0.911
-0.088
70.0%
+0.0%
0.891
ok
embedded__topk
rung4
firing_rate
🎓 trained (loss=1.768)
32
clusters 3, kept 3, zeroed 13
70.0%
0.779
-0.220
70.0%
+0.0%
0.891
ok
embedded__topk
rung5
firing_rate
🎓 trained (loss=1.768)
64
clusters 4, kept 4, zeroed 4
70.0%
0.947
-0.052
70.0%
+0.0%
0.888
ok
raw__jumprelu
rung5
firing_rate
🎲 random
32
clusters 4, kept 4, zeroed 5
47.3%
—
—
—
—
0.841
ok
raw__l1
rung5
firing_rate
🎲 random
32
clusters 5, kept 5, zeroed 7
56.4%
—
—
—
—
0.847
ok
raw__topk
rung5
firing_rate
🎲 random
32
clusters 2, kept 2, zeroed 3
56.4%
—
—
—
—
0.842
ok
Reading the post-A and
post-B columns. Axes A (reconstruction quality) and
B (GT alignment) at the top of the report measure the SAE
alone — they are encoding-independent because they
don't depend on polygram's compression choices. post-A
and post-B re-score the same axes on the
kept feature subset the Compressor produces, and those
numbers are encoding-dependent: different polygram encodings
cap feature counts differently (MPSRung1 = 8, Rung3 = 16, Rung5(4) =
128) and the Compressor's merge decisions depend on the encoding's
gram matrix. The Δ A / Δ B columns
are the cost of compression for that cell — green (≥ -0.02) is
essentially no degradation, amber (-0.10 to -0.02) is a measurable
cost worth flagging, red (< -0.10) means the kept subset lost a
meaningful fraction of the SAE's signal.
Reading the host column.
sae-forge's ForgePipeline.run_synthetic projects a host
transformer's weights into the polygram feature basis. With a
🎓 trained host (one that has actually been trained
on cascade transitions via
scripts/train_cascade_host.py --n-embd <input_dim>),
the resulting forge score reflects how well the projection preserves
cascade structure — this is what the benchmark exists to
measure. With a 🎲 random host (the fallback when no
trained host exists for the SAE's input_dim), the forged
model's residuals carry no real signal and the score reflects the
AUC-pooling stumbling onto coincidental correlations. Rows with
🎲 are wiring sanity checks, not scientific claims; only 🎓 rows
belong in Axis-C comparisons.
Reading the selection column.
The polygram encoding caps the Dictionary at a number of features
determined by its rung and amplitude-qubit count (the current default
Rung5 with n_amp_qubits=4 caps at 128, matching the
cascade SAE's full width; earlier defaults capped lower). When the SAE
has more features than the cap, we pick a subset. head
is the legacy behaviour: take the first N features by ID,
which is arbitrary because SAE feature IDs reflect training-time
allocation order, not utility. firing_rate (the current
default) keeps the features that actually fire on the feed;
gt_alignment keeps the features most discriminative
against GT labels. Selection moves downstream Compressor cluster
counts non-trivially: on cascade__jumprelu at the
current Rung5/cap=128 default,
firing_rate yields 12 clusters; under the legacy
rung3/head setting that combination produced only 1
cluster because the arbitrary slice through 128 features happened to
miss most of the confirmed pairs the ValidationReport identified.
To populate this table, run
python scripts/forge_pipeline.py <feed>__<variant>
[--select-by firing_rate|gt_alignment|head]. The pipeline writes
runs/sae_forge/<run_id>/forge_results.json
which this section reads on rebuild.
Recommended defaults — the bottom line
One of the things a benchmark is supposed to produce is concrete
configuration guidance, not just scores. Below are the recommended
keyword arguments for both packages on this kind of fixture
(ground-truth-rich, compression / extraction quality is the primary
signal). Each is tagged with the level of evidence behind it:
measured — directly supported by the sweep
results above (cells in the sweep table).
inherited — package ships this default and the
sweep gives no reason to change it.
provisional — defensible from first principles
or upstream docs, but not yet directly verified on this
fixture.
Polygram
parameter
location
upstream default
sm-sae recommendation
evidence
encoding
Dictionary(encoding=...)
(must specify)
Rung3(bond_dim=2) (pending sweep)
Rung3 passes 4/4; MPSRung1+phase-only passes 0/4. The polygram docstring predicts this; the sweep confirms.
provisional
Cancellation(encoding=...)
polygram.Cancellation
None (= phase-only 2-knob)
"rung3"
Selects the 4-knob default list [a.phi, b.phi, b.theta_amp, b.psi_aux] which unlocks the amplitude branch.
provisional
optimize.method
Cancellation(optimize={...})
"grid"
"grid" for ≤4 knobs; "scipy" for >4
GRID_KNOB_LIMIT = 4 in polygram; Rung4/Rung5 (6+ knobs) require scipy.optimize.differential_evolution. scipy is in polygram[opt].
measured
assign_amp_knobs
SAEImportConfig / from_sae_lens(...)
False
True (for ground-truth-rich fixtures)
Polygram defaults to neither amp nor phase knobs when ingesting an SAE. Without amp knobs, the cancellation step is stuck at the structural floor (see Axis C baseline). For any benchmark that measures structured-feature recovery, this should be flipped.
measured
assign_phase_knobs
SAEImportConfig
False
True
Required to populate the .phi entries the 4-knob Rung3 search expects.
measured
tolerance
CancellationConfig
0.05
0.05 (no reason to change)
All Rung3 passes met this comfortably — every Δ landed at exactly 0.0, well inside tolerance.
inherited
preserve_tiers
CancellationConfig
True
True
Cluster hierarchy is what makes the SAE features physically interpretable; default behaviour preserves it through cancellation without harming the score.
inherited
strategy
CompressionConfig
"merge"
"merge"
Not directly tested by the sweep (Compressor adapter not yet built); the default is the only path with end-to-end sae-forge support per its README.
inherited
rep_selection
CompressionConfig
"scale_aware"
"scale_aware"
sae-forge's FeatureBasis depends on the scale-aware merge to preserve decoder magnitudes (see sae-forge README, "forge-feature-scales").
inherited
n_clusters
SAEImportConfig
2
match the fixture's known cluster count
For sm-sae's 8-particle slice, the right value is 4 (charged_lepton / up_quark / down_quark / boson). For general fixtures, this should track the number of natural clusters in the GT label set.
provisional
sae-forge
sae-forge v0.5.1 ships the
FaithfulnessTarget protocol (v0.4), the bundled
GroundTruthTarget (v0.5.0), and the
WorldModel protocol (v0.5.1). The pipeline runs
end-to-end against this fixture today — faithfulness numbers
in the Forge pipeline runs table.
Recommendations below are upgraded from provisional to
measured where the wiring path actually executed;
entries gated on auxiliary cascade-host supervision (next-step
follow-up) remain provisional with a note.
parameter
location
upstream default
sm-sae recommendation
evidence
faithfulness
ForgePipeline.__init__
None (defaults to per-token KL vs host)
KLTarget() for LLMs; GroundTruthTarget(labels=Y) for label-rich fixtures (sm-sae, mixture-of-gaussians, etc.). Bundled in saeforge.eval.targets as of v0.5.0.
Wired end-to-end as of sae-forge v0.5.1; scripts/forge_pipeline.py uses the bundled GroundTruthTarget against the sm-sae GT label matrix. Hard-coded KL would compare the forged model to its host (random-init or cascade-trained, see host column), not to the physical labels we care about.
measured
scale_boost
SubspaceProjector.__init__
1.0 (with footgun warning on over-complete bases)
"auto" (resolves to min(1.0, d_model / n_features))
sm-sae bases are over-complete in every cell where the kept feature count exceeds the host's d_model (e.g. Rung3 keeps 16 features in a 16-d embed → projector resolves 0.28 on embedded__topk); for under-complete cells (Rung5 on cascade kept 12 in 16-d) auto stays at 1.0 so the existing results don't move. Measured: embedded__topk faithfulness 0.8892 → 0.8910 (Δ +0.0018, no regression); cascade__jumprelu unchanged at 0.7262. Footgun warning no longer fires.
measured
strategy
Regrower.__init__
"protected"
"protected"
Protects features above a configured threshold from being dropped during compress; this is the structural-EWC behaviour the basis loop is built around (see docs/advanced-fsm-options.md).
inherited
n_init
Regrower.__init__
4
4 for small fixtures; 8+ at LLM scale
The basis loop's regrow re-tries this many initializations. Default suffices for sm-sae sized inputs; larger initial counts matter more when the SAE has thousands of features.
provisional
polygram_overlap_threshold
BehaviouralValidator / EpochCompressor
0.7
0.7
Default validator threshold for behavioural equivalence; sae-forge ships this as a sensible single-shard default.
inherited
cosine_threshold
EpochCompressor
None (off)
None for sm-sae-sized SAEs; 0.85+ for LLM-scale
Adds a cosine-similarity criterion to per-feature inclusion. For a 128-feature SAE the extra signal is small; for tens of thousands of features it becomes load-bearing.
provisional
save_intermediate_reports
EpochCompressor
False
True for benchmark runs
Lets the basis loop's per-iteration scoreboard be inspected, which is exactly what a benchmark wants.
provisional
The Rung3 sweep has not finished yet, so a couple of the polygram recommendations are tagged provisional. They will be re-tagged measured after the next rebuild once sweep_results.json shows Rung3 at 4/4.
What would move each axis
Axis A: already saturated for this fixture size;
becomes interesting again at LLM scale.
Axis B (the SAE itself): try gated SAEs;
Anthropic-style top-k with auxiliary reconstruction loss; richer
cascade distributions (deeper decay trees, multi-parent events);
feature-splitting with hierarchical SAEs.
Axis C (polygram specifically): the baseline
MPSRung1+phase-only failure is now superseded by three
passing configurations (Rung3, Rung4,
Rung5, all with amplitude knobs). The cost ordering
ended up being the surprising part of the sweep: Rung5
with n_amp_qubits=2 converges in seconds where
Rung4 at the same knob-count takes minutes — so
the recommended default has shifted accordingly. Remaining sweep
directions worth exploring: preserve_tiers=False (cost
of relaxing hierarchy); higher n_amp_qubits on Rung5;
running the same sweep against a larger SAE slice (the current cap
is 8 features for MPSRung1, 16 for Rung3, etc.).
Other compression/extraction techniques worth benchmarking
The benchmark is encoding-agnostic. Anything that consumes a trained
SAE checkpoint and proposes a structured representation fits, including:
Hierarchical / group-sparse coding — impose
the known cluster structure as a prior
Causal scrubbing / attribution patching
— identify which features are actually load-bearing
Direct probing of feature factorization —
attempt to recover charge ⊗ color ⊗ flavor ⊗ generation
as an explicit tensor product
Distilled small transformer (the
sae-forge direction) — forge a tiny interpretable
model whose internals match the SAE's features
Each gets a row on the Axis-C table. The benchmark stays the same;
what varies is the entry.
(d) The benchmark loop, and how it scales to real models
The top row is the single-shot forward pipeline; the red dashed arrows
below are the feedback loops that make this a loop rather than a pipeline.
Each loop is one mechanism by which a poor benchmark score drives a
change earlier in the chain — from the cheapest to the most
expensive intervention:
basis loop (compiled artifact → polygram):
sae-forge's compress↔regrow refinement; or, in our sweep,
switching encoding from MPSRung1 to Rung3/5. Cheapest to iterate
on, no SAE re-training needed.
refine loop (compiled artifact → SAE):
re-train the SAE with a different objective, more features, or
different sparsity penalty when the basis loop bottoms out.
Mid-cost.
stream loop (compiled artifact → substrate):
enrich the data feed itself — deeper cascade trees,
multi-event aggregation, larger cascade distributions. Most
expensive; only justified when the prior two loops have hit
their ceiling.
A candidate technique enters at the third box (it takes the SAE's
discovered features and proposes a simpler structured representation)
and gets scored against the SM's exact ground truth at every arrow. The
scoreboard in the previous section reports the current best entry
— including, for honesty, where it fails.
Everything above used the SM as a ground-truth oracle on the left side
of every comparison. At LLM scale you lose the oracle; the visualization
shifts from "did the SAE recover X?" to
"what does the SAE propose as X, and does it survive
perturbation?". The same four boxes still apply, with substitutions:
"this SAE feature fires on Python code" + steering test
polygram Dictionary with clusters
UMAP / agglomerative clusters over SAE features
cancellation e⁻/e⁺ overlap → floor
"are these two features truly orthogonal, or is there a shared component?"
compiled artifact (today: q.orca.md)
whatever the downstream stage consumes — format depends on
the technique; see the table below
What goes in the "compiled artifact" box?
The fourth box of the flow diagram is intentionally generic. polygram
emits q.orca.md because polygram models features as
quantum states (qubit registers + Ry/Rz/CNOT
preparation circuits), which fits the
q-orca-lang state-machine grammar exactly. Other
extraction techniques would emit different things:
extraction technique
natural compiled artifact
state-machine shape?
polygram (any rung)
q.orca.md (quantum state machine)
yes — q-orca
hierarchical / group-sparse SAE
safetensors with cluster manifest JSON
no — continuous weights
decision-tree distillation
orca.md (classical state machine) or polygram
decision-table format
yes — classical orca
logic-program / rule extraction
Datalog / Prolog / decision table
yes — finite
sae-forge "native model"
safetensors weights + architecture JSON
no — small transformer
concept-bottleneck network
safetensors + concept-label sidecar
partial
symbolic regression
SymPy / JSON expression tree
no — algebra
So the question "could any encoding be expressed as a q.orca.md
state machine?" has a precise answer: only the techniques
whose output is genuinely a finite-state structure.
quantum-state encodings (polygram), classical state machines, decision
tables, and rule sets all fit. Continuous-weight outputs (distilled
transformers, concept-bottleneck networks, symbolic regression) need
different serializations because forcing them through q.orca.md would
either lose the parameters that matter or require synthesizing fake
discrete states to wrap them.
The Orca family (q.orca.md for quantum,
orca.md for classical, decision-table format) is broad
enough to cover a large fraction of the candidate techniques — but
it isn't universal. The lifecycle box should be read as "some
serialized form the next stage can consume", with the format
captured per row in the
recommended-defaults table.
What new Orca-shaped dialects might we need?
Each "no" row in the table above is a gap. Orca's design strengths
(declarative, hand-readable, verifiable invariants, compile/simulate
semantics) are valuable enough that asking "what would an Orca for
this look like?" is worth doing for each gap, rather than defaulting
to opaque binary formats. Sketches of what each extension would need to
express:
proposed dialect
covers
key primitives
characteristic invariants
n-orca (neural)
distilled small transformers (sae-forge native models),
concept-bottleneck networks
Bayesian networks, factor graphs, Markov random fields over
extracted features
variable declarations with domains, factor tables (extension of
decision-table syntax), conditional independencies as
assertions
well-formed joint (factors sum to 1 when normalized),
conditional-independence claims verifiable from factor
structure
c-orca (circuit / dataflow)
mechanistic-interpretability circuit findings (subgraphs of an
LLM that implement a specific behaviour)
DAG of operations with explicit signal types, named nodes from
the host model (residual stream component, attention head,
MLP neuron), edge weights / attribution scores
circuit completeness (intervening on the circuit reproduces
host behaviour to a tolerance), node-attribution
consistency
Two of these (n-orca, s-orca)
could plausibly be sub-grammars of existing orca-lang plus a sidecar
binary, rather than entirely new dialects. The other three
(a-orca, p-orca, c-orca) have
shapes that don't fit a state-machine framing at all and would be new
languages in the Orca family, sharing only the design philosophy
(declarative, verifiable, hand-readable).
Anticipated future requirements (worth designing for now)
Cross-dialect composition. Real distilled models
are rarely one shape. A useful interpretable model might be
transformer body + algebraic concept layer + decision-tree
output head — three different dialects in one artifact.
An umbrella "manifest" format that imports from multiple Orca
dialects (and a clear handoff semantics for signal types crossing
dialect boundaries) is the closest analogue to ONNX's graph-import
mechanism, but with Orca's verifiable-invariants discipline.
Schema versioning. Extraction techniques evolve.
Today's q.orca.md is at a specific Q-Orca version; an
artifact a year old needs to round-trip on a current toolchain.
Standard semver on dialect grammar + capability negotiation between
producer / consumer.
Round-trip verification. Given an artifact, can
we re-run the original extraction and confirm we get the same
compiled output (up to documented non-determinism)? This is the
"lossless compression" check, and it's what makes the benchmark's
Axis A measurable post-compression.
Hardware targeting. The same artifact might
compile to GPU, CPU, neuromorphic, or quantum back-ends.
q-orca already has this via its compile step; the
other dialects need analogous targets. Important once
n-orca-style artifacts ship at LLM scale.
Differentiability / continual learning. If a
compiled artifact needs to be re-finetuned (sae-forge's basis
regrow loop is the immediate example), the dialect must preserve
the gradient surface — either by inlining weights or by
referencing them with a guarantee that the host autograd graph
can attach.
Sharded artifacts. A 9B-parameter forged model
can't fit one file. The manifest needs to declare shard boundaries
with content-addressed references (sha256 + size) so a downstream
tool can validate completeness before running.
Provenance / audit trail. Each artifact should
carry the extraction technique's identifier, version, config
hash, source-SAE hash, and the benchmark scores it achieved.
Without this, a year from now nobody knows whether a given
compressed.safetensors came from
Rung3-default or Rung5-budget, and the
scoreboard becomes uninterpretable.
Partial-information artifacts. An intermediate
result that hasn't passed all gates (e.g. polygram cancellation
that stopped at the structural floor) is still valuable for
diagnosis. The format should accommodate "this is what we got
and these gates didn't pass", not just "this is the
final answer" — the current
SM_..._summary.md files in the sweep are a good
example of this done well; codify it.
The honest summary: q.orca.md is one good answer for
quantum-state encodings. A handful of cousin dialects covering algebra,
sparsity, probabilistic structure, and circuits would let a much wider
benchmark span (decision trees, distilled transformers, mechanistic
circuits, concept bottlenecks) all emit Orca-flavoured artifacts that
the same downstream tooling can consume. Designing for cross-dialect
composition, schema versioning, and provenance now — even before
the cousin dialects exist — would keep the lineage coherent as
the family grows.
sm-sae's purpose is to be the
calibration rig for every arrow in that flow: at this
scale each arrow is verifiable; at LLM scale each arrow becomes a research
question, and you debug it by checking the technique still behaves
correctly on the fixture. A technique that doesn't clear the SM benchmark
shouldn't be trusted at LLM scale.
