From Primitives to Features
Introduction
Project introduction
From Primitives to Features
Your four NumPy functions define the core measurements of the lab. The forward pass produces features and reconstructions, the loss supplies the training objective, and the other two measure how often features fire and retrieve the evidence behind them. In the notebook you call these same functions on a real language model.
The model is a one-layer transformer trained on short children's stories, small enough to run in NumPy. You collect its MLP activations, train a small SAE with your functions, then analyze a 1024-feature SAE trained with the same code for longer.
What does a feature ablation actually test?
Let x be an original model activation, z its SAE features, and D[j] one decoder direction. Removing feature j from the original activation gives x_ablated = x - z[j] × D[j]. This preserves the original SAE reconstruction residual instead of silently replacing x with an imperfect reconstruction.
Compare the loss change at the positions where the feature fires with the change from removing random features that fire about as often. A clear gap is evidence that the model relies on the feature in these stories. It does not establish a universal feature meaning.
One activation, opened up
1-layer story modelActivation
dense, hard to read
Sparse features
3 of 8 active
Reconstruct
low error
Sparse
few active
Tested
by intervention
Keep the experiment traceable
01
Keep the setup fixed
Use the same model, stories, and SAE for every measurement so that differences come from the intervention, not from the setup.
02
Challenge the label
Save token positions and surrounding text. Include high-activation examples outside the proposed concept and low-activation examples inside it.
03
Compare against controls
Report the loss change where the feature fires next to random control features and a feature the model clearly depends on.
Build the complete Feature Lab
Run the full investigation in the browser and write an evidence-based verdict on one feature.