What is an SAE?
Introduction
Project introduction
What is an SAE?
Inside a language model, each token is represented by an activation vector. Individual coordinates can mix several kinds of information, so reading one coordinate rarely gives a complete explanation of what the model represents.
A sparse autoencoder, or SAE, learns a larger dictionary of directions that can reconstruct those vectors using relatively few active features per sample. The feature values are the strengths of those directions. Their meanings must be investigated using text examples and experiments.
What are we learning to reconstruct?
The SAE reconstructs internal activation vectors, rather than the original text. Its encoder subtracts a decoder bias, projects the input into feature space, adds an encoder bias, and applies ReLU. The decoder combines those feature strengths with learned directions and restores the decoder bias.
A reconstruction term rewards accuracy while a sparsity term discourages large total feature magnitude. Good reconstruction alone does not establish interpretability. You will measure activation statistics, propose labels, seek counterexamples, and test how interventions change a compatible language model.
One activation, opened up
1-layer story modelActivation
dense, hard to read
Sparse features
3 of 8 active
Reconstruct
low error
Sparse
few active
Tested
by intervention
The investigation you will build
01
Build the primitives
Implement forward computation, the loss, feature-frequency statistics, and example retrieval on small supplied arrays.
02
Inspect feature evidence
Train a small SAE and compare strong examples, weak matches, and counterexamples before assigning provisional labels.
03
Run controlled ablations
Remove one feature from the model's activations and compare the change in loss with random control features.
Start with the forward pass
Map a batch of activation vectors to sparse features and back.