9. The practitioner's stack
Script: scripts/stack.py ·
Tests: tests/test_stack.py ·
Results: runs/stack/
This repository wrote its own hooks, lens, patches, dictionary and steering so that nothing is hidden. Libraries exist for each of these pieces. This step maps the pieces to them, and runs one library on the open model of step 8 to see that it gives the same numbers.
Two rules for this note. Every call below was read on 2026-11-21 in the
library's documentation or, for decompose_resid, logit_attrs,
logit_lens and the fwd_hooks argument with its hook function, in the
docstrings and signatures of the installed package.
AutoModelForCausalLM.from_pretrained was run and not looked up. And each
row says whether it was run here or only read. Versions are given where a
library states one. circuit-tracer was read at its main branch (its latest
tag is v0.5.3), and the documentation sites of TransformerLens and nnsight
do not say which version they describe.
The first rule earned its place while this note was written. Tutorials for
TransformerLens written before version 3 load a model with
HookedTransformer.from_pretrained. Its README says that this "was
removed in TransformerLens 4.0", and the migration guide that importing
the class raises an error. The replacement is TransformerBridge.
The map #
| In this repository | A library | The call, as documented | |
|---|---|---|---|
Tap: record a module's output, or replace it |
PyTorch 2.15 | module.register_forward_hook(hook): the hook receives (module, args, output) and may return another output. register_forward_pre_hook(hook): (module, args), the positional arguments, and may return others. Both return a handle with remove(). |
run: it is what Tap uses |
with Tap(model) as tap: model(tokens) |
TransformerLens 4.4.0 | bridge = TransformerBridge.boot_transformers(name), then logits, cache = bridge.run_with_cache(tokens) |
run, with hf_model= and tokenizer= from the files on disk |
tap["resid", i] |
cache["blocks.{i-1}.hook_resid_post"], and cache["hook_embed"] for tap["resid", 0] |
run | |
tap["attn", i], tap["mlp", i] |
cache["blocks.{i-1}.hook_attn_out"], cache["blocks.{i-1}.hook_mlp_out"] |
run | |
tap["heads", i] |
cache["blocks.{i-1}.attn.hook_z"]: per head, the input of the output projection |
run | |
attention.patterns |
cache["blocks.{i-1}.attn.hook_pattern"] |
run | |
lens.decode |
cache.logit_lens(), over cache.accumulated_resid(...) |
read | |
lens.attribute, summed over the heads of a block |
cache.decompose_resid(pos_slice=-1, return_labels=True) and cache.logit_attrs(stack, tokens=..., incorrect_tokens=..., pos_slice=-1) |
run, with a caveat below | |
Tap(model, edit=...), patch.run, steer.steered |
bridge.run_with_hooks(tokens, fwd_hooks=[(name, fn)]), where fn(activation, hook) returns the activation and its second parameter must be called hook |
run, on the stream at one place | |
| the same, without names of its own | nnsight 0.8.3 | model = TransformersModel(name, task='text-generation', ...); inside with model.trace(prompt): read model.transformer.h[-1].output.save(), write model.transformer.h[0].output[:] = 0, and take model.lm_head.output.save(). These are GPT-2's paths, the quick start's example. |
read |
probe.fit |
none named here | a ridge or logistic regression on recorded activations | neither |
sae.SAE, encode, decode |
SAELens 6.56.1 | sae = SAE.from_pretrained(release=..., sae_id=...), sae.encode(activations), sae.decode(features) |
read |
sae.train |
LanguageModelSAETrainingRunner(LanguageModelSAERunnerConfig(..., sae=TopKTrainingSAEConfig(k=..., d_in=..., d_sae=...))).run() |
read | |
attribute.through |
circuit-tracer | circuit-tracer attribute --prompt ... --transcoder_set ... with --graph_output_path or --slug and --graph_file_dir: a graph of the direct effects among the features of transcoders, error nodes, tokens and logits |
read |
minllm.pretrained.load_model |
transformers 5.24.0 | AutoModelForCausalLM.from_pretrained(folder) |
run |
Five of the six TransformerLens names in the table are its older ones,
which it keeps as aliases; attn.hook_pattern is already the canonical
one. Its Model Structure page asks new code to use the canonical names:
embed.hook_out, blocks.{i}.hook_out, blocks.{i}.attn.hook_out,
blocks.{i}.mlp.hook_out and blocks.{i}.attn.o.hook_in. On a model as
wrapped, a hook function is given the canonical name in hook.name. In
compatibility mode it was given the alias for two of the four names tried
here.
What the documentation adds to these calls:
- TransformerLens says it loads "15,000+ open source language models
across 140+ architecture families". Its registry file lists 15,753
models, 1,204 of them verified, and among those
Qwen/Qwen2.5-0.5B-Instruct. A wrapped model keeps the published weights.bridge.enable_compatibility_mode()folds the norms into the weights, centers them and moves the attention's value biases into its output bias, as the removed class did by default. - nnsight wraps a model without renaming anything, so the paths depend
on the model. The open model, as transformers builds it, has its blocks
under
model.layersand notransformer.h. A value read inside a trace must be saved with.save()to exist afterwards. Its quick start warns that with transformers 5 a block returns a bare tensor, while an attention module still returns a tuple, so that an index[0]copied from older code silently takes the first prompt of the batch. - SAELens describes its dictionaries as standard PyTorch modules that work with activations from any source. It lists BatchTopK, JumpReLU and Matryoshka BatchTopK as the state of the art, and the plain top-k dictionary of step 5 among the legacy ones.
- circuit-tracer implements the attribution graphs of Ameisen et al.
(2025) with transcoders, which stand in for the MLPs, one per layer or
across layers. Step 6 used a dictionary on the stream and a gradient
between two runs. Its README lists transcoders for certain sizes of
Gemma 2 and 3, Llama 3.1 and 3.2, Qwen 3 and GPT-OSS. The open model of
step 8 is not among them. It pins transformers below 5 and by default
builds on the removed
HookedTransformer, so it needs an environment of its own, apart from the versions in this table.
One of them, run #
uv run --with transformer-lens==4.4.0 --with transformers==5.24.0 scripts/stack.py
The library is not a dependency of this repository. --with lays it over
the project for one command, here at the versions the record was made
with. The script loads the open model from the same files by minllm's
loader and by transformers, compares the two, and then wraps the
transformers model in a TransformerBridge. Both get the same token ids:
the first sixteen of the 132 clean prompts of the first sentence pattern,
in which the first name is Lily in eleven and Tom in five.
| Compared | Largest absolute difference, any entry | Largest absolute entry |
|---|---|---|
| logits: minllm's loader and transformers, before wrapping | 0.00008 | 21.4 |
| logits: the library and minterp | 0.00008 | 21.4 |
| logits: the library and transformers before wrapping | 0.00008 | 21.4 |
| the embedding | 0 | 0.07 |
| the stream after blocks 1, 12 and 24 | 0.0002, after block 24 | 40 after block 24; 1,394 after block 12, at the first token (12 elsewhere) |
| what the attention and the MLP of those blocks write | 0.0001 | 63 |
| the heads of those blocks, at the input of the output projection | 0.00002 | 6 |
| the attention weights of those blocks | 0.000007 | 1 |
Wrapping changes the transformers model in place. The library puts its own attention in place of transformers', computed from transformers' projection layers, so that it can expose the attention weights: its code says it "reimplements attention step-by-step". One pass of the unwrapped model calls transformers' 24 attention modules 24 times, and one pass of the wrapped model calls them 0 times. The MLPs that run are still transformers' own, 24 calls. The logits move by 0.00008. So what agrees here is minllm's forward pass with transformers' modules and the library's attention, and the mapping of the names.
Two interventions give the same answer both ways, as a mean over the
sixteen prompts. With the flipped run's stream put into the clean run at
the acting name after block 12, the logit difference goes from +6.552 to
−7.431 by patch.run and to −7.431 by run_with_hooks. With half of the
difference between the two runs' streams at that place added there
instead, one vector per prompt, it is +0.087 by steer.steered and +0.087
by the library.
One helper needed care. cache.logit_attrs splits a logit difference
over the embedding and every block's attention and MLP: 49 parts here.
Step 1's lens.attribute splits each attention further, by head, 361
parts in all, and the script sums its heads within a block to compare. Those 49
sums add up to the logit difference, +6.552. The library's 49 parts sum to
+0.793 on the weights as published.
The helper divides each part by the final norm's scale and leaves out the norm's learned gain. That gain runs from −0.01 to 16.8 over the dimensions, so the result is not a rescaling of the right one: single parts are off by up to 5.04 logits. With the stack multiplied by the gain before the helper is called, its parts sum to +6.552 and agree with minterp's part by part, within 0.00002. In compatibility mode, where the gain is folded into the unembedding, the helper is right as it stands for a difference of two logits: +6.552, and within 0.00002 part by part.
So the helper assumes folded weights, and the wrapped model does not fold them unless asked. The library documents this, though not on the helper. Its compatibility-mode page lists direct logit attribution among the analyses that need the mode: "raw HF weights produce different (wrong) attributions". The helper's own docstring is silent, and nothing is printed when it runs on unfolded weights. The check that caught it here is the one step 1 used on its own code: the parts must add up to the whole.
The mode changes other numbers. It also centers the unembedding. The logits then differ from the published model's by up to 5.33. At each position of each prompt the difference is one constant over the whole vocabulary: the mode subtracts the mean of the published logits there. That mean is negative, between −5.33 and −0.25 here and −4.19 to −3.06 at the last position, so the mode's logits are higher by as much. The log-probabilities agree within 0.0001, and a difference of two logits is unchanged. For one token's logit the helper's parts sum to the centered logit: for the right name, as a mean over the sixteen prompts, 22.48 where the published logit is 18.91.
And the names of the table were compared on the published weights. In the
mode hook_z of block 12 differs from tap["heads", 12] by up to 1.3,
where its largest entry is 2.9. The difference is the same vector at every
prompt and position, and as large as that block's value bias, which the
mode has moved into the output bias. What the attention and the MLP write
agrees as before. In the mode the library computes the MLPs itself as
well: transformers' own are called 0 times.
What this does not show #
- That the other libraries give these numbers. Only TransformerLens was run. The rows for nnsight, SAELens and circuit-tracer are what their documentation says.
- A fully independent implementation. As wrapped, the library runs transformers' own blocks, norms and MLPs, and computes the attention itself from transformers' weights. The agreement is between minllm's forward pass and that mixture, plus the mapping of names.
- Which library is used most, or which is best for what.
- Tomorrow's versions. The calls were read on 2026-11-21. Three versions are in the record; the others are in the table. Two documentation sites state no version, and the command above is pinned because the newest transformers was published on the day of the record.
- More than a corner of the model. One model, the first sixteen prompts of one sentence pattern with two first names, three of 24 blocks, the CPU in 32-bit floats. Only the stream at one place was edited through the library: no attention, MLP or head, and no attribution per head.
- Compatibility mode beyond three checks. The attribution sum, the logits and the pieces after block 12 were looked at there.
- The whole of step 8 through a library. It trains no dictionary and attributes nothing through latents.
- Text passed as a string. The script passes token ids. For this
model the library's tokenizer adds no first token to a string, while
this repository opens every prompt with
<|endoftext|>.
Exercises #
- Redo the position-by-depth patch of step 8 on the open model with
run_with_hooks, on the same prompts asruns/patch/patch-qwen.json, and compare. Depth 0 ishook_embed. - Write the same patch with nnsight, in a file or a notebook cell and not
piped to the interpreter, since a trace reads its own source. The open
model's blocks are
model.model.layers[i]. - Train the dictionary of step 5 with SAELens on the story model's stream
after block 4, once as top-k and once as BatchTopK. The runner in the
table takes a model that TransformerLens or transformers can load, and
the story model is neither: SAELens's documentation (Extending SAELens)
points to
SAETrainerwith your own source of activations. Does the shared latent of step 6 appear in both? - Split the logit difference by head through the library:
bridge.set_use_attn_result(True), thencache.get_full_resid_decomposition(expand_neurons=False, pos_slice=-1, return_labels=True)andcache.logit_attrson that stack multiplied by the gain. Compare its head parts withlens.attribute's. - In an environment of its own, from a clone of circuit-tracer, pick a model it has transcoders for and trace the prompt of step 1. Some of those models need an access token. Is there a feature for the kind of name?
References #
Documentation, as read on 2026-11-21:
- PyTorch 2.15, torch.nn.Module:
register_forward_hookandregister_forward_pre_hook. - TransformerLens 4.4.0: the README, Migrating to TransformerLens 4.0, Migrating to TransformerLens 3, TransformerBridge Model Structure for the hook names, and Compatibility Mode.
- nnsight 0.8.3: the quick start.
- SAELens 6.56.1: the documentation, Training SAEs and Extending SAELens.
- circuit-tracer, main branch: the README.
Papers:
- Fiotto-Kaufman et al. (2024), NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internals.
- Ameisen et al. (2025), Circuit Tracing: Revealing Computational Graphs in Language Models.