Dog — a thing Prime knows

What it is

A dog is a domesticated carnivorous mammal of the family Canidae, kept as a pet or for work. It has a keen sense of smell, barks, wags its tail, and does not fly.
as it appears in the curriculum, taught as “狗是什么？” → “狗通常指家犬，是犬科的一种家养哺乳动物。”

Every sense, at once

Not a caption and a photograph: the same thing as a picture, a word, a voice, a moving clip, a character, and a thought. Each card holds the real prompt and the real answer.

sees

“What is this?”

Dog.

a real 32×32 training photo, shown enlarged

hears

“What did you hear?”

Dog.

says

“How do you say “dog”?”

(a voice)

draws

“Draw a dog.”

(a 32×32 picture)

reads

“What character is this?”

“狗”

moves

“Make the dog move from right to left.”

The dog moves from right to left.

thinks

“Can it fly?” → No.

think: it is a dog · dogs do not fly

the thought is written first, then read back

What it said, version by version

Four models, four stages of the same child. Pick one and you see what it was — its size, its lessons, the stage it had reached — and then real photos it had never seen, each with the true label and with what it actually answered. The confusions are the interesting part.

parameters 165M
layers 12 × 768
steps 1,500
things it knows 10
curriculum v2
stage seeing & hearing

The first attempt: 12 layers, 768 wide, taught on 10 things with about 20 pictures each, seen over and over.

Taught on picture book 40 · more pictures 960 · heard names 8 · total 1,008

Each card: the true label on top, what Prime actually said underneath.

Random photos from the 256-thing test set — none of them seen in training. On these 6 it answered 3 .

parameters 339M
layers 16 × 1024
steps 2,000
things it knows 10
curriculum v2
stage seeing & hearing

Twice the size, and the same 10 things shown with a different picture almost every time.

Taught on picture book 40 · more pictures 960 · heard names 8 · total 1,008

Each card: the true label on top, what Prime actually said underneath.

Random photos from the 256-thing test set — none of them seen in training. On these 6 it answered 3 .

parameters 341M
layers 16 × 1024
steps 1,000
things it knows 10
curriculum v2
stage seeing & hearing
borrowed eye DINOv2 · 0.5

Its pictures are pulled toward DINOv2, a frozen vision model that has seen far more of the world.

Taught on picture book 40 · more pictures 960 · heard names 8 · total 1,008

Each card: the true label on top, what Prime actually said underneath.

Random photos from the 256-thing test set — none of them seen in training. On these 6 it answered 4 .

parameters 341M
layers 16 × 1024
steps 8,700
things it knows 256
curriculum v3
stage knowledge & reasoning
borrowed eye DINOv2 · 0.5

The full curriculum: 256 things, roughly 250× the pictures, and four stages through to knowledge and reasoning.

Taught on heard names 37 · heard sounds 64 · more pictures 7,950 · picture book 100 · total 8,151

Each card: the true label on top, what Prime actually said underneath.

Random photos from the 256-thing test set — none of them seen in training. On these 6 it answered 2 .

Where this thing lives inside it

A matrix can be read as a graph: each row is a node, each non-zero entry an edge. Read that way, the weight matrices are a wiring diagram, and the units that light up for one thing are its neurons. So we opened Prime and looked for where dog actually sits.

A straight-line probe reads cat and dog out of each layer's picture tokens. Chance is 50%. It climbs, then peaks at layer 12 — that is where the two are most distinct inside the model.

Then we asked which single units carry it. These are the top FFN units of each layer, ranked by how far apart their average response to cat and to dog sits, measured in pooled standard deviations. Layer 7, unit 824 stands out: it fires on cat and stays quiet on dog, by more than a full standard deviation.

Where	Which unit	on cat	on dog	selectivity
layer 7	ffn.gate_proj · unit 824	+1.37	+0.35	1.21
layer 7	ffn.gate_proj · unit 2778	-1.17	-0.61	0.89
layer 7	ffn.gate_proj · unit 2051	-1.50	-0.94	0.87
layer 12	ffn.gate_proj · unit 1752	-0.41	-0.86	0.85
layer 12	ffn.gate_proj · unit 243	-0.89	-0.41	0.85
layer 12	ffn.gate_proj · unit 953	+0.50	-0.01	0.84
layer 16	ffn.gate_proj · unit 2633	+0.50	-0.04	1.09
layer 16	ffn.gate_proj · unit 662	+0.74	+0.32	1.04
layer 16	ffn.gate_proj · unit 1061	+0.29	-0.15	0.98

Here are those neurons drawn as what the matrices say they are: a graph. The lines are not drawn by hand — every one is a weight read out of the model.

Each dot is one feed-forward neuron — the 16 most cat-or-dog-sensitive units from each of the 16 layers, 256 in all. Each line is a real weight between two of them, read straight out of the matrices: the next layer's gate row times this layer's down column. Only the strongest fifth are drawn, 780. Clay leans dog, grey leans the other way; size is how strongly. At the centre is layer 15 · unit 575, the neuron that separates the two most sharply. Each ring outward is one layer further from it, up or down the network, and within a ring, neurons sit next to the ones they are wired to. The farther out, the fainter. Hover a dot for its numbers.

Now the causal test. Zero out one part at a time and re-run the exam. Left bar of each pair is attention, right is the feed-forward block; the shaded band is three standard errors, so anything inside it is noise. Only one part matters: layer 16 FFN — turn it off and the score falls from 20.7% to 10.0%. The rest, including the layer where the probe peaked, can be cut without the answers changing.

Two things fall out of this, and they disagree. Where the concept is readable is layer 12 — a probe over all 256 things gets 32% there, against 73% for the frozen DINOv2 encoder. But where it is used is layer 16: cut that feed-forward block and the exam score drops from 20.7% to 10.0%, while cutting the peak layer changes nothing. The representation is not the decision. And raw 32×32 pixels alone carry almost nothing (5%), so the model is not merely storing the picture — it built something, and then built a way to use it somewhere else.