A book in progress · 2026

Learning Theory

A theory of how much one observation teaches, the architecture that follows from it, and the experiments that test it — on a model we are teaching the way you would teach a child.

Every claim in the book is tagged: definition, theorem (proved, with a source), conjecture (to be tested), observation (measured, with sample size) or design (built or planned, not yet validated). Chapters are revised as experiments come in.

Part I

Theory

What a lesson is, and the laws it obeys.

01What an observation is worthHartley, Shannon, Fano — and bits measured on a child-like model.Read02The definition of a lessonSurprise is not learning. A lesson is the surprise it removes from what comes next.Read03Six lawsNovelty, ceiling, channel, transfer, prior, order — and how sure we are of each.Read
04Where the bits goMemorized bits and learned bits: 3.32 on the picture book, 0.03 on a new picture.In writing
05Only what is predicted is learnedThe channel law, and why every modality has to be generated.In writing
06Where to look when bandwidth is shortRate–distortion and the token budget.In writing

Part II

The talking blind

How today’s models see, and why they cannot learn what they do not predict.

07The talking blindPictures cut into patches and poured into a stream of words.In writing
08A one-time inputWhy multi-turn edits drift: decode, re-encode, lose.In writing
09The tyranny of textEverything flattened into one discrete sequence.In writing

Part III

The learner

Prime, piece by piece: one way to generate everything, eyes that choose where to look, a canvas that stays, memory.

10Structure first, then contentThe block protocol: headers, budgets, END and CONT.In writing
11One way to generate everythingFlow matching across every modality.In writing
12Words also flowText from noise: shift 9, cross-entropy on x̂₁, clamping — 12/50 to 35/50.In writing
13Pictures, video and soundPatches, frames and mel; silence is a valid sound.In writing
14Senses on one clockThree-axis RoPE and joint blocks that share one t.In writing
15One sequence, two pathsClean and noisy streams; training equals inference to 1e-6.In writing
16Every part of the trunkadaLN-single, sandwich norm, gates, QK-norm, mode layers — and where each came from.In writing
17Eyes I: read the whole firstA text encoder conditioned on the full context.Needs an experiment
18Eyes II: spend tokens where it mattersQuadformer: the question decides where to look closely.Needs an experiment
19The canvas stays in handGenerated latents stay in context; editing without decoding.In writing
20Circle it, change only thatFrom a circled region to the patches that regenerate.Needs an experiment
21Hands: using toolsActions and their results, flowed together.Needs an experiment
22Fast memory: registersFast weights carry what attention cannot see: 4/4 against 1/4.In writing
23Dreaming and recallingThe intent bank, dream and think, and three ways it collapsed.In writing
24Borrowed eyesSelf-Flow, REPA, DINOv2 against DINOv3, and a negative result for MCR².In writing

Part IV

The curriculum

From the world to samples, and the order in which to show them.

25From the world to samplesSources, licences, voices, fonts — append-only.In writing
26What to show nextVariety, replay, at most four repeats, sampling by lesson.In writing
27Growing upDevelopmental order, v1 to v3.In writing
28Knowledge and languageWikidata, encyclopaedias and graded reading, without inventing facts.In writing

Part V

Training

Making it run, and reading the gradients.

29Running on eight cardsMulti-GPU, bf16, checkpointing, a 35% speed-up from caching indices.In writing
30What the gradients sayAlways-on clipping, conflicting losses, Muon.Needs an experiment

Part VI

Evidence

How to examine a child, what we measured, and what we still do not know.

31How to examine a childUnseen data, generated answers, sample sizes.In writing
32Lessons, measuredFour stages, no forgetting, the whole curriculum in bits.In writing
33How it recognises a catInteraction diagnostics, inside a memorizing model.Needs an experiment
34What we do not know yetWhich laws failed, which are open.Needs an experiment