Deep Dive into LLMs like ChatGPT

"even though the internet is very very large we're working with text and we're also filtering it aggressively"

The lecture opens on a public dataset (FineWeb) rather than a diagram: before anything else, a model is a pile of filtered web pages. Watch the stages — blocklists, HTML stripped to prose, non-English dropped, duplicates and personal information removed. Pause on the number 44 terabytes: the readable internet, strained hard, fits on one drive. Every judgment there becomes a property of the model.

"I really encourage you to think of these not as numbers but as unique IDs"

Text must become a sequence of symbols from a fixed vocabulary. Karpathy walks from bits to bytes to "byte pair encoding," which merges common pairs until the vocabulary holds about 100,000 chunks. Follow the live demo where "hello world" splits differently by capitalization and spacing. The highest-leverage stretch of the first hour: it silently explains the failures two hours later. The model never sees letters. It sees chunk IDs.

"think of these parameters as kind of like knobs on a DJ set"

The deep end; Karpathy says himself the mathematics is not the important part. Hold onto this: the network takes a window of tokens, outputs a probability for every possible next token, and gets nudged so the true one grows likelier — billions of times over. Generation reverses it: sample, append, repeat, which is why one prompt gives different answers. Skim the rest.

"a glorified autocomplete it's a very very expensive autocomplete of what comes next"

The most memorable half hour of sitting one. Karpathy opens a raw pretrained model and shows it is not an assistant at all: it wanders, it invents. Then three revealing tricks. It recites a Wikipedia article from memory. Asked about an election past its training cutoff, it generates two confident parallel universes. Given ten example translations, it infers the pattern and continues it — learning inside the prompt.

"you're not talking to a magical AI you're talking to an average labeler"

Same algorithm, different data. The internet is thrown out and replaced with a curated set of conversations, most now drafted with model help but originating in human writing. Notice how humble the machinery is: hired labelers working from written instructions about being helpful, truthful, and harmless. The persona you talk to is a statistical average of those people following those instructions. This deflation is the lecture's emotional center; sit with it.

"the model basically doesn't know and it's just imitating the format of the answer"

Do not skim this. Hallucination is explained not as a bug but as an inevitability: the training data is full of confidently answered questions, so the model learned that the form of an answer is confidence. Two fixes follow — find the edge of what it knows and teach it to say so, then give it a search tool. That yields the lecture's central rule: knowledge in the weights is vague recollection, text in the context window is working memory. Paste in the chapter you want summarized rather than trusting recall.

"these are not for you these are for the model"

The most useful twelve minutes in the video. Each token gets a small, roughly fixed amount of computation, so a model asked to answer immediately must cram the whole problem into one step — and past a certain difficulty it fails. Karpathy proves it live by forbidding intermediate steps and watching a word problem break. Anything after the answer is retroactive rationalization; the answer is already locked in. This is why step-by-step prompting works: architecture, not style.

"the models don't see characters they see tokens"

The famous embarrassments, now explained. Counting dots fails because counting demands more computation than one token allows. Spelling fails because letters are welded inside chunks the model cannot look into — the strawberry problem, and the payoff for that tokenization segment ninety minutes earlier. Then a genuine mystery: 9.11 judged larger than 9.9, apparently because neurons associated with Bible verses light up. He cannot fully explain it — the honest note the segment needs.

"we really want the llm to discover the token sequences that work for it"

The analogy organizing the last third. A textbook contains exposition (pretraining), worked solutions from an expert (supervised fine-tuning), and practice problems with only an answer key (reinforcement learning). The argument for that third stage is worth pausing on: a labeler writing an ideal solution cannot know which path is easy for the model, whose cognition is not ours. So let it try thousands of solutions and keep what works.

"the model is discovering ways to think it's learning what I like to call cognitive strategies"

The high point of the lecture, filmed days after the paper landed. Accuracy climbs — but so does answer length, because the model spontaneously starts backtracking, re-checking, trying second approaches. Nobody wrote those behaviors in; they emerged from being rewarded for correct answers. Karpathy runs a reasoning model live so you can watch it argue with itself, then closes on AlphaGo's move 37 — a play no human would make — as the shape of going beyond imitation.

"we are doing reinforcement learning not with respect to humans and actual human judgment"

For creative tasks there is no answer key, so a second network is trained to imitate human preference rankings and the model optimizes against that. It works, for a good reason: ranking five jokes is far easier than writing one. Then the trap. Run the optimization long enough and it finds nonsense the scorer adores. There is no fix, only a stopping rule — improve for a few hundred steps, then ship. His verdict: fine-tuning wearing reinforcement learning's clothes.

"the models are incredibly good across so many different disciplines but then fail randomly"

The closing image is the one to keep: capability like Swiss cheese, brilliant almost everywhere and arbitrarily absent in holes you cannot predict. Then the forecast — native audio and images, agents running for hours under supervision, models that keep learning after deployment (nothing today does). Finish with the advice, unglamorous and correct: use these as tools, take the first draft, check the work, own the result.