How LLMs actually work
The one heavy-theory stretch of the course — and the payoff is huge. Understand the four tricks that turn “autocomplete on steroids” into an agent, and you’ll understand every technique that follows.
Today is mostly reading, not doing — and it’s foundational, so some of you already know it. That’s fine: skim the familiar bits. But this is the ground everything else stands on, so if it’s new to you, slow down here.
01What an LLM really is
A large language model like GPT is designed to do one thing: predict what comes next. You give it a sequence of input, and it tells you what’s likely to follow. It’s autocomplete on steroids — an enormous statistical pattern-matcher trained on vast amounts of text.
Two precise points people gloss over:
- The input isn’t words, it’s tokens — chunks of a few characters, often a whole word, often a piece of one.
- The output isn’t “the next word” — it’s a probability for every possible next token. Given “two plus two is”, the token for “four” gets a very high probability; “bananas” gets a tiny one.
LLM vs. AI application — keep them separate
Confusingly similar, importantly different:
| The LLM | The AI application | |
|---|---|---|
| Example | GPT | ChatGPT |
| What it is | A model that predicts the next token. Nothing else. | Software wrapped around calls to an LLM — with memory, web search, tools, a UI. |
| Others like it | — | The Cursor agent · Duolingo Max · Atlassian Rovo |
Ever since ChatGPT launched in 2022, we’ve been surprised that predictive text can seem intelligent. That apparent intelligence comes from four tricks written in software around the raw next-token engine.
02The four tricks
Trick 1 — The illusion of memory
Every call to an LLM is completely stateless. Call GPT with “I’m Ed” and it says “Hi, Ed.” Call it again — a brand-new call — with “Who am I?” and it answers “I don’t know.” It has no memory of the previous call.
So how does ChatGPT remember your name? A sneaky trick: every time, it sends the entire conversation so far back in. “I’m Ed / Hi Ed / Who am I?” all go in together — so the most likely next tokens become “You’re Ed, you just told me.” The memory is an illusion produced by re-sending everything each turn.
Trick 2 — Reasoning (thinking)
It started as a discovery: appending “think step by step” to a prompt produced better answers. That grew into training models to output their reasoning before the answer. It sounds hokey, but generating tokens that describe the approach genuinely improves the result.
“You toss two coins; one is heads. What’s the chance the other is tails?” Without reasoning, a model blurts “half.” Asked to think first, it spots the trick and answers two-thirds — because you didn’t say which coin is heads. Reasoning tokens are why modern models get these right.
Trick 3 — Tools
Realise that the tokens an LLM emits don’t have to be a text answer — they can be a request to take an action. In the prompt you say: “You may also reply with special tokens to search the web, use a calculator, or run Python. If you do, I’ll run it and call you again with the result.”
# You prefix your question with a rule: To use Python to answer the next question, just reply Python: and then a Python expression. # Your question: What is the square root of pi? # It replies not with a number, but with: Python: __import__('math').sqrt(math.pi)
The LLM never actually searches the web or runs code. It only generates tokens saying it wants to. It’s always your software that interprets those tokens and runs the calculator, then feeds the result back. That whole dance is “calling tools.”
Trick 4 — The loop
What’s better than calling an LLM once? Calling it in a loop: run it, ask “are you done?”, and if not, call it again… and again… until the goal is met. That simple idea is what lets agents achieve far bigger things than a single request ever could.
From autocomplete to agent
1 Illusion of memory · 2 Reasoning · 3 Tools · 4 Loops. Together they carry us from the ChatGPT of 2022 to today’s Cursor and Claude Code.
What an agent is
An LLM that runs tools in a loop to achieve a goal. Popularised by Simon Willison in late 2025 — and it ties trick 3 and trick 4 together.
03Defining an “agent”
“AI agent” has meant many things. The definition evolved:
| Era | Definition | Source |
|---|---|---|
| Early | “AI systems that can do work for you independently” — it can act, not just talk. | OpenAI & others (e.g. Operator / GPT Agent) |
| Early 2025 | “Systems where an LLM controls the workflow” — its output tokens orchestrate what happens next. | Hugging Face; Anthropic’s Building Effective Agents |
| Late 2025 → | “An LLM that runs tools in a loop to achieve a goal.” | Simon Willison (the prevailing definition) |
Yesterday’s Cursor game was a textbook agent: you gave it a goal (“a first-person shooter in a web page”), it clearly ran in a loop (files appearing one after another), and it used tools (writing files, running code). Goal + tools + loop = agent.
✓ Key takeaways
- An LLM predicts the probability of the next token — nothing more. It is stateless.
- Keep the LLM (GPT) separate from the application (ChatGPT) built around it.
- Four tricks make it feel intelligent: memory illusion, reasoning, tools, loops.
- An agent is an LLM that runs tools in a loop to achieve a goal.