From machine learning to generation
Empirical research in the social sciences usually proceeds from some prior understanding of the problem. A researcher has reasons for thinking a relationship exists (a thesis), collects data capable of testing it, and interprets the result in light of both the strength of the prior reasoning and the quality of the data. The account we accept depends partly on whether there is a plausible mechanism behind it.
Machine learning shifts the emphasis. Rather than specifying a relationship and testing it, you assemble a large quantity of data, split it into training and test sets, fit progressively better models to the training data, and accept the winner if it performs comparably on data it has never seen. The approach still embeds plenty of assumptions and choices at a procedural level (which features to use, which architecture, which objective to optimize, what counts as the target, how to sample, how to validate, etc.), but the results are not validated by any substantive understanding of the phenomenon being studied. A machine learning model can be trained to detect cancer on an x-ray without any understanding of or theory about x-rays or cancer.
Large language models are a particular kind of machine learning model, trained to do something that sounds trivial: predict the next token in a sequence. A token is usually a chunk of text—roughly a word or part of a word. Over and over again, the model is shown sequences of text and asked what comes next, it makes a guess and the parameters of the model are adjusted according to how right or wrong that guess was.
Why is predicting the next token useful? If you can predict the next token in a sequence, you can generate anything that can be represented as a sequence of tokens—which turns out to be most things, so prediction becomes generation. That insight underlies text and code generation, and related generative techniques, though not always the same architecture, now produce images, audio, and video as well.
Prediction, not truth
LLMs are optimized to predict text (actually, a lot of language models are now multimodal, but we can simplify the discussion by focusing on text). They are not optimized to tell the truth, as such. Nothing in the training process rewards accuracy in the sense of objective truth: it rewards producing the continuation that the training distribution makes likely. Most of the time the two coincide, because most text is not deliberately false and truly random errors average out.
When a model tells you the Declaration of Independence was signed in 1776, the statement is true, but not because the model is a truth machine or is bound by a sense of honor to only convey accurate information. If you trained a language model on everything written up to 1887, you might have an excellent model: fluent, coherent, well calibrated to its corpus, and full of assured explanations of how light propagates through the luminiferous ether (this is what people thought around the turn of the previous century). Your olde worlde language model’s account of the ether would be indistinguishable in tone and confidence from its account of the tides, a subject about which it would be entirely sensible.
The validity of a model’s output is therefore external rather than internal. This is why benchmarks are so important in AI right now. A benchmark is an attempt to import an external standard, and it establishes only what it was built to test.
Why now?
Four developments explain why systems that had been improving slowly for decades became suddenly, so capable.
First, backpropagation combined with deep architectures gave an efficient way to improve a network from one guess to the next.
Second, the transformer proved much better at drawing meaning from context, and much easier to run in parallel.
Third, and following directly from the second, “scaling laws”. Researchers established around 2020 that increases in model size and training data produced predictable improvements in performance. These scaling laws mattered enormously, because they converted an open-ended research problem into a capital expenditure decision. If you know that more compute buys more capability, you can justify spending on a scale that nobody would risk on a hunch.
Fourth, the labs found ways to make models better behaved. OpenAI’s method—reinforcement learning from human feedback—uses human preferences between model outputs to train the model toward responses people actually want. Not every LLM uses RLHF, but they all use some analogous process.
Data dependency
Like every machine learning system, a language model reflects its training data. Image models trained on material scraped from the internet reproduce the internet’s assumptions about which people hold which jobs. This is not because algorithms harbor prejudice. Math is not racist. But math applied to a record of a flawed society will reproduce those flaws, because it is math and not magic. A large share of the objections to AI are, at bottom, objections to the data. Or as my coauthor and I wrote in 2024, “We Are the AI Problem.”
← Previous: What Is AI? Definitions and History · Contents · Next: What Has Happened Since ChatGPT →