How should I look at a model? Five questions · Yixin

First, why I’m writing this.

Lately I’ve been re-organizing what I studied, and I want to step back and look at it from one level up. The push came from someone I’ve always respected, who gave me a piece of advice: learning is like collecting apples. Once you’ve gathered enough, you sort them into baskets. And once you have baskets, the dots start to connect into lines.

This piece is one product of that experiment, and the first topic in my methodology: models.

I studied math and law as an undergraduate, then did a master’s in machine learning. The word “model” shows up in all three, and it means something different each time. In math, it’s a set of assumptions plus the proofs that follow. In law, it’s a set of rules written in advance. In machine learning, it’s a function learned from data. I put the apples from all three into one basket, and the basket’s name is the five questions I ask about any model. Think of this piece as a note to my future self: a way of thinking I can run again.

Here’s the framework:

#	Keyword	What to ask
1️⃣	Purpose	What is it for?
2️⃣	Trade-offs	What did it throw away? Does that matter?
3️⃣	Assumptions	What does it stand on? Who chose that ground?
4️⃣	Complexity	How detailed is it? Is there evidence for that level of detail?
5️⃣	Correction	When it’s wrong, who notices, and how does it get fixed?

The order is not random. It’s a chain. Purpose decides the trade-offs. The most important trade-off is the assumptions. Once the assumptions are solid, you can ask how detailed to be. And the last question is the safety net. The first four are prep work. The fifth is the master switch.

First, one thing to get straight: a model is a stand-in for the world, not a description of it

A lot of arguments get stuck right here. I’d rather not get pulled into a fight over academic definitions.

My working definition is plain: a model is a simpler, workable thing that stands in for something too complex to fully grasp, built for a specific purpose.

Two keywords: stand in and purpose.

“Stand in” means it will never be complete. In Sylvie and Bruno Concluded (1893), Lewis Carroll has a character called Mein Herr tell a story about a country that built a map at the scale of a mile to the mile. It couldn’t be used. The farmers objected: they said it would cover the whole country and shut out the sunlight. So now the country itself serves as its own map, and he says it does nearly as well.

Borges wrote an even shorter version in 1946, and it makes the same point: a perfect map is a useless map.

Alfred Korzybski said “the map is not the territory” in 1931. People forget the second half of the idea: a map is useful because its structure resembles the territory. Resembles, not equals.

Which leads to this. “Is this model correct?” is a bad question. It never is. The better questions are: for what purpose is it good enough, where is it wrong, and what happens when it’s wrong? The five questions below are just that sentence, taken apart.

1️⃣ Purpose: what is it for?

In one line: change the purpose or the setting, and the same model can flip from good to bad.

Newtonian mechanics has been known to be wrong for a long time. It still works for building bridges and computing braking distances. But put it in GPS, and if the satellite clocks don’t get the relativity correction, the position error piles up day after day. Same model. In one use it’s a good tool. In another, it’s an accident waiting to happen.

Maps work the same way. The Mercator projection preserves angles, so a sailor can follow a straight line. The cost is that areas near the poles get stretched, and Greenland ends up looking about as big as Africa. A navigator doesn’t care about area. A geography student does.

I ran into this myself at work, with a model I’ll call the user flow map. It worked well inside China’s media ad-traffic system. Moved to Europe, it didn’t work. The model was the same, and the goal looked the same. What changed was the setting around it. I’ll explain why in a separate piece under Selected Work, where I’ll walk through how this model played out in practice. Consider this a teaser.

In machine learning, “purpose” usually lands on two concrete things: the loss function and the evaluation metric. You get what you optimize. If you haven’t stated the purpose, you’ve handed the choice to the default settings.

👉 Rules of thumb

Before you use a model, write one sentence: “What decision am I using this for?” If you can’t, don’t evaluate it yet.
If the use or the market changes, treat it as a new model. That’s why I need a method: one that finds repeatable patterns in new settings and new tasks. When you know nothing yet, start with the simplest method that works, and run the five questions again.

2️⃣ Trade-offs: what did it throw away? Does that matter?

In one line: a model is useful not because it’s wrong less often, but because it’s wrong in places that don’t get in the way.

You’ve probably heard George Box’s line: all models are wrong, but some are useful. Most people remember only half of it. He wrote the first half in 1976, and added the second half eleven years later with a co-author. What I care about more are the two pieces of advice he gave along the way:

Don’t try to build the “correct” model by piling on detail. More detail won’t make it correct.
Worry selectively. If there’s a tiger outside, stop worrying about the mice.

That line often gets used as an excuse: “It’s all wrong anyway, so mine can be sloppy.” He meant the opposite. He wants you to find out where the tiger is first.

The three fields throw things away in different ways. The difference is mostly about whether what’s thrown away is said out loud:

Field	Approach
Math	remove things on purpose, and list your assumptions in the open. Ideal gases and frictionless slopes don’t exist in the real world, but each one says clearly what it dropped. (At least in principle. In practice there are hidden assumptions too.)
Law	use a legal fiction. Everyone knows it isn’t true, but it keeps the system running. The biggest example is the company. A company is not a person, yet the law lets it sign contracts, own property and be sued like one.
Machine learning	it doesn’t have to look like the world. In 2001, Leo Breiman split statistical modeling into “two cultures.” One assumes there is a data-generating mechanism you can write down, and tests the model by asking whether the assumptions hold. The other treats the mechanism as a black box and only asks whether the predictions are good. The second kind is wrong in a deeper way. It never tried to look like the world.

Machine learning adds one more layer. It isn’t drawn from the world. It’s drawn from the training data, and the training data is already a map of the world. Doing well on a test set only tells you it works in places that look like the training data. When user behavior shifts, market rules change, or sensors get swapped, the road has changed but the map hasn’t. That’s called distribution shift. The model won’t hesitate. It will confidently point you down a road that no longer exists.

👉 Rules of thumb

For every model, write a list of “things it threw away.” If you can’t write it, you haven’t understood the model yet.
Then ask, item by item: could this become important in my setting?

3️⃣ Assumptions: what does it stand on? Who chose that ground?

In one line: assumptions aren’t a flaw. They’re the starting condition. What’s dangerous are the assumptions nobody can see.

Start with math. Euclid’s parallel postulate is long and clumsy, and it doesn’t look like an “obviously true” axiom. For a very long time, people kept trying to prove it from the other axioms. No one succeeded. In the 19th century, Lobachevsky, Bolyai and Riemann gave up on proving it and swapped it out. The result was a new geometry that was just as consistent. Riemann’s version later became the language of general relativity.

So an axiom isn’t something proven true. It’s something chosen as a starting point. Inside an axiom system, you can’t overturn an axiom, because the moment you do, you’re already in a different system. To change one, you swap the whole system.

Law treats assumptions differently. It uses a presumption: when the facts aren’t clear, assume things a certain way, and state in advance who has to bring the evidence to overturn it. (The legal examples below are based on Chinese legal practice.)

Two examples:

Presumption of innocence: trials will get things wrong sometimes, so decide up front that it’s better to let a guilty person go than to convict an innocent one. In effect, it allocates the cost of being wrong.
Declaration of death: when someone has been missing for a long time, a court can presume the person is dead. If the person turns up, the declaration can be revoked, and the rules for what happens to the marriage, adoptions and property are already written down.

Note that a presumption is not the same as the legal fiction we saw earlier. A presumption can be rebutted. A fiction cannot. Nobody can stand up in court and “prove the company isn’t really a person,” because that setup was never meant to describe the world in the first place.

Assumptions in machine learning come in two kinds:

Priors: beliefs held before seeing the data, updated as the data comes in.
Inductive bias: built into the structure of the model. For example, a convolutional neural network assumes that a local pattern in an image means roughly the same thing wherever it appears.

David Wolpert’s 1996 “no free lunch” theorem says, roughly: if you assume nothing about the problem, all learning algorithms perform the same on average. No assumptions, no generalization. It’s close to a mathematical version of Hume’s problem of induction. (Strictly speaking the theorem has preconditions. I’m only borrowing its meaning here.)

Put the three ways of handling assumptions side by side:

Approach	How it works
Axioms	laid out on the table, can’t be overturned inside the system, can only be replaced as a whole.
Presumptions	can be overturned, and it’s decided in advance who has to bring evidence, and what kind.
Inductive bias	often hidden in the architecture and data choices, and usually nobody is responsible for challenging it.

An assumption that nobody can see, and nobody is responsible for questioning, is really just an invisible axiom.

👉 Rules of thumb

Write your assumptions as a list. Next to each one, note two things: who chose it, and what evidence would make me drop it.
If you can’t fill in the second one, treat that assumption as an invisible axiom, and say it out loud in your report.

4️⃣ Complexity: how detailed is it? Is there evidence for that detail?

In one line: Occam’s razor doesn’t say “simpler is truer.” It says “don’t pay for complexity you don’t need.”

I’m only covering the part that touches models here. Occam’s razor deserves its own piece, and I’ll write one later.

“Entities should not be multiplied beyond necessity.” The line is credited to the 14th-century friar William of Ockham, although scholars apparently haven’t found those exact words in his writing. It’s more often misread as “the simpler explanation is more likely true.” The razor doesn’t say that. It says complexity costs something, and every extra bit should come with a reason.

Math and statistics gave this idea a quantitative form: the bias-variance trade-off.

Prediction error ≈ bias² + variance + noise

If the model is too simple, bias is high and it misses real patterns (underfitting). If it’s too complex, variance is high and it memorizes noise as if it were pattern (overfitting). The best model sits somewhere in between. That’s the classic U-shaped curve from the textbooks: model complexity on the x-axis, training error falling all the way down, and test error falling first, then rising.

Schematic of the classical U-shaped curve and the double descent curve

📈 Figure note / sources: The figure above is a schematic drawn for this article (conceptual curves, not experimental data). It is not a reproduction. Concept: Belkin, Hsu, Ma & Mandal (2019), Reconciling modern machine-learning practice and the classical bias–variance trade-off, PNAS 116(32), Figure 1(a) is the classical U-shaped risk curve, and Figure 1(b) is the double descent curve. Full text: https://arxiv.org/abs/1812.11118

The textbook version: Hastie, Tibshirani & Friedman, The Elements of Statistical Learning (2nd ed.), Chapter 2, the section “Model Selection and the Bias–Variance Tradeoff”, which has the test / training error vs. model complexity plot.

Then deep learning showed up and broke the picture. Many neural networks have more parameters than training samples. They can memorize the training data perfectly. By the textbook they should overfit badly, and yet they do quite well on new data. Belkin and colleagues drew this as the “double descent” curve: as the model grows, test error falls, then rises, peaks right where the model can just barely fit all the training data, and then falls again. The textbook U-shape turns out to be only the first half of the story.

Does that overturn the razor? My take is no. One explanation is that among the huge number of solutions that fit the training data perfectly, training tends to prefer the “smoother” one. The razor is still there. What changed is the meaning of “simple,” from “few parts” to “simple shape.”

Law has argued over the same question for a long time. It’s called rules vs. standards (Louis Kaplow). “Speed limit 120” is a rule: fixed in advance, expensive to write, but cheap for every driver and every judge to use afterwards. “Drive at a safe speed” is a standard: cheap to write, but someone has to put in effort to judge it every time. The cost of a rule is that it doesn’t fit individual cases: 110 in thick fog may not be safe, and 125 on an empty highway may not be dangerous. His conclusion: for things that happen often and look alike, rules work better. For things that are rare and different from each other, standards work better.

Looking at the same issue across three fields, what I take away is a way to judge. Choosing simple is a bet: you bet that the world is no more complicated than your model. Whether the bet is worth it depends on three things:

How much data or how many cases you have: the more you have, the more complexity you can support.
How similar the cases are: the more alike, the better a simple rule works.
Who pays if you lose the bet: the higher the cost, the more evidence you need for both the complexity and the simplicity.

👉 Rules of thumb

Every extra bit of complexity needs evidence: an improvement on the validation set, or a specific failure case. Pick one.
Neither “more complex, so better” nor “simpler, so more stable” counts as evidence.

5️⃣ Correction: when it's wrong, who notices, and how does it get fixed?

In one line: a good model isn’t just useful. It has already thought about what to do when it’s wrong.

With the first four questions out of the way, back to the master switch.

The three fields answer this question very differently:

Field	Approach
Math	put the assumptions on the table so you can see where it’s wrong. The fix is to swap in another model, and peers check the work.
Law	write “what if I’m wrong” right into the statute. Declaration of death is a good example: the conditions for revoking it, what happens to the marriage, to a child who was adopted in the meantime, and to property that was inherited, are all specified in advance.
Machine learning	often says nothing about this. For most models, you find out the map no longer matches the road only after a monitoring metric goes off.

If you take “correction” apart, it’s really three things:

Where it’s wrong can be said out loud: the assumptions are public, and what was thrown away is on record.
Who notices: peer review, the affected party filing a request, or a monitoring alert.
How it gets fixed, and who bears the cost: swap the model, revoke the ruling, or retrain.

You can see these three things very clearly in an agent.

The core of an agent is an LLM. It can break down tasks, call tools and write results. The LLM will be wrong sometimes, so what decides whether the agent can actually be deployed is the layer around the model, the harness (think of it as a seat belt for the agent). A harness answers exactly the three questions above:

The question	What the harness does
Where it’s wrong can be said out loud	every call and decision leaves a trace / log, so when something goes wrong you can replay it instead of guessing.
Who notices	continuous evals, monitoring and alerts, and human-in-the-loop on key steps (where a human should confirm, a human must confirm).
How it gets fixed	permission boundaries (what it can and can’t call), a rollback mechanism, and a process for changing the prompt or tools and re-evaluating.

An agent without a harness is the clearest case of “nobody is responsible for correction.” When it goes wrong, nobody knows, and nobody has the authority to stop it. An agent with a harness has answered question 5 up front. For the same LLM, having a harness or not is pretty much the difference between a demo and production.

To be honest, when I judge a model or an agent now, I first look at whether these three things were designed, and only then at how accurate it is. Accuracy can come from luck. A correction mechanism can’t.

👉 Rules of thumb

Before launch, write a one-page “how we admit we’re wrong” protocol: which metrics you watch, how much drift triggers retraining, who makes the call, and who cleans up when something goes wrong.
For an agent, that page is your harness design checklist. If you can’t write it, the work isn’t done yet.

One more look: how the 5 questions connect

#	Keyword	In one line
1️⃣	Purpose	what it’s drawn for.
2️⃣	Trade-offs	what it threw away, and whether that matters.
3️⃣	Assumptions	what ground it stands on, who chose it, and who can overturn it.
4️⃣	Complexity	how detailed it is, and where the evidence is.
5️⃣	Correction	who notices when it’s wrong, and how it gets fixed.

When someone hands you a model, a report or a conclusion, don’t rush to ask “is it right?” It isn’t. Run it through these five questions, and you’ll spot most of the traps early.

One last thing I haven’t figured out yet: is a model for making sense of the world, or for placing bets on it? Explaining the world and predicting it often get mixed together, but they may be two different jobs, and they tolerate being wrong in very different ways. I’ll leave that for the next piece.

Author’s note: personal views, for reference only.

Further reading

George Box, Science and Statistics (1976)
Alfred Korzybski, Science and Sanity (1933)
Lewis Carroll, Sylvie and Bruno Concluded (1893), chapter 11, “The Man in the Moon”
Leo Breiman, Statistical Modeling: The Two Cultures (2001)
David Wolpert, The Lack of A Priori Distinctions Between Learning Algorithms (1996)
Mikhail Belkin, Daniel Hsu, Siyuan Ma, Soumik Mandal, Reconciling modern machine-learning practice and the classical bias–variance trade-off (PNAS, 2019)
Louis Kaplow, Rules Versus Standards (1992)
Civil Code of the PRC, Articles 46–53 (declaration of missing and declaration of death)