An Unverifiable WorldPreface: A World Without OraclesContents中文

Preface: A World Without Oracles

Before going to war, marrying, or founding a city, the ancient Greeks traveled to Delphi to consult the oracle. What made the oracle matter was not that it was always right. It was the promise that, before you acted, there was somewhere you could go for the answer. Two thousand years later, computer scientists borrowed the word. In their usage, an oracle is a black box: you hand it a problem you cannot solve, and it instantly returns the correct answer. Both oracles rest on the same fantasy, that you can find out whether you are right before you move.

This book is about the world after that fantasy breaks.

We almost never verify. We act, and then, sooner or later, we find out, or we never find out at all. "Everything can be checked" feels like the normal state of affairs because our earliest training comes from an unusually narrow class of tasks: arithmetic, sorting a list, checking a receipt. In those tasks the answer is close at hand, so we mistake them for the shape of the whole world. Step outside that narrow door and verification quickly becomes a luxury. You can verify seven times eight. You cannot verify, before you say "I do," that the marriage will last. You cannot verify before a release that the code has no bugs, or, before you commit yourself, that a theory is true, a company is healthy, or a decision is right. Most actions that matter are taken on unverified ground. The oracle stays silent, and you still have to move.

People usually respond to this in one of two ways: they lament it, or they pretend it away. Those who lament say that if nothing can be made certain, every judgment is a guess. Those who pretend build a false oracle: they crown some measurable number and act as if it were the unmeasurable truth. This book does neither. It asks a more interesting question. When there is no oracle, what do capable people actually do? Scientists, engineers, mathematicians, and those who govern all face this.

Follow that question across enough fields and you run into a surprising observation, the one this whole book grew out of: the reasons things cannot be verified differ wildly, yet the responses that work keep converging on the same small set.

That observation gives the book two layers. Keep it in mind, because everything later rests on it.

The first layer: the problem comes in different kinds. The sentence "I cannot check it" covers five structurally different situations. In some, no procedure for deciding can exist, even in principle (undecidable). In some, a procedure exists, but its cost explodes (intractable). In some, the state you would need to see is hidden from you (partially observable). In some, checking is possible in principle, but you lack the time, computing power, or samples (budget-constrained). And in some, an opponent is actively working to defeat your checks (adversarial). Treating these five as one is the most common mistake in this territory. Part I pulls them apart.

The second layer: the responses converge. Whatever form the problem takes, people who handle it well keep reaching for the same few things. They replace a target they cannot measure with a proxy they can. They prove a bound on a slice they can inspect. They spend expensive checks where those checks reveal the most. They bring in an outside judge. They shrink the blast radius of failure. They put a probability on the risk that remains. They move checking from before the fact to after it. And they combine several independent judgments so that no single error decides the outcome. This book calls these the eight moves, and argues that they come in four pairs. Part II visits four concrete settings and lets the moves appear in each field's own vocabulary. Part III then takes each move out of its setting, strips it down, names it, and lays all of them out in one cross-domain table. That table is the book's main contribution. Part IV asks why it is these moves and not others.

I owe the reader one caveat up front. Is this convergence a law, in the sense that something forces every bounded actor toward these moves? Or is it only a strong empirical pattern, something we keep seeing but cannot prove must be so? At present I have no evidence that it is a law. What this book offers is a clearly stated conjecture with clearly drawn limits, together with a shared vocabulary that connects many fields. It is not a theorem. Chapter 14 takes this question on directly.

That creates a loop I cannot escape, and should not try to. A book about acting when you cannot verify cannot verify its own central claim. So it can only do what it describes: state a calibrated belief, mark where the claim stops, invite people to refute it, and go ahead anyway. The book practices the methods it studies. If it is right, that is not a flaw. It is the only defensible way to write it.

One image to end on; the afterword will come back to it. A ship is changing course in heavy fog. The captain has charts, a compass, and estimates of the current. What she does not have is a way to see through the fog. Before she turns the wheel, she cannot confirm whether there is a reef ahead. The fog will not lift. The oracle will not come. But the ship cannot stop sailing because of that. This book is not about how to wait for the fog to clear. It is about how a good captain actually steers inside it.

Next chapter: 1. The Luxury of Verification →

References

Waypoints: 1. historical scientific judgment; 2. theoretically studied material; 3. how science progresses; 4. how to live in an unverifiable world. This section was checked source by source.

  1. H. A. Simon (1969). The Sciences of the Artificial. MIT Press. Google Books [2][4] Simon distinguishes natural science from "the sciences of the artificial" and argues that design is a discipline for coping with complex environments under bounded rationality. His ideas of near-decomposability, hierarchy, and satisficing provide a background for this book's central stance: actors do not verify everything; under limits of computation and information, they design responses that are good enough.
  2. F. H. Knight (1921). Risk, Uncertainty and Profit. Houghton Mifflin. Google Books [2] Knight draws the influential line between risk, where probabilities are known and measurable, and true uncertainty, where even the probability distribution is unavailable. He then attributes entrepreneurial profit to bearing the latter. The distinction is one conceptual source for this book's use of "unverifiability," reminding the reader to separate situations where a probability can be measured from those where even the probability cannot.
  3. N. N. Taleb (2007). The Black Swan: The Impact of the Highly Improbable. Random House. Google Books [2][4] Taleb argues that rare, hard-to-foresee, high-impact events dominate history and markets, while conventional bell-curve statistics systematically underestimate them. His diagnosis of prediction's limits matters here: when tail events cannot be verified in advance, changing one's exposure to surprise is often more useful than pursuing precise forecasts.
  4. W. C. Wimsatt (2007). Re-Engineering Philosophy for Limited Beings: Piecewise Approximations to Reality. Harvard University Press. doi:10.2307/j.ctv1pncnrh [2][3][4] Wimsatt argues that limited beings cannot possess complete truth. They rely on biased but useful heuristics, robustness analysis, and piecewise approximations to reality. This is a philosophical counterpart to the chapter's main claim and especially to the role of robustness and multiple independent routes of support.
  5. J. M. Keynes (1921). A Treatise on Probability. Macmillan. Google Books [2] Keynes understands probability as a logical relation between propositions: the rational degree of belief given evidence. He notes that many probabilities cannot be precisely numbered and may not even be comparable. His notion of the "weight" of evidence reminds us that when evidence is thin, quantified confidence may itself be unwarranted.
  6. L. J. Savage (1954). The Foundations of Statistics. Wiley. Google Books [2] Savage gives subjective expected utility a set of axiomatic foundations: if a person's preferences satisfy certain consistency requirements, their choices can be represented as maximizing expected utility under a subjective probability. It is the baseline for later disputes: only by seeing its consistency requirements can one see how Ellsberg and others locate real judgment's departures from it.
  7. D. Ellsberg (1961). "Risk, Ambiguity, and the Savage Axioms." Quarterly Journal of Economics, 75(4), 643-669. doi:10.2307/1884324 [2] Ellsberg's urn experiments show that people systematically prefer known probabilities to unknown ones. This ambiguity aversion violates Savage's axioms and cannot be reconciled by a single subjective probability. It gives experimental force to the Knightian distinction.
  8. H. A. Simon (1955). "A Behavioral Model of Rational Choice." Quarterly Journal of Economics, 69(1), 99-118. doi:10.2307/1884852 [2][4] Simon introduces bounded rationality and satisficing: actors constrained by cognition and information do not enumerate all options and optimize globally. They set an aspiration level and stop when they find an option that meets it. In unverifiable settings, "good enough" is often a rational form, not a failure.
  9. H. A. Simon (1947). Administrative Behavior: A Study of Decision-Making Processes in Administrative Organization. Macmillan. Google Books [2][4] Simon treats organizations as decision structures that amplify individual bounded rationality. Organizations set premises, divide responsibilities, and build routines so members can act under incomplete information. This book extends bounded rationality from individuals to institutions.
  10. A. Tversky and D. Kahneman (1974). "Judgment under Uncertainty: Heuristics and Biases." Science, 185(4157), 1124-1131. doi:10.1126/science.185.4157.1124 [2] Tversky and Kahneman show that people estimate probabilities through a small set of heuristics, such as representativeness, availability, and anchoring. These shortcuts often work but also produce predictable biases. The paper is a starting point for understanding where human judgment is reliable and where it fails.
  11. D. Kahneman and A. Tversky (1979). "Prospect Theory: An Analysis of Decision under Risk." Econometrica, 47(2), 263-291. doi:10.2307/1914185 [2][4] Prospect theory models real choice through a value function relative to a reference point and nonlinear weighting of probabilities. People weigh losses more heavily than equal gains and distort small and large probabilities. It is a descriptive correction to classical expected utility.
  12. D. Kahneman (2011). Thinking, Fast and Slow. Farrar, Straus and Giroux. Google Books [2][4] Kahneman synthesizes decades of research on judgment and decision-making through the dual-process frame of System 1 and System 2. The book helps build a general picture of cognitive limits and why even experts need external correction mechanisms.
  13. G. Gigerenzer and D. G. Goldstein (1996). "Reasoning the Fast and Frugal Way: Models of Bounded Rationality." Psychological Review, 103(4), 650-669. doi:10.1037/0033-295x.103.4.650 [2][4] Gigerenzer and Goldstein argue that simple rules using few cues and stopping early can, in real environments, match or outperform more complex statistical models. Their work counters the idea that heuristics are merely biases and asks why simplicity can be effective.
  14. G. Gigerenzer, P. M. Todd, and the ABC Research Group (1999). Simple Heuristics That Make Us Smart. Oxford University Press. Google Books [2][4] This collection develops the adaptive toolbox program: minds carry a set of simple heuristics tuned to particular environments, and their success depends on ecological rationality. The cases show how simple rules can make robust decisions under limited information.
  15. F. A. Hayek (1945). "The Use of Knowledge in Society." American Economic Review, 35(4), 519-530. link [2][4] Hayek argues that economically relevant knowledge is dispersed, local, and tied to particular circumstances. No central planner can gather it all; prices coordinate it in decentralized form. This reveals one source of unverifiability: the relevant information is never fully held by a single actor.
  16. M. Polanyi (1958). Personal Knowledge: Towards a Post-Critical Philosophy. Routledge and Kegan Paul. Google Books [1][3][4] Polanyi argues that all knowing contains tacit knowledge and personal commitment. Fully objective, fully formalized knowledge is an illusion. His account explains why scientific judgment cannot be replaced entirely by rules.
  17. P. E. Meehl (1954). Clinical versus Statistical Prediction: A Theoretical Analysis and a Review of the Evidence. University of Minnesota Press. doi:10.1037/11281-000 [1][4] Meehl reviews evidence showing that simple statistical or actuarial predictions often match or exceed clinical expert judgment. It is classic evidence for outsourcing judgment to checkable rules, while warning that expert confidence and expert accuracy may diverge.
  18. D. A. Schön (1983). The Reflective Practitioner: How Professionals Think in Action. Basic Books. Google Books [1][4] Schön describes reflection-in-action: professionals in ambiguous and unique situations do not simply apply theory. They converse with the situation, act, and reframe the problem. This captures a professional ability that cannot be verified in advance.
  19. G. A. Klein (1998). Sources of Power: How People Make Decisions. MIT Press. Google Books [1][4] Klein's field studies of firefighters, nurses, and other experts lead to the recognition-primed decision model. Experienced people under time pressure often generate a workable action by recognizing a pattern, then mentally simulating it. This helps explain when expert intuition can be reliable.
  20. P. E. Tetlock (2005). Expert Political Judgment: How Good Is It? How Can We Know? Princeton University Press. Google Books [1][4] Tetlock tracks political and economic experts over many years and finds that their average forecasts often fall short of simple extrapolation. Cognitive style explains more than credentials: pluralistic, self-questioning "foxes" outperform single-theory "hedgehogs." The book puts expert judgment under scoreable test.
  21. P. E. Tetlock and D. Gardner (2015). Superforecasting: The Art and Science of Prediction. Crown. Google Books [1][4] This book extends Tetlock's forecasting tournament work and describes "superforecasters": people who decompose problems, assign scoreable probabilities, update frequently, and use team correction. Forecasting appears not as a gift but as a learnable practice.
  22. N. N. Taleb (2001). Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets. Texere. Google Books [2][4] Taleb argues that people regularly mistake random outcomes for skill or necessity, especially in markets, where survivors are treated as masters. The book warns that success itself does not verify a judgment when causal structure is uncertain.
  23. N. N. Taleb (2012). Antifragile: Things That Gain from Disorder. Random House. Google Books [4] Taleb introduces antifragility: some systems do not merely survive volatility, but benefit from it. In an unpredictable world, he argues, one should preserve optionality and cap downside risk. This directly connects to shrinking the blast radius of failure.
  24. C. E. Lindblom (1959). "The Science of 'Muddling Through'." Public Administration Review, 19(2), 79-88. doi:10.2307/973677 [4] Lindblom argues that real public policy is often incremental rather than globally rational: actors make limited changes near the status quo, compare as they go, and remain tied to existing means. This legitimizes cautious correction as a reasonable response to complexity.
  25. K. R. Popper (1959). The Logic of Scientific Discovery. Hutchinson. Google Books [3] Popper develops falsificationism: scientific theories cannot be verified as true, only exposed to possible refutation. Falsifiability, not verification, marks the boundary between science and non-science. The book is one philosophical source for the present argument.
  26. T. S. Kuhn (1962). The Structure of Scientific Revolutions. University of Chicago Press. Google Books [1][3] Kuhn argues that science alternates between normal science under a paradigm and revolutionary shifts after accumulated anomalies produce crisis. Judgment and community matter; progress is not a simple linear accumulation of verified truths.
  27. W. V. Quine (1951). "Two Dogmas of Empiricism." The Philosophical Review, 60(1), 20-43. doi:10.2307/2181906 [3] Quine attacks the analytic-synthetic divide and the idea that each statement faces experience alone. Beliefs meet evidence as a web; any statement can be held if changes are made elsewhere. This is a classic argument for the underdetermination of theory by evidence.
  28. P. Duhem (1954). The Aim and Structure of Physical Theory. Princeton University Press. doi:10.1515/9780691233857 [3] Duhem argues that physical experiments never test a single hypothesis in isolation. They test a whole bundle of theory and auxiliary assumptions, so a failed prediction does not uniquely identify what is wrong. This is central to understanding why science does not rest on decisive single verifications.
  29. I. Hacking (1983). Representing and Intervening: Introductory Topics in the Philosophy of Natural Science. Cambridge University Press. doi:10.1017/cbo9780511814563 [3] Hacking shifts attention from representation to intervention: when we can reliably manipulate an entity to intervene in the world, we gain reason to believe in it. His experimental realism reminds us that verification is not only passive observation; it is also action.
← Cover1. The Luxury of Verification →

Open with WeChat

Scan this page in WeChat, then use WeChat's share menu.