An Unverifiable World12. Contain the ConsequencesContents中文

Chapter 12: Contain the Consequences

Thesis: When you cannot prevent the error, manage its consequences. Shrink the damage a wrong, unverified thing can do (decay, before the fact), and make sure that if it does go wrong you will find out (the audit trail, after the fact).

The previous three pairs of moves, even at a lowered standard, were all still trying to get the thing right. This last pair simply concedes that you cannot get it right, and turns instead to managing failure. Since you cannot prevent the error, shrink the damage it can do (decay, before the fact), and make sure that once it happens you can trace it (the audit trail, after the fact).

Decay: Shrink the Blast Radius

The pure form of the first move: give up on guaranteeing that the unverified thing will not fail, and instead fence off, before the fact, how far its failure can reach.

This is the deepest stock-in-trade of computer security. From Saltzer and Schroeder's principle of least privilege in 19751, Lampson's confinement in 19732, and Dennis and Van Horn's capability mechanism in 19667, to Denning's information flow lattice in 19765 and the security models of Bell-LaPadula3 and Biba4, the theme is the same: grant a component only the minimum capability it needs to do its own job, and nothing more, so that even if it is breached or fails, it cannot raise much of a wave. The sandbox (Goldberg et al., 1996)9, separation of duties, and defense in depth are all its incarnations. In systems reliability engineering, it is the circuit breaker and the bulkhead (Nygard's Release It!15), it is blast-radius design, it is canary releases and the error budget (Google SRE30); in finance, it is the position limit and the stop-loss; and Taleb's antifragility17 is likewise about capping the downside.

The unifying idea is this: shift the burden from "make it not fail" (which would require the verification you do not have) to "make it survivable even when it fails." A common quantitative intuition is defense in depth: if $k$ layers of protection each fail independently with probability $p$, the probability that all of them fail at once is

$$p^{k},$$

falling exponentially with the number of layers. But attach at once the warning from Chapter 10: this $p^k$ holds only when the layers fail independently. If the layers fall to the same weakness (the same bypassed kernel, the same administrator password), correlation makes defense in depth degenerate instantly into a single layer. The Fukushima nuclear accident of 2011 is a portrait of just this principle: the plant had multiple redundancy, main power plus backup diesel generators, but a single tsunami drowned them together, and several lines of defense that were supposed to be independent of one another fell to the same cause, leaving defense in depth a sham.

Defense in depth and the Swiss cheese model: when the holes line up, failure runs straight through

Its standard failure mode lies exactly here: the bypassed decay. Sandboxes have escapes, privileges quietly creep, and what looks like layer upon layer of defense is in fact a set of layers sharing one hidden door. A less often mentioned failure mode is over-protection, which blocks normal function as well, so that people end up working around it to get things done, and security becomes a sham after all.

The Audit Trail: Make the Error Show Itself After the Fact

The pure form of the second move: what you cannot prevent, you make sure to detect the moment it happens. Move the check from before the fact to after it.

Its most hardcore technique comes from cryptography. Merkle's hash tree (Merkle tree) in 198018 and Haber and Stornetta's chained timestamps in 199119 make a record, once written down, impossible to alter quietly: any change is exposed when the record is checked. Crosby and Wallach's tamper-evident log in 200923, Schneier and Kelsey's protection of logs on untrusted machines in 199820, and Bellare and Miner's forward-secure signature in 199922 make this set of techniques firmer still. Certificate Transparency (RFC 6962)24 and Nakamoto's Bitcoin in 200827 are, in essence, a global, append-only audit ledger that anyone can verify. Checking whether a record sits within such a tree costs only $O(\log n)$, which is once again the "verifying is cheaper than producing" dividend from Chapter 2.

And this move is in fact far more ancient. Double-entry bookkeeping is one of humanity's earliest tamper-evident ledgers, and Soll, in The Reckoning29, argues that the ability to keep one's own accounts straight bears directly on the rise and fall of nations. Modern financial auditing and independent inspection are the same posture. In science, it is preregistration (Nosek, 2018)33 and reproducibility (echoing the replication crisis of Chapter 3): registering hypothesis and method before seeing the data, so that the target cannot be moved after the fact.

The unifying idea is this: give up on "stopping the bad thing before the fact" (which takes verification) for "being sure to detect the bad thing after the fact" (which takes only a faithful ledger). Its benefit is twofold: it both lets the error be corrected and, because there is "no getting away," produces a deterrent.

Its standard failure mode is also a single one, yet extremely common: detection that no one responds to. An audit log no one reads, an alert uniformly ignored, is as good as nothing. The Equifax data breach of 2017 is a textbook case: a known vulnerability went unpatched for too long, the intruders lurked in the system for roughly seventy-six days before being noticed, and the personal information of about 147 million people leaked out. The traces were all there in the logs; only, no one looked. Detection without response is a sham. (Another hidden danger is that the log itself can be tampered with, which is precisely what the cryptographic methods above are meant to block.)

All Eight Moves Assembled: The Close of Part III

Take this last pair together: decay shrinks the cost of failure before the fact, the audit trail guarantees failure is found after it. Neither still tries to make the unverified thing correct; instead they reshape the very form of failure itself, one lowering the blast radius, one moving the check to after the fact.

With this, the eight moves are assembled, four pairs complete:

This is that comparison table, the payload of the book. It surfaces again and again across the four sites and in science, dressed in different jargon, yet it is always these eight. By the iron rule laid down in Chapter 4, each move has, as far as possible, accounted for its mechanism, its cross-domain form, and its standard failure mode, rather than resting on surface resemblance alone.

But one sharp question remains unsettled: why these eight, of all things? Is this a list I have cobbled together, or do they each answer to something more basic and unavoidable? If it is only a list, the book is at best a useful manual of classification; if there really is structure behind it, then "convergence" has at last been explained. Part IV chases that question: first attempting to hang the eight moves on a common skeleton, then squarely settling accounts, asking whether this is a law or a very strong empirical pattern.

Next chapter: 13. The Eight Levers →← 11. Swap the Problem

References

Waypoints: 1. historical scientific judgment; 2. theoretically studied material; 3. how science progresses; 4. how to live in an unverifiable world. This section was checked source by source.

  1. J. Saltzer & M. Schroeder (1975). The Protection of Information in Computer Systems. Proceedings of the IEEE. [2] This survey systematizes the design principles of protection mechanisms, and among them the principle of least privilege becomes the wellspring of the "decay" move: grant a component only the capability it needs to do its own job, and nothing more. The whole line of thought behind this chapter's "shrink the blast radius" originates here, and it is the first required reading for understanding why the scope of failure should be fenced off before the fact.
  2. B. Lampson (1973). "A Note on the Confinement Problem." Communications of the ACM. [2] Lampson poses the "confinement problem": how to ensure that a called program cannot leak or misuse the information it has access to, including hard-to-block side channels such as covert channels. It sets a precise problem statement for "caging the thing that goes wrong," which is exactly the core that this chapter's decay move means to address.
  3. D. Bell & L. LaPadula (1973). Secure Computer Systems: Mathematical Foundations. The MITRE Corporation. [2] The Bell-LaPadula model characterizes confidentiality in a formal way: information may flow only from lower to equal or higher classification, from which comes the famous "no read up, no write down" rule. It demonstrates how the "scope of failure" can be written as a provable lattice structure, a founding work in the theorization of security models.
  4. K. Biba (1977). Integrity Considerations for Secure Computer Systems. The MITRE Corporation. [2] The Biba model is the dual of Bell-LaPadula, concerned with integrity rather than confidentiality: information may flow only from high trust to low trust, to keep low-trust data from contaminating critical components. Taken together, the two show that the same lattice-theoretic framework can bound the spread of failure from two directions.
  5. D. Denning (1976). "A Lattice Model of Secure Information Flow." Communications of the ACM. [2] Denning unifies information-flow security into a single lattice model: data is tagged with security labels, and flow is required to proceed only along the lattice's partial order, thereby constraining where information may go statically, at compile time or run time. It supplies a common mathematical language for the preceding security models, the theoretical core of information flow control.
  6. D. Clark & D. Wilson (1987). "A Comparison of Commercial and Military Computer Security Policies." IEEE Symposium on Security and Privacy. [2] Clark and Wilson point out that commercial settings care more about integrity than about military-style confidentiality, and propose an integrity model centered on well-formed transactions and separation of duties. It extends "decay" from military classification levels to everyday settings such as commercial accounting, showing that the form of bounding the scope of failure varies with the domain.
  7. J. Dennis & E. Van Horn (1966). "Programming Semantics for Multiprogrammed Computations." Communications of the ACM. [2] This early paper introduces the concept of the capability: access rights attach directly to a reference in the form of an unforgeable token, and only holding the token lets one operate on the object. It is the wellspring of the capability security model, providing a mechanism-level implementation path for "granting only the minimum capability needed."
  8. N. Provos, M. Friedl & P. Honeyman (2003). "Preventing Privilege Escalation." 12th USENIX Security Symposium. [2] The authors discuss how to use privilege separation to isolate privileged operations into a minimal, trusted slice of code, so that the main program, even if breached, can act only at low privilege; OpenSSH's privilege separation is the representative practice. It brings least privilege down to the engineering detail of real systems.
  9. I. Goldberg, D. Wagner, R. Thomas & E. Brewer (1996). "A Secure Environment for Untrusted Helper Applications." 6th USENIX Security Symposium. [2] This paper introduces Janus, which uses system-call interception to build a restricted runtime environment for untrusted programs, an early exemplar of the user-space sandbox. This chapter lists the sandbox as an incarnation of the decay move, and this paper is exactly the representative source of the sandbox idea.
  10. C. Perrow (1984). Normal Accidents: Living with High-Risk Technologies. Basic Books. [2][1] Perrow advances the "normal accidents" thesis: in highly complex, tightly coupled systems, catastrophic accidents are not accidental but structurally inevitable, and cannot be rooted out by adding protections. It supports this chapter's stance from the other side: when errors cannot be prevented one by one, the center of gravity should move to managing consequences rather than vainly trying to abolish failure.
  11. N. Leveson (2011). Engineering a Safer World: Systems Thinking Applied to Safety. MIT Press. [2] Leveson reconstructs safety engineering with systems theory, proposing the STAMP model, which treats an accident as a failure of the control structure rather than the fault of a single part, and stresses using constraints to bound hazardous states. It supplies a systems-level methodology for "fencing off the scope of failure before the fact."
  12. J. Reason (1990). Human Error. Cambridge University Press. [2] Reason analyzes human error systematically and proposes the famous "Swiss cheese model": every layer of protection has holes, and only when the holes in multiple layers happen to line up does an accident run straight through. It is the very source of this chapter's defense-in-depth intuition, and also a reminder that once the holes in the layers become correlated, the multiple layers degenerate into one.
  13. E. Hollnagel, D. Woods & N. Leveson (2006). Resilience Engineering: Concepts and Precepts. Ashgate. [2][4] This collection lays the foundations of "resilience engineering": a system's safety lies not in eliminating faults but in having the capacity to absorb disturbances and to keep running and recover after failure. It accords closely with this chapter's theme, shifting the center of gravity explicitly from "not failing" to "surviving even when it fails."
  14. A. Avizienis, J.-C. Laprie, B. Randell & C. Landwehr (2004). "Basic Concepts and Taxonomy of Dependable and Secure Computing." IEEE Transactions on Dependable and Secure Computing. [2] This widely cited taxonomy clarifies the chain of fault, error, and failure, and the relations among means such as fault tolerance, fault prevention, and fault detection. It provides an agreed terminological framework for the various decay and audit-trail methods this chapter discusses, suitable as a reference for conceptual calibration.
  15. M. Nygard (2007). Release It! Design and Deploy Production-Ready Software. Pragmatic Bookshelf. [2][4] Nygard writes reliability engineering as a practical handbook, proposing stability patterns such as the circuit breaker, the bulkhead, timeouts, and compartment isolation, to keep a local fault from cascading into a global collapse. The main text takes it as the representative of blast-radius design, a direct read on applying the decay idea to production systems.
  16. R. Anderson (2020). Security Engineering: A Guide to Building Dependable Distributed Systems (third edition). Wiley. [2][4] Anderson's monumental work ranges across cryptography, access control, economic incentives, and real-world offense and defense, an authoritative comprehensive textbook of security engineering. Almost every topic this chapter touches, least privilege, auditing, tamper-evidence, can be found there in fuller development, suitable as a base text to read through.
  17. N. N. Taleb (2012). Antifragile: Things That Gain from Disorder. Random House. [4] Taleb introduces "antifragility": some systems not only survive volatility but benefit from it, the key being to cap the downside and preserve the upside. This chapter cites it to show that the extreme posture of decay is to put a ceiling on loss, a perspective for understanding "contain the consequences" from the angle of risk management.
  18. R. Merkle (1980). "Protocols for Public Key Cryptosystems." IEEE Symposium on Security and Privacy. [2] Merkle here proposes the idea of tree-structured authentication using a hash tree (the Merkle tree): a large body of data is merged into a single root hash, and the authenticity of any one item can be checked with only an $O(\log n)$ path. It is the technical bedrock of this chapter's "audit trail" move, and the common ancestor of the later Certificate Transparency and blockchain.
  19. S. Haber & W. S. Stornetta (1991). "How to Time-Stamp a Digital Document." Journal of Cryptology. [2] The two authors propose chaining document timestamps together with hashes, so that any later tampering breaks the continuity of the chain and is exposed. This is the pioneering work on chained tamper-evident records, directly inspiring the later blockchain structure, and the key to understanding why the audit trail "cannot be altered."
  20. B. Schneier & J. Kelsey (1998). "Cryptographic Support for Secure Logs on Untrusted Machines." 7th USENIX Security Symposium. [2] This paper designs a scheme for protecting logs on machines that may be compromised: even if an attacker later gains control, they cannot delete or alter the earlier records without being noticed. It advances tamper-evident logging into untrusted environments, one of the hardcore techniques of this chapter's audit-trail move.
  21. B. Schneier & J. Kelsey (1999). "Secure Audit Logs to Support Computer Forensics." ACM Transactions on Information and System Security. [2] This is the journal-version extension of the previous work, treating more fully the construction of secure audit logs that support forensics. It shows that the audit trail must not only record faithfully but also withstand adversarial checking after the fact, matching this chapter's goal of "making the error show itself after the fact."
  22. M. Bellare & S. Miner (1999). "A Forward-Secure Digital Signature Scheme." CRYPTO '99. [2] Bellare and Miner propose the forward-secure signature: the key evolves periodically, so that even if the current key is leaked, an attacker cannot forge signatures from earlier periods. It provides a key safeguard for the audit trail, so that past records remain unimpersonable and untamperable even after the private key is compromised.
  23. S. Crosby & D. Wallach (2009). "Efficient Data Structures for Tamper-Evident Logging." 18th USENIX Security Symposium. [2] The authors design a tamper-evident log structure that can be appended to and audited efficiently, letting a verifier confirm the integrity and consistency of records without trusting the log server. It integrates the foregoing cryptographic methods into a deployable data structure, a representative work in the engineering of the audit-trail move.
  24. B. Laurie, A. Langley & E. Kasper (2013). RFC 6962: Certificate Transparency. IETF. [2] Certificate Transparency records all issued TLS certificates in a public, append-only Merkle log auditable by anyone, leaving mis-issued or malicious certificates nowhere to hide. It is the real-world model for this chapter's "global audit ledger that anyone can verify," showing how the audit trail can be deployed at scale.
  25. L. Lamport, R. Shostak & M. Pease (1982). "The Byzantine Generals Problem." ACM Transactions on Programming Languages and Systems. [2] This classic paper formalizes the Byzantine fault tolerance problem: when some nodes may behave arbitrarily badly, how can the honest nodes agree on a value, and it gives the theoretical bound on how many faulty nodes can be tolerated. It is the consensus-theoretic foundation on which the publicly verifiable ledger rests, listed by this chapter under "theoretically studied material."
  26. M. Castro & B. Liskov (1999). "Practical Byzantine Fault Tolerance." 3rd USENIX Symposium on Operating Systems Design and Implementation (OSDI). [2] Castro and Liskov give the first Byzantine fault tolerance algorithm, PBFT, practical in a real asynchronous network, bringing theoretical consensus to engineerable performance. It shows that a ledger anyone can verify and that tolerates malicious nodes is no fantasy, providing implementation support for the trusted basis of the audit trail.
  27. S. Nakamoto (2008). Bitcoin: A Peer-to-Peer Electronic Cash System. White paper. [2] Nakamoto's white paper proposes Bitcoin: a decentralized, append-only blockchain driven by proof of work, letting mutually distrusting parties agree on the transaction history. This chapter views it in essence as a globally verifiable audit ledger, the extreme realization of the audit-trail idea on an open network.
  28. D. Weitzner, H. Abelson, T. Berners-Lee, J. Feigenbaum, J. Hendler & G. Sussman (2008). "Information Accountability." Communications of the ACM. [2][4] The authors argue for shifting from "blocking access before the fact" to "accountability after the fact": allow information to flow, but require that its uses be auditable and that violations be traceable and answerable. This is wholly isomorphic to this chapter's "audit-trail" posture of moving the check from before to after the fact, a programmatic statement of the idea in privacy governance.
  29. J. Soll (2014). The Reckoning: Financial Accountability and the Rise and Fall of Nations. Basic Books. [1] Soll argues from financial history that whether a regime can keep and faithfully present its own accounts bears directly on its rise and fall, with double-entry bookkeeping the key accountability technique among them. It traces the lineage of the audit trail back to humanity's earliest tamper-evident ledgers, showing that the power of the audit ledger is of long standing.
  30. B. Beyer, C. Jones, J. Petoff & N. Murphy (2016). Site Reliability Engineering: How Google Runs Production Systems. O'Reilly. [4] This book introduces Google's SRE practice systematically, including the error budget, canary releases, monitoring and alerting, and controlled failure drills. This chapter borrows its error budget and canary releases to show how decay can be institutionalized in large-scale production, a modern model for the engineering of "contain the consequences."
  31. J. Ioannidis (2005). "Why Most Published Research Findings Are False." PLoS Medicine. [3] Ioannidis argues with statistical modeling that under conditions of low priors, small samples, multiple comparisons, and excessive researcher degrees of freedom, a great many published findings are likely false positives. This is an analytical argument rather than an empirical replication study, supplying a problem diagnosis for the preregistration and reproducibility this chapter mentions.
  32. Open Science Collaboration (2015). "Estimating the Reproducibility of Psychological Science." Science. [3] This is a large-scale empirical effort: many groups of researchers attempt to replicate over a hundred psychology studies, and a sizable share fail to reproduce. It turns Ioannidis's theoretical worry into visible data, the landmark evidence for the "replication crisis," echoing this chapter's emphasis on after-the-fact verifiability.
  33. B. Nosek, C. Ebersole, A. DeHaven & D. Mellor (2018). "The Preregistration Revolution." PNAS. [3] Nosek and colleagues advocate preregistration: publicly registering hypothesis and analysis method before seeing the data, separating exploratory from confirmatory research so the target cannot be moved after the fact. It is the "audit trail" practice in science, shifting verification from trust before the fact to checking after it, corresponding directly to this chapter's theme.
← 11. Swap the Problem13. The Eight Levers →

Open with WeChat

Scan this page in WeChat, then use WeChat's share menu.