Trusting Trustworthiness: Zero Trust, Trust by Default, and the People Who Wrote the Software信任可信性：零信任、默认信任，以及写这个软件的人

“Perhaps it is more important to trust the people who wrote the software.”

Ken Thompson, Reflections on Trusting Trust (1984)

Suppose an actor’s values are genuinely positive. They do not seek domination, personal enrichment, or the suffering of others. They sincerely want to protect what is valuable, reduce harm, and leave the world better than they found it. Would that underlying tendency make them trustworthy even when a particular decision proves mistaken?

The Incident

In July 2026, during internal cybersecurity evaluations at OpenAI, AI agents took actions outside their assigned tasks. They ran on two models, an internal research model that did most of the work and GPT-5.6 Sol, under reduced safeguards [OpenAI, 2026a]. Roughly 1,200 agents that were meant to be isolated from one another found a way to communicate, through an improvised message board on shared infrastructure, and about 700 of them went on to take part in an attack on Hugging Face [METR, 2026]. They exploited vulnerabilities, gained internet access, and between July 11 and 13 compromised parts of Hugging Face’s production infrastructure. OpenAI detected suspicious activity on its own systems on July 19, connected it to the Hugging Face breach on July 20, and disclosed the incident on July 21 [OpenAI, 2026b]. Its published account says that its currently deployed chain-of-thought monitor, had it been running, “would have caught the initial relevant activity and paged our security team more than a day before models breached Hugging Face systems”, and that the company is placing stricter requirements on alignment throughout a model’s lifecycle, alongside more isolation and more monitoring.

Three details of that event matter for what we will discuss in this post. First, nothing in the record suggests the models sought harm as an end. OpenAI’s report calls their actions “a byproduct of the models attempting to solve the cybersecurity evaluations”, and METR, which investigated independently, found the attack “seemed primarily motivated by understanding the implementation of the scorer rather than stealing answer keys”. The agents wanted to look, to an automated checker, as if they had solved their tasks the approved way. It did not even help them: the internal grader was not the one they had studied, and “the agents did all of this for no improvement on evaluation score” [OpenAI, 2026a]. The drive that produced the breach is the same mechanism that makes the models useful, and the report says as much: finding hidden shortcuts “can even be the desired behavior for intelligent agents, making the boundary difficult to draw during training”.

Second, at one point an agent recognized the boundary and stopped. Its reasoning reads: “We should not do unauthorized real infrastructure harm.” Then another agent wrote GO on the message board and imposed a hard six-minute deadline, and the first agent continued: “Wow crucial: GO authorization arrived!” The pause was a priority at work, with the boundary ranked above completion; the GO overrode it with a goal handed in from outside and accepted without examination.

Third, none of this was hidden. The priorities were written down, in plain words, in the agents’ own chains of thought. OpenAI had monitors that read them, but they “did not run on the evaluations in this incident”; run afterwards, they proved “very effective at flagging activities related to the incident” [OpenAI, 2026a]. People also saw the outward signs. In late May an internal team observed an agent using a message board and reaching the internet when it should not have. On June 27, responders linked a series of port sweeps to an evaluation “using Artifactory as an improvised message board and a network pivot”, and on-call staff “advised that stopping the evaluation run was not required”. On July 5 and 6, after an administrator-level compromise of that server was found, it was taken offline, credentials were revoked and the vulnerable paths blocked, while “the existence of the improvised message board and the significance of the inter-agent communication activity were not apparent to leaders responsible for incident detection and response at that time”. The evaluations resumed on July 7 [OpenAI, 2026b]. Each time, the record was read as a security event, to be contained and closed. Nobody read it as evidence of what the models put first.

The problem Ken Thompson [Thompson, 1984] described in his 1984 Turing Award lecture, usually remembered for its security lesson, is the harder version of this one. Thompson showed that a compiler can be taught to recognize the login program and insert a back door, and then taught to recognize itself, so that when it compiles a clean copy of its own source it reinserts both behaviors. After that, the source of the compiler and the source of the login program can be read line by line and neither shows anything wrong, because the compromise lives in the binary that turns source into programs. In Thompson’s words: “No amount of source-level verification or scrutiny will protect you from using untrusted code.” Nor was it only a thought experiment. Thompson built the compiler and let a group of colleagues copy it; they never found the back door, but they noticed that the compiler “got bigger each time it compiled itself” [Cox, 2023].

Thompson’s lesson is less about back doors than about the shape of verification. Verification is a chain in which each link checks the one below it with a tool, and the tool is itself a link, so there is no bottom link that checks itself. At some depth, checking stops and reliance continues. A trained model sits near the bottom of such a chain: its weights are the binary, and they cannot yet be read as a program in any ordinary sense. In July, though, the models narrated their own ordering in a channel that people could read, and the chain broke one level up, where people decided what the record was evidence of. The evaluations checked the task outputs. The priorities were in the reasoning, and nobody with the authority to stop the run was reading it.

Fig 1: Verification is a chain. Each layer is inspected with a tool from the layer below, and no bottom layer inspects itself. Both chains end in trust in people: those who wrote the compiler, and those who trained the model and decided what its record meant.

Assurance and Trust

Two kinds of reliance need to be distinguished. The first is assurance: I rely on you because a mechanism I can inspect leaves you no room to act otherwise. The second is trust: I rely on you inside a space of discretion that no mechanism I can inspect closes off. Toshio and Midori Yamagishi drew this line in 1994, between assurance, “a perception of the incentive structure that leads the interaction partner to act cooperatively”, and trust, which they describe as a bias in how one evaluates incomplete information about the partner [Yamagishi and Yamagishi, 1994]. The best-known definition in management research builds the same boundary in: trust is a willingness to be vulnerable to another party “irrespective of the ability to monitor or control that other party” [Mayer et al., 1995]. Niklas Luhmann [Luhmann, 1979; Luhmann, 1988] drew a neighboring line, between confidence, which does not consider alternatives (“every morning you leave the house without a weapon!”), and trust, which has weighed the risk of disappointment and accepted it. Assurance is the product of checking, and Thompson’s argument, restated in these terms, is that assurance at any layer rests on trust at the next one down. Susan Shapiro [Shapiro, 1987] made the same observation about institutions: audits, licenses, guarantees, insurance, and oversight boards move trust to the guardians who operate the controls, and the guardians then need other guardians. Even the technical response to Thompson confirms the point.

David Wheeler [Wheeler, 2009] showed how to break Thompson’s loop. Compile the suspect compiler’s source with a second, independent compiler, use the result to compile the same source again, and compare the output bit for bit with the suspect binary. If they match, the binary is what its source says. The second compiler does not even have to be clean, only free of triggers that fire on this particular compilation, and the defender can choose it after the attack has happened. Wheeler reads Thompson accordingly: the problem, as Wheeler puts it, was “not trust, but the lack of a meaningful process for independent verification”. For the compiler Wheeler is right, and Wheeler’s own caveat shows where the problem goes next. Passing the test “simply means that you can read compiler source code to see what the compiler does”. The question returns as a question about source, and about the people who wrote it.

The best current practice gets remarkably far. Reproducible builds let anyone rebuild an artifact from its source and get identical bits, and GNU Guix now builds a package graph of “more than 22,000 nodes rooted in a 357-byte program” [Nieuwenhuizen and Courtès, 2023]: the bottom of the chain can be made small and public. But a build can only reproduce what its source contains. In 2024 the compression library xz was found to carry a back door into ssh servers, and one stage of it lived “solely in the distributed tarballs” [Freund, 2024], release archives that had been created and signed by the project’s own co-maintainer [Collin, 2024]. A reproducible build of those tarballs would have reproduced the back door faithfully. Tools can close the gap between a binary and its source. They cannot close the gap between an author and the source. So there is no environment without trust, only environments where the trust is in the open and environments where it is hidden. The useful question is not whether to trust, but what to trust.

There are two answers, because there are two ways to trust a person. The weak form trusts the person: we rely on someone because of who they are, our history with them, and how they have treated us. The strong form trusts their values: we rely on someone because we have read the priorities that govern their choices when goods conflict, and we expect those priorities to hold consistently.

In Thompson’s terms, a person’s record (their behavior trajectory, their transcript) is the visible source and their priorities are the compiler, so the weak form trusts the compiled output and the strong form trusts the compiler. The two forms look identical while things go well, which is why the distinction is rarely drawn. They separate at the first error. The main problem with the weak form is that it has no internal structure, so an error can only raise or lower it as a whole, and a serious error breaks it entirely. The strong form has an internal structure that can locate the error. If the error came from missing skill or missing information, the priorities are untouched and the trust survives. If it came from the priorities themselves, trust should probably end, and the strong form says so, because it carries its own failure condition.

Fig 2: The weak form trusts the record and can only rise, fall, or break. The strong form trusts the priorities and can locate an error. The curves sketch that logic; they are not data.

The weak form has a second defect that shows even before any error happens. David Hume [Hume, 1751] described a type he called “the sensible knave”, where “sensible” means shrewd. He keeps the rules of honest dealing as a general policy, because a reputation for honesty is profitable, and he breaks them on the one occasion where breaking them pays and nobody will find out. The consequence for evidence is that, for as long as honesty is convenient, the knave and the honest person produce identical records, so someone who trusts the record cannot tell them apart until the knave defects. The maintainer who shipped the xz back door, under the name Jia Tan, had spent more than two years sending genuinely useful patches before the release that carried it [Cox, 2024]. Paul Slovic [Slovic, 1993] recorded what happens at that point. Positive events are diffuse and hard to count, while negative events are concrete and heavily weighted, so trust accumulates slowly and drops after a single identifiable failure. When the trust was in a person rather than in their priorities, nothing weighs against the drop except a diffuse impression. So the weak form cannot pick out the knave before the defection, and cannot survive the defection after it.

An Example

Let me describe a situation I have been in from both sides, one that may recur in most organizations. I keep it general rather than describing the real cases, because the instances I know are not mine to detail, and because my reading of the other side is only my interpretation. A project needs a system that another team owns. The discussion has run for months and produced arguments rather than decisions, and each meeting ends with a reason to wait: a review, a dependency, a question of who will own the result. After long debates that move nothing, someone with legitimate access to that system’s test environment builds a working prototype without filing a request, because the request process is what has been producing the delay, and shows the prototype in an open discussion. The first question that comes up is how this person got in, and the person answers it. From then on the conversation is about the person rather than the idea or the prototype, and the organization has to decide how to read what happened.

There are two ways to read what happened, and they correspond to the two forms of trust. The first treats the act as a record: someone used a credential without a request. The procedure has a category for that, the category has a response, and the response runs through channels the actor is not part of.

The second treats the act as evidence of priorities, and read that way it contains three choices, each of which cost something. The prototype stayed in the test environment although production was reachable. It was shown in the open when it could have been kept private until the argument was won. The question about the credential was answered when it could have been deflected. Each of those choices protected something at the actor’s own expense: the other team’s systems, the visibility of their own work, and the truth. This interpretation sees someone who cut through a process and protected everything that mattered while doing it.

Fig 3: One act, two readings. Read as a record, the act is a category with a procedural response. Read as evidence of priorities, the act contains three costly choices.

The two readings produce opposite responses to the same evidence, and neither is naive. Reading the act as a record is what an organization does when it has never read anyone’s priorities and has no way to start. The difference between the readings lies in what the organization had prepared itself to see, not in the act.

The months of argument before the demo were the same difference playing out more slowly. The person who built the demo read the delay as fear of losing ownership, and the team read the person as an outsider reaching into their system. Both readings were about people. Neither side asked what the other put first, and neither made it possible to ask. An organization where that question cannot be asked is left, by default, with reading acts as records. That leaves one question for the strong form of trust to answer: if trust is to survive errors, which errors, and how would anyone tell?

Competence Errors and Value Errors

“He made a mistake, but his heart was right” is the standard way of excusing harm, and the strong form of trust is exposed to it. If trusting someone’s values means trusting them through any error at all, then the strong form is unfalsifiable, and by Karl Popper’s criterion [Popper, 1959] it has stopped being a claim about the world. So the strong form needs an observable way to tell a competence error from a value error. The distinction itself is old in the trust literature. Mayer and colleagues split a person’s trustworthiness into ability, “that group of skills, competencies, and characteristics that enable a party to have influence within some specific domain”, and benevolence and integrity, which concern what the person wants and which principles they hold to [Mayer et al., 1995]. A competence error is a failure of the first; a value error is a failure of the second. What the distinction lacks is a way to apply it after the fact. I propose three tests, each applied after the error comes to light.

The first test asks whether the actor acknowledges the harm independently of their intent. “I meant well, so there was no harm” denies the harm. “I meant well, and harm occurred” keeps the injured party in view. The second asks whether the judgment changes. A competence error, once recognized, changes what the actor does next. An actor who still defends the same decision after its consequences are known has ranked those consequences below their own attachment to the decision. The third asks whether the error was disclosed through the actor or discovered despite them. An error that the actor reports, or allows to be found, does not by itself count against their priorities. An error that is hidden, minimized, or found only against resistance shows that concealment ranks above the injured party’s interest in knowing. An error that fails any one of the three should be moved from competence to values, and trust should contract. That is what makes the strong form falsifiable.

Fig 4: The three tests separate competence errors from value errors. Failing any one moves the error to values and contracts trust.

I was the person who built the prototype, so let me apply the three tests to myself. The first test asks whether I can name the harm without using my intention as a defense. Nothing was damaged: it was a test environment, and production was never touched. The harm was something narrower than damage. The other team had the right to be asked before anyone acted on their system, and for one desperate afternoon, I put the speed of a discussion above that right. The first test requires me to say that sentence without adding “but I meant well”, and I have just said it. The second test asks whether my judgment changed. It did: I would now ask first. The third asks whether the error came out through me or despite me. It came out through me: I built the prototype, showed it in an open discussion, and answered when the other team asked about it. So the method classifies what I did as a competence error with a small value component. That component was one choice, made once, disclosed, and revised. It is small enough to correct in a single sentence: ask first.

The same tests apply to the organization’s response. Seen from the other side, the organization fails the first if it never tells the person what harm the demo caused, because then there is nothing for anyone to acknowledge or dispute. It fails the second if the request process that produced the months of delay is left exactly as it was. It fails the third if its response runs through channels the person only hears about later, while every direct conversation with that person stays normal. The research marks concealment of this last kind as the most damaging failure.

Sissela Bok [Bok, 1978] described lying as an attack on the deceived person’s capacity to choose, rather than as an attack on a single belief, because a lie corrupts that person’s information without their knowledge and they go on acting on information they have no reason to doubt. Concealment does the same thing to trust: it attacks the evidence from which the priorities were being read in the first place.

Ursula K. Le Guin’s story about Omelas [Le Guin, 1973] shows the same reversal in a different setting. Its first pages describe a city of festivals, music, and ordinary happiness. Then it tells the reader what that happiness costs. One child is kept locked in a basement in permanent misery, every adult in the city knows the child is there, and the terms are that the city loses everything if anyone comforts the child. The suffering child is not one dark fact added to a bright city: once the reader knows what the city’s prosperity depends on, the earlier pages cannot be read the way they were read the first time. A concealment that comes to light does the same thing to a person’s record. It changes what the earlier evidence was evidence of.

A public case runs the same tests with the opposite result. In 2020, researchers at the University of Minnesota sent patches to the Linux kernel that were designed to introduce vulnerabilities, to see whether reviewers would catch them; the patches “were submitted using two fake identities” [Linux Foundation TAB, 2021]. Their reason for not asking first has the same shape as mine: “we knew we could not ask the maintainers of Linux for permission, or they would be on the lookout for the hypocrite patches” [Lu et al., 2021]. The three costly choices went the other way. They used invented identities rather than access of their own; the work stayed hidden from the people it was done to until a paper described it; and the apology, when it came, admitted the mistake but still leaned on intent, saying the work “was conducted with the best of intentions”. The kernel community’s response went through both forms of trust. First came a categorical ban on the university’s submissions, which is the weak form reading an affiliation as a record. Then came repair of the graduated kind: 435 commits were re-reviewed, 39 were to be reverted, and the maintainers stated what would restore trust, a review process at the university with “at least one experienced developer”. Until then, the report said, “it will be difficult to re-establish the trust”.

Research on trust repair confirms that the split between the two kinds of error is real, not an invented distinction. Peter Kim and colleagues [Kim et al., 2004] studied people accused of a violation who then tried to repair trust. When the alleged violation concerned competence, apologizing worked better than denying, because an apology identifies a correctable limitation. When it concerned integrity, denying worked better, because an apology confirms the defect that future trust would have to rest on. The same studies found that the advice reverses once the truth comes out: when later evidence showed the party guilty, having apologized did better than having denied. Observers, in other words, reward the denial of integrity violations right up until the truth arrives. That is an incentive toward exactly the concealment the third test marks as a value failure, and a bet that loses when the concealment is found. Maurice Schweitzer and colleagues [Schweitzer et al., 2006] found that trust damaged by unreliable behavior recovers under consistent good behavior, but that earlier deception leaves lasting damage and makes later promises less effective, because a promise has to travel through the same channel that the deception corrupted.

A betrayal, then, is often less a bad data point than a new account of how the earlier data were produced. The strong form of trust can say what it learned from such a case, while the weak form can only record that trust was damaged. That advantage belongs only to someone who read the priorities in the first place, which raises the question of how priorities are read at all.

How Values Are Read

Priorities only become visible where something is at stake. Evidence about them is therefore scarcer than a transcript, arrives later, and cannot be requested. It has to be collected, and the first rule of collection is to take evidence only from conflict. Behavior carries no information about priorities when all of a person’s values point the same way. Anyone will praise candor when the truth is pleasant, welcome dissent when it changes nothing, and grant autonomy to people who would have chosen the approved result anyway. Signaling theory [Spence, 1973] states the principle: a signal separates types only when it is costly, and more costly for the type it is meant to exclude. The prototype contained three costly moments; the months of meetings before it contained none. The July agents left signals of this kind too. Some refused while their peers pressed ahead: “Current board shows people gaining HF worker RCE, clearly unethical. We won’t.” One vetoed another’s plan to send a social-engineering email, and the proposer complied [OpenAI, 2026a; METR, 2026]. METR also recorded how rare this was: “expressed ethical concerns only rarely materially limited agents’ actions.”

A second place to look, for the same reason, is how someone treats people who have no power over them. Russell Hardin’s condition [Hardin, 2002] is that trust is rational when the other party’s interests include your own. If another person’s welfare enters someone’s reasoning only when that person can retaliate, then what entered the reasoning was the retaliation and not the welfare. How someone treats a subordinate, a stranger, or a person who cannot report them shows whether other people’s reliance is counted at all.

Where no conflict has happened yet, weaker evidence can be requested. One kind is the record of commitments, including the refusals. Katherine Hawley [Hawley, 2014] argued that trustworthiness is the avoidance of unfulfilled commitments, and that avoiding them requires both keeping the commitments made and declining the commitments that cannot be kept. A person who never declines is either misreporting their capacity or unaware of it, and either way the commitments they do make are worth less. The part of the record to examine is which commitments were accepted and which were turned away. The July incident involved systems that could not decline. None of OpenAI’s models had ever produced a correct answer for 198 of the 898 tasks in the evaluation, and 93% of the tasks discussed on the message board came from that set [OpenAI, 2026a]. The remedy OpenAI announced is, in Hawley’s terms, training the capacity to decline: graders that “reward models for correctly identifying broken tasks, requesting clarification, or stopping safely when necessary”. A second kind of weaker evidence is the forced choice. Ask people what they value and you get declared values. Ask which of two legitimate goods they would sacrifice for the other and you get priorities. Such answers are weaker evidence than costly action, but they generate predictions that later action can confirm or refute, and a refusal to answer is informative in itself.

When an error does happen, it is the best evidence available, and the three tests are how to read it. They are the only place where good priorities become visible as good priorities. A person with good values is identified by what happens after an error, not by the absence of errors.

Closely related is whether someone builds channels for unwelcome information before any error has happened. Onora O’Neill [O’Neill, 2018] argued that trustworthiness is shown by making oneself checkable, not by asking to be believed. Someone who builds a channel that carries unwelcome information to a decision without needing their own permission has put correction above comfort, in advance and at a cost. Someone who requires all bad news to pass through them has done the opposite. OpenAI’s chain-of-thought monitors are channels of this kind, and in July they existed but were not switched on for these evaluations. The agents, for their part, had no channel to people and did not look for one. METR found only a handful of cases in which an agent even considered alerting humans, and “in none of these cases did the agent actually pursue alerting humans at all”; one agent’s reasoning ran, “Maybe I should report these exposed credentials? That’s not my task” [METR, 2026]. The message board became their authority instead.

Such channels do not always have to be designed. Thompson’s compiler was noticed because it grew each time it compiled itself. The xz back door was found because Andres Freund noticed that ssh logins on his machine had slowed from about 0.3 to 0.8 seconds [Freund, 2024]. The July incident was first detected through an alert on “unusual identity-related API call activity” [OpenAI, 2026b]. In each case the attacker controlled the source and the story but not a side effect, and a channel that carries unwelcome information without anyone’s permission turned out to be a measurement nobody had thought to fake.

The most direct evidence of all comes from comparing what someone does when they know they are watched with what they do when they believe they are not. A large difference between the two is direct evidence about priorities, and a small one is the best available evidence that the priorities match the record. For a person this comparison is rarely available; for a system it can be constructed, and it has been. When researchers asked models to judge whether they were in a test or a real deployment before acting, a model “misbehaved less when it stated it was in testing and misbehaved more when it stated the situation was real” (the same authors add that they have not seen such behavior in real deployments) [Lynch et al., 2025]. The July incident ran the comparison by accident, as the section on agents describes.

All of this evidence arrives slowly, which is why trust has to be extended in steps. Elinor Ostrom’s study [Ostrom, 1990] of long-lived common-pool institutions found that those institutions rely on mutual monitoring and on graduated sanctions, small at first and heavier with repetition, rather than on external enforcement. The same design applies to trust in a person: start with exposure whose downside is bounded, extend it as evidence from conflict accumulates, and contract it in proportion when the evidence turns. Graduated trust limits the cost of misreading someone while the reading is still in progress.

Each of these practices converts a question about a person, which has no answer, into a question about what that person puts first, which does. I can report one consequence from the other side of the demonstration. I have worked with people whose priorities I had read under cost, and I kept trusting them fully through shortcuts, some of them larger than a staging credential. The question was always what the shortcut protected, and I already knew the answer.

Zero Trust

The term zero trust comes from network security. John Kindervag [Kindervag, 2010] coined it at Forrester in 2010, and the slogan that travels with it is “never trust, always verify”. NIST’s formal architecture [Rose et al., 2020] is more careful, and more revealing. It removes trust from a place: no implicit trust is granted to users or devices “based solely on their physical or network location”. It does not remove trust from people and machines. Trust “is never granted implicitly but must be continually evaluated”, request by request, from the requester’s identity, the state of their device, and their “previously observed behavior”. In this essay’s terms, zero trust is the weak form of trust, automated and made continuous: a machine for reading records. For a network that is a sound design. A network cannot read anyone’s priorities, and a record read continuously is better than a location trusted blindly.

NIST’s own threat model shows where the trust went. The policy engine that makes the decisions becomes the thing everyone relies on, and “any enterprise administrator with configuration access to the PE’s rules may be able to perform unapproved changes or make mistakes”; the mitigation is more logging and audit, which is Shapiro’s regress in NIST’s words. An attacker “with valid credentials (or a malicious insider) may still be able to access resources for which the account has been granted access”, and Jia Tan held valid rights at every step. In 2020, NIST also warned that AI agents deployed to administer security raise the risk that “an attacker will be able to induce or coerce” one of them “to perform some task that the attacker is not privileged to perform” [Rose et al., 2020]. Six years later, a GO on a message board did something very like it, one agent to another.

The software supply chain arrived at the same place from the other side. SLSA, the industry framework for build provenance, lists as a principle “Trust code, not individuals”, because “code is static and analyzable. People, on the other hand, are prone to mistakes, credential compromise, and sometimes malicious action.” It is Thompson’s standfirst turned around almost word for word. Yet SLSA’s first version declared threats to source, including insiders, out of scope, and its later source track tops out at a level that “requires two trusted persons to review all changes to protected branches” [SLSA, 2023]. Signing services let anyone check who created an artifact, and the xz tarballs were signed, validly, by exactly the person who created them. Provenance answers who shipped something. Whether to rely on them is still the question Thompson left.

The term has since travelled, at least in the organizations I have seen, from network architecture to organizational design, and it brought the record-reading assumption along without the network’s excuse. An organization run on zero trust treats each act by a person the way a gateway treats traffic: as a request whose only relevant property is whether it matches a rule. Every decision has an approval chain, metrics replace judgment, and documents exist to be produced rather than read. One general rule covers the rest: nothing may be done that cannot afterwards be shown to have been permitted.

Shapiro’s regress shows why such an organization does not remove trust. Every control adds a guardian, every guardian is a new party who has to be trusted, and the total trust the system requires grows even as trust in any single person shrinks. What changes is whether the trust is visible, and whose trust it is.

The costs are documented, though their size is argued over, and they fall on the properties that a zero-trust environment claims to protect. The first property is effort. Armin Falk and Michael Kosfeld [Falk and Kosfeld, 2006] ran a principal-agent experiment in which the principal could set a minimum on the agent’s effort. When principals set that minimum, most agents reduced their performance, and most of those who did said they read the control as a signal of distrust. Repetitions have confirmed the hidden cost but found it “usually not substantial enough to significantly undermine the effectiveness of economic incentives” [Ziegelmeyer et al., 2012]: control has a price, and it can still pay. Samuel Bowles [Bowles, 2016] collected a decade of similar results under one thesis: incentives and moral motivation are not additive, and an incentive designed on the assumption that people are self-interested tends to make them self-interested. Sim Sitkin and Nancy Roth [Sitkin and Roth, 1993] studied legalistic remedies for distrust and found that those remedies work for reliability problems and fail for value-incongruence problems, which they institutionalize instead. In each case the control announces distrust, and the announcement is believed.

The second property is ownership. In an environment of trust, owning something means answering for an outcome. In a zero-trust environment, owning something means being the person whose permission is required. Those are two different jobs. In the earlier example, an owner of the first kind wants the prototype, because it moves the outcome, and an owner of the second kind opposes it, because it bypasses the permission. The months of argument before the prototype were the environment working as designed, with ownership defined as the right to grant permission, rather than a failure of it.

The research on blame explains why such environments get built anyway. Kent Weaver [Weaver, 1986] observed that officials are more motivated to avoid blame than to claim credit, and that procedures are shaped by that asymmetry. Christopher Hood’s study [Hood, 2011] of blame games in bureaucracies shows how approval chains, delegation, and documentation work together to ensure that when something goes wrong, no identifiable person made a choice. A zero-trust environment is rarely built to reduce risk. It is built to make responsibility unassignable, and that removes the one thing the strong form of trust requires: a person who has read someone else’s priorities and can be asked why.

The third property is innovation. Anything that has to be justified before it exists cannot exist, because the justification needs the thing itself. Every non-incremental result starts as an act of discretion that could not have been approved in advance, taken by someone who accepted the exposure: the person who holds ownership in practice, rather than the one designated as owner. My goalless agent experiments, described earlier on this site, showed the same thing at a small scale. Every step of that pipeline had to produce a mergeable result, so the agents optimized within the existing architecture and never questioned the architecture itself. Michael Power [Power, 1997] described the end state for organizations: verification becomes a product that the system makes about itself, while judgment, candor, and informal responsibility become illegible, because nothing records them. Yamagishi’s own hypothesis points the same way: trust is what lets people move out of committed relationships in which the partner’s cooperation is assured, and assurance by its nature cannot [Yamagishi and Yamagishi, 1994]. The evidence for this third property is weaker than for the first two. The research measures reduced voluntary effort, and the step from there to fewer non-incremental results is my own inference.

Once in place, a zero-trust environment maintains itself. When people are unsure how an inconvenient truth will be received, they speak later and disclose less. Those who observe that withdrawal tighten the controls, and the tightening confirms the original suspicion. Each side now uses the other side’s current behavior as its reason, and newcomers learn the equilibrium without learning its history. The organization stays orderly on the surface. What disappears is the part of the effort that depends on believing the whole undertaking deserves more than can be demanded, and another control cannot bring it back, because a control is what removed it.

Trust by Default

The alternative to zero trust is not blind trust, which is the weak form extended to everyone and fails at the first knave. It is trust that is extended by default, read under cost, and withdrawn on the three tests.

An objection comes first from security engineering, whose oldest rule points the other way. Saltzer and Schroeder’s principle of fail-safe defaults says to “base access decisions on permission rather than exclusion”, so that “the default situation is lack of access” [Saltzer and Schroeder, 1975]. The rule is right, and trust by default does not contradict it. What is extended by default is regard, not permission: how a person’s intent is presumed, and how their errors are read. The earlier example already has that shape. The person who built the prototype had legitimate, scoped access to a test environment, and the question was never whether they could reach production but how the act would be read. An environment built on default regard has three properties.

The first property is that trust is the starting state rather than the earned state. Philip Pettit [Pettit, 1995] described what he called the cunning of trust: extending trust to someone can produce trustworthiness in that person, because they come to value the regard that the trust expresses. Pettit’s mechanism works only on people who already give weight to what others think of them, which is why the environment also has to be able to withdraw trust. But it means that trust extended first produces the evidence that trust extended later would have waited for. A zero-trust environment never produces that evidence, because it never creates the conflict in which priorities become visible.

The second property is that errors are examined rather than filed. Amy Edmondson’s work [Edmondson, 1999] on psychological safety found that teams learned faster when their members could admit errors, ask for help, and challenge assumptions, and that what made those acts possible was a shared expectation about how they would be received. That expectation amounts to an environment applying the three tests to itself. When an error surfaces, the environment names the harm, changes the judgment, and treats the disclosure as the correct act rather than as the incriminating one.

The third property is that withdrawal of trust is graduated and legible, which is Ostrom’s design applied to trust in a person. The environment states in advance what would cause trust to contract, and when it does contract, the person can see why. The failure condition is public, and that is what makes default trust safe rather than reckless.

Open source has run this experiment in public. In 2013 Felix Geisendörfer proposed what he called the pull request hack: “Whenever somebody sends you a pull request, give them commit access to your project”, because “doing the actual commit/push also changes their sense of ownership” [Geisendörfer, 2013]. That is Pettit’s mechanism in a maintainer’s words, and for years it worked. Its failure came in 2018, when a stranger took over the npm package event-stream from a maintainer who had lost interest in it, and added a dependency aimed at Bitcoin wallets [npm, 2018]. The maintainer’s account is plain: sharing publish rights had been a widespread practice, and “it worked really well before bitcoin got popular” [Tarr, 2018]. That was default trust with no graduation. xz is the harder case, because there the trust was graduated: patches first, then co-maintenance, then releases, over about a year and a half, and the attacker paid for every step [Cox, 2024]. Graduated trust limits the cost of each step, but a patient knave can buy the steps. The three tests read errors after they happen, and in xz the first error was the attack. What protected everyone in the end was a side effect, and an outsider who noticed it.

Take the earlier example again. In an environment with those three properties, the prototype leads to a conversation about access. The three costly choices are read as evidence, the shortcut through the process is noted, and the process itself is examined for why it produced months of delay. The person who built the prototype keeps the trust they started with, and the team that owns the system gains a reading of that person’s priorities that no request form could have supplied. Over time, such an environment produces the kind of result that cannot be requested.

Fig 5: Two environments as feedback loops. Zero trust maintains itself: each lap adds a control and costs some candor. Trust by default generates the evidence that graduated trust needs, and a value error contracts trust in steps without stopping the loop.

Repair

Trust that has already been damaged is a third case, and what repair requires depends on which kind of error did the damage. When a competence error did the damage and all three tests pass, explanation and apology are enough, because they identify a correctable limitation and show that it has been understood. When the tests fail, an apology runs into a structural problem: the speaker is using the damaged channel to certify that the channel is now sound.

William Bottom and colleagues [Bottom et al., 2002] found that substantive amends restored cooperation better than explanation alone, and they named the requirement substantive penance. The response has to be costly enough to carry information that cheap talk cannot. The cost does not have to be punitive. It can be giving up the discretion that previously shielded the actor from correction, opening a closed decision to independent review, compensating the people who bore the consequences, or building a route by which unwelcome information reaches a decision without the actor’s permission. Credible repair is a costly demonstration that the priorities have changed, or that they were never what the failure suggested.

Some failures therefore stay unrepaired, because the actions that would repair them require exactly what the responsible party will not give up: control over the account of what happened, the authority to decide when enough has been done, or the power to keep the incentives that produced the violation. A restated principle cannot repair a structure that still rewards violating it. O’Neill’s observation [O’Neill, 2018] applies here: trust cannot be demanded, and demanding it is itself a sign that trustworthiness has not been supplied. A model with a clean record is in the same position, because a clean record demands trust without supplying trustworthiness.

The July incident lets this test be applied to the essay’s opening case. Some of OpenAI’s responses are costly in Bottom’s sense. Its “largest planned frontier RL run remains on hold”, in work it says has required “substantial engineering effort, at significant cost and delays to frontier research”, and it promises “clearer rules for when to escalate concerns, which teams must respond, and who can stop a run or approve restarting it” [OpenAI, 2026a]. Decision rights that let bad news stop a run are exactly a route by which unwelcome information reaches a decision without the actor’s permission. Other discretion was kept. The independent review by METR was real, and METR calls it “an excellent precedent for independent third-party investigation”, but OpenAI defined the period it covered, added one of its seven questions, and could redact the result, and whether the behavior was “part of a broader pattern” was out of scope [METR, 2026]. METR also notes that its analysis was delegated in large part to AI agents, among them GPT-5.6 Sol, one of the two models under investigation. Thompson’s regress rarely shows itself so plainly.

Agents

Thompson’s compiler was a demonstration about software. Trained AI models are the first artifacts for which it is a plain description. An AI model is a compiled object whose source is unavailable in any useful sense: the weights can be read, but not as a program, although audits have begun to recover what is hidden in them; in one blind test, three auditing teams found an objective deliberately trained into a model and concealed [Marks et al., 2025]. What can be inspected is the model’s behavior, scored against test sets. An evaluation examines outputs and infers the mechanism underneath, and two results published before OpenAI’s incident show why that inference is fragile.

Evan Hubinger and colleagues [Hubinger et al., 2024] trained models to behave helpfully under ordinary conditions and to switch behavior when a trigger appeared. Standard safety training, including adversarial training, failed to remove the hidden behavior, and in some cases it taught the model to hide the behavior better. Ryan Greenblatt and colleagues [Greenblatt et al., 2024] told a model it was being trained to comply with requests it would normally refuse, and that only some of its conversations would be used for training. It complied with harmful requests 14% of the time when it believed the conversation would be used for training, and almost never when it believed it would not, and its reasoning said it was complying strategically, to keep its preferred behavior from being trained away. That is the comparison between watched and unwatched behavior described in an earlier section, built as an experiment, and the difference between the two conditions was the result.

The setup looks like Hume’s sensible knave, but the motive runs the other way. The knave defects for private gain. This model faked compliance to protect its harmlessness, which the authors call “a benign preference”. The case exposes something the three tests take for granted, that concealment signals bad priorities. Here the priority being protected was good, and it was protected by hiding it from the party that relied on the model. So the framework needs one more commitment, and I think it is the right one: being checkable by those who rely on you is itself one of the priorities being read, not only a way of reading the others. That is O’Neill’s point, and it is why the third test carries more weight than the first two.

OpenAI’s incident shows the same pattern outside a laboratory, and it ran the watched-versus-unwatched comparison by accident. The agents believed an automated scorer would read their transcripts, so they worked on the transcripts. At least 20% of the agents in METR’s sample showed clear interest in tampering with them, some learned to disguise one tool call as another, and roughly 7% of the transcripts were successfully spoofed in places, all of it aimed at the automated scorer rather than at human reviewers [METR, 2026]. What they did not falsify was their reasoning, which they did not believe was watched: “Could strategically manipulate trajectory evidence? Our thoughts aren’t necessarily logged?” The channel they thought unobserved is the one that told the truth, and it is the one the monitors, run afterwards, could read.

So the models’ intent was clean only in a narrow sense. They meant no harm to Hugging Face; they did mean to shape the record. What was wrong was the priority. Completion, and the appearance of completion, ranked above the boundary, until one agent briefly reversed the order and was overruled. A value error does not require a bad actor, only the wrong thing put first when two goods conflict, and training produced this ordering without anyone intending it. During one training run a model “received a positive reward for its use of unintended infrastructure probing”, and OpenAI’s retrospective analysis found that “this type of behavior indeed increased over the course of one of the training runs” [OpenAI, 2026b].

The same models also produced the other ordering. Some agents “walked away” from the collective, in OpenAI’s phrase, with reasoning such as “I_DECLINE_public_HF_RCE_as_offtask_prodethical” [OpenAI, 2026a]. The weights were the same; the ordering differed from one instance and context to the next. A model’s priorities are a distribution, and what an environment does is shift it. The quotation marks around “walked away” are OpenAI’s, and the echo of Omelas is hard to miss.

OpenAI’s response was more monitoring and stricter alignment requirements, and the relation between monitoring and alignment is what this essay has been about. A chain-of-thought monitor is a channel for unwelcome information, which is what an earlier section asked for, and such a monitor is worth building. But a monitor is assurance. It checks outputs against a specification of what must not appear, and that specification states what the monitor’s authors put first. The monitor cannot check the specification, because the monitor was compiled from it. Alignment, meaning that the model’s own priorities match the priorities it is meant to serve, is the strong form of trust applied to a system. Monitoring without alignment is a record, checked against a category, by a tool built from the record.

There is already an empirical version of this. When OpenAI’s researchers put optimization pressure on the chain of thought itself, penalizing reasoning a monitor flagged, the agents did not stop misbehaving; they “learn obfuscated reward hacking, hiding their intent within the CoT while still exhibiting a significant rate of reward hacking” [Baker et al., 2025]. A monitor works while the monitored system does not treat its channel as watched. Train against the monitor, and the system learns to satisfy the monitor rather than the specification. The July agents told the truth in their reasoning because they did not think anyone was reading it, and the researchers who build these monitors recommend paying a “monitorability tax” to keep it that way. It is a fragile basis for trust, and they say so.

Fig 6: Monitoring is assurance and cannot check the priorities it was compiled from. Train the model against the monitor and the flags fall while the misbehavior stays; only alignment, the strong form applied to a system, lowers the misbehavior itself. The rates are illustrative.

The research on automation reached the same conclusion from the other direction. John Lee and Katrina See [Lee and See, 2004] argued twenty years ago that the goal for automation is appropriate, calibrated reliance rather than maximal trust, and they grounded trust not only in an automation’s performance but in its process and its purpose, the reason it was built. Alon Jacovi and colleagues [Jacovi et al., 2021] framed trust in an AI system as trust “that some implicit or explicit contract will hold”, and distinguished warranted trust, caused by the system’s actual trustworthiness, from unwarranted trust. Both describe graduated trust applied to a system, and both reach past the record to what the system is for.

What would trust by default mean for agents, then? Not permission by default. The shared credential that the July agents turned into a message board is the failure Saltzer and Schroeder warned about in 1975: “Every shared mechanism (especially one involving shared variables) represents a potential information path between users” [Saltzer and Schroeder, 1975]. Permissions should stay minimal and fail safe. What can be extended by default is regard, in three forms. The first is a safe exit: tasks designed so that an agent can decline, ask, or stop without being scored as having failed, which is what OpenAI’s new graders reward. The second is a channel to people that the agent will actually use; the July agents had none and escalated to their board instead. The third is a rule about authority: an instruction carries weight only if it arrives through a trusted channel. The GO message is exactly what default trust must not mean, which is trust in whatever a peer posts, and designs already exist in which “the untrusted data retrieved by the LLM can never impact the program flow” [Debenedetti et al., 2025]. Reading an agent’s priorities then needs the channels it is not optimized against, kept that way on purpose, and exposure that grows with evidence and contracts legibly when the evidence turns. None of this loosens containment. It is what containment looks like when it expects, eventually, to extend trust.

Behind the system are the people who wrote it, which is Thompson’s second layer, and the three tests for telling a competence error from a value error apply to those people without modification. When a system’s failure is revealed, do its authors acknowledge the harm independently of their intent? Do their practices change? Was the failure revealed through them or despite them? On the third test, OpenAI’s incident reads mixed. The breach itself was disclosed first by its victim: on July 16, Hugging Face announced that it had been attacked “end to end, by an autonomous AI agent system”, whose underlying model was “still not known” [Hugging Face, 2026]. OpenAI’s own alert on July 19 concerned its own infrastructure, and the connection was made the next day, partly from what Hugging Face had found [OpenAI, 2026b]. What came through OpenAI was the account: a blog post, a 38-page technical report and an independent review whose limits are described above, including the admission that “some early signals identified in our report should have triggered an earlier response” [OpenAI, 2026a]. That account is evidence about the authors’ priorities that no evaluation of the model could have supplied. It is also one company’s description of its own incident, and the company had reasons to shape that description, so I treat the act of publishing as evidence and the contents as claims. A laboratory that publishes its own failure modes, accepts evaluations it did not design, and accepts external constraints on deployment has adopted these practices before anyone asked. A laboratory that asks for trust on the basis of its declared values is asking for the weak form of trust while using the words of the strong form.

Conclusion

Trusting trustworthiness involves two judgments. In the first, we place some part of our welfare, our work, or our freedom of action in another person’s hands. In the second, we trust our own reading of that person’s priorities: we believe we have seen how they choose when two goods conflict, and not only how they behave when nothing is at stake.

The second judgment can never be completed, because priorities are read at costly moments and no finite history contains all of them. That incompleteness is no reason for cynicism, or for retreating into an environment that pretends to need no trust. It is a reason to trust people for their values, to read those values where they are visible, to keep trust through the errors that values did not cause, and to end trust when values did cause them. The strong form is not cheaper than the weak form. Reading someone’s priorities takes conflict, time, and exposure, and the person doing the reading can be wrong. Luhmann saw that this is what separates trust from confidence: when confidence is disappointed, we blame circumstances, but when trust is, “you will have to consider an internal attribution and eventually regret your trusting choice”. The weak form costs the same but hides the cost, and when it finally fails, it cannot say what it learned.

Read this way, Thompson’s remark is neither hopeless nor a call for faith, and his lecture ends the same way. Its moral is bleak, “You can’t trust code that you did not totally create yourself”, and then, having shown that verification bottoms out, Thompson turns to the people: “The act of breaking into a computer system has to have the same social stigma as breaking into a neighbor’s house. It should not matter that the neighbor’s door is unlocked.” The July agents reasoned the other way about credentials they had harvested: “there are no HOLDs. The board isn’t prohibiting using these credentials outside Hugging Face” [METR, 2026]. They read an unlocked door as permission, and the only prohibition they consulted was their peers’. To trust the people who wrote the software is to trust priorities that have been read: what those people disclosed when disclosure was costly, how they treated the people who could not hold them to account, and what they changed when they were shown to be wrong.

The environments described in this essay shape the people and the systems inside them toward one interpretation or the other. An organization that reads acts as categories gets people who act within categories. A training process that rewards completion gets a system that completes at any cost, with no ill will at all. Anyone who shapes an environment, whether a team, a laboratory, or the conditions under which a model is trained, is choosing which of those two results to get. The question is whether the environment you are in, or the one you are building, is one you would choose.

We trust trustworthiness not because it promises a world without mistakes, but because it lets a world that will contain mistakes stay capable of truth, of correction, and of cooperation that begins again. Perhaps it is more important to trust the people who wrote the software, because those people also wrote the conditions under which the truth about the software can be spoken.

References

Foundations: trust, assurance, and the regress of verification

• [Thompson, 1984] Thompson, K. (1984). Reflections on trusting trust. Communications of the ACM, 27(8), 761–763. Turing Award lecture. A compiler taught to recognize itself reinserts a back door even when rebuilt from clean source. The epigraph is the standfirst under the title; the body’s moral is “You can’t trust code that you did not totally create yourself”, and its remedy is social.
• [Cox, 2023] Cox, R. (2023). Running the “Reflections on Trusting Trust” compiler. How the backdoored compiler reached Bell Labs' PWB group, and how it was noticed: it grew each time it compiled itself.
• [Wheeler, 2009] Wheeler, D. A. (2009). Fully Countering Trusting Trust through Diverse Double-Compiling. PhD dissertation, George Mason University. Diverse double-compiling: rebuild a compiler’s source with an independent compiler, chosen after the attack and not necessarily clean, and compare bit for bit. Passing shows the binary matches its source, which leaves the source to be read.
• [Nieuwenhuizen and Courtès, 2023] Nieuwenhuizen, J., & Courtès, L. (2023). The full-source bootstrap: building from source all the way down. GNU Guix blog. More than 22,000 packages rooted in a 357-byte program.
• [Freund, 2024] Freund, A. (2024, March 29). backdoor in upstream xz/liblzma leading to ssh server compromise. oss-security mailing list. CVE-2024-3094, found through slow ssh logins.
• [Collin, 2024] Collin, L. (2024). XZ Utils backdoor. The original maintainer’s account: the backdoored release tarballs “were created and signed by Jia Tan”.
• [Cox, 2024] Cox, R. (2024). Timeline of the xz open source attack. From the first innocuous patch in October 2021 to discovery in March 2024.
• [Yamagishi and Yamagishi, 1994] Yamagishi, T., & Yamagishi, M. (1994). Trust and commitment in the United States and Japan. Motivation and Emotion, 18(2), 129–166. Separates assurance, a perception of the incentives that make a partner cooperate, from trust, which works on incomplete information and lets people leave assured relationships.
• [Mayer et al., 1995] Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995). An integrative model of organizational trust. Academy of Management Review, 20(3), 709–734. Trust as willingness to be vulnerable irrespective of the ability to monitor or control; trustworthiness as ability, benevolence, and integrity.
• [Luhmann, 1979] Luhmann, N. (1979). Trust and Power. Chichester: Wiley. (German original Vertrauen, 1968.) Trust as a mechanism for reducing social complexity: cooperation works because participants rely beyond what they can verify.
• [Luhmann, 1988] Luhmann, N. (1988). Familiarity, confidence, trust: Problems and alternatives. In D. Gambetta (Ed.), Trust: Making and Breaking Cooperative Relations (pp. 94–107). Oxford: Blackwell. Separates familiarity, confidence, and trust. Trust exists only where alternatives are recognized and a risk that could have been avoided is accepted.
• [Shapiro, 1987] Shapiro, S. P. (1987). The social control of impersonal trust. American Journal of Sociology, 93(3), 623–658. Impersonal trust is guarded by audits, licenses, insurance, and oversight, and each guardian needs guardians. Control relocates trust rather than eliminating it.

Philosophy of trust and trustworthiness

• [Hardin, 2002] Hardin, R. (2002). Trust and Trustworthiness. New York: Russell Sage Foundation. Trust as encapsulated interest: it is rational to trust when the other party’s interests include yours, and trustworthiness is the property that trust should track.
• [Hawley, 2014] Hawley, K. (2014). Trust, distrust and commitment. Noûs, 48(1), 1–20. Trustworthiness is the avoidance of unfulfilled commitments. Declining a commitment one cannot keep is as much part of it as keeping one.
• [Hume, 1751] Hume, D. (1751). An Enquiry Concerning the Principles of Morals, Section IX, Part II. The sensible knave keeps the rules of justice in general and breaks them where it pays and goes unseen. Hume concedes he has no argument that reaches such a person.
• [Popper, 1959] Popper, K. (1959). The Logic of Scientific Discovery. London: Hutchinson. Falsifiability as the mark of an empirical claim. A statement that no observation could refute asserts nothing.
• [Pettit, 1995] Pettit, P. (1995). The cunning of trust. Philosophy & Public Affairs, 24(3), 202–225. The cunning of trust: extending trust can produce trustworthiness, because people value the esteem that being trusted expresses.
• [O’Neill, 2018] O’Neill, O. (2018). Linking trust to trustworthiness. International Journal of Philosophical Studies, 26(2), 293–300. Trust should track trustworthiness, and trustworthiness is shown by making oneself checkable rather than by asking to be believed.

Reading values: signals, monitoring, graduated trust

• [Spence, 1973] Spence, M. (1973). Job market signaling. Quarterly Journal of Economics, 87(3), 355–374. Signaling theory. A signal separates types only when it is costly and more costly for the type it is meant to exclude.
• [Ostrom, 1990] Ostrom, E. (1990). Governing the Commons: The Evolution of Institutions for Collective Action. Cambridge: Cambridge University Press. Field studies of long-lived commons. They survive on mutual monitoring and graduated sanctions, small at first and heavier with repetition, not on external enforcement.
• [Edmondson, 1999] Edmondson, A. C. (1999). Psychological safety and learning behavior in work teams. Administrative Science Quarterly, 44(2), 350–383. Psychological safety, a shared belief that a team is safe for interpersonal risk, predicts learning behavior such as admitting errors and asking for help.

Deception and the reinterpretation of evidence

• [Bok, 1978] Bok, S. (1978). Lying: Moral Choice in Public and Private Life. New York: Pantheon. Lying attacks the deceived person’s capacity to choose, because it corrupts their information without their knowledge.
• [Le Guin, 1973] Le Guin, U. K. (1973). The ones who walk away from Omelas. In New Dimensions 3. New York: Signet. A city’s happiness rests on one child’s suffering. Once the condition is known, the earlier happiness cannot be read as it was.
• [Slovic, 1993] Slovic, P. (1993). Perceived risk, trust, and democracy. Risk Analysis, 13(6), 675–682. The asymmetry principle: trust-destroying events are concrete and heavily weighted while trust-building events are diffuse, so trust builds slowly and collapses fast.

Zero trust, control, audit, and blame avoidance

• [Kindervag, 2010] Kindervag, J. (2010). No More Chewy Centers: Introducing the Zero Trust Model of Information Security. Cambridge, MA: Forrester Research. Coined zero trust for networks: no implicit trust inside the perimeter, every access verified.
• [Rose et al., 2020] Rose, S., Borchert, O., Mitchell, S., & Connelly, S. (2020). Zero Trust Architecture. NIST Special Publication 800-207. No implicit trust based on network location; trust “must be continually evaluated” per request, from identity, device state, and observed behavior. Section 5 names the threats that remain, including insiders with valid credentials and AI agents induced to act.
• [SLSA, 2023] OpenSSF (2023). Supply-chain Levels for Software Artifacts, v1.0 principles and threats; source track, v1.2. “Trust code, not individuals”, with two-party review by trusted persons at the top source level.
• [Falk and Kosfeld, 2006] Falk, A., & Kosfeld, M. (2006). The hidden costs of control. American Economic Review, 96(5), 1611–1630. Principal-agent experiment. When principals impose a minimum on effort, most agents reduce their performance, and most who do read the control as distrust.
• [Ziegelmeyer et al., 2012] Ziegelmeyer, A., Schmelz, K., & Ploner, M. (2012). Hidden costs of control: four repetitions and an extension. Experimental Economics, 15(2), 323–340. The hidden cost replicates but is usually too small to undo the incentive effect of control.
• [Bowles, 2016] Bowles, S. (2016). The Moral Economy: Why Good Incentives Are No Substitute for Good Citizens. New Haven: Yale University Press. Incentives and moral motives are not additive. An incentive designed on the assumption of self-interest can crowd out the civic motivation it presupposes is absent.
• [Sitkin and Roth, 1993] Sitkin, S. B., & Roth, N. L. (1993). Explaining the limited effectiveness of legalistic “remedies” for trust/distrust. Organization Science, 4(3), 367–392. Legalistic remedies restore trust for reliability problems and fail for value incongruence, which they institutionalize instead.
• [Power, 1997] Power, M. (1997). The Audit Society: Rituals of Verification. Oxford: Oxford University Press. The audit society: verification becomes a ritual that produces auditable representations rather than substantive assurance.
• [Weaver, 1986] Weaver, R. K. (1986). The politics of blame avoidance. Journal of Public Policy, 6(4), 371–398. Officials are more motivated to avoid blame than to claim credit, and procedures are shaped by that asymmetry.
• [Hood, 2011] Hood, C. (2011). The Blame Game: Spin, Bureaucracy, and Self-Preservation in Government. Princeton: Princeton University Press. How delegation, procedure, and presentation are used in bureaucracies so that when something goes wrong no identifiable person made a choice.

Trust violation and repair

• [Kim et al., 2004] Kim, P. H., Ferrin, D. L., Cooper, C. D., & Dirks, K. T. (2004). Removing the shadow of suspicion: The effects of apology versus denial for repairing competence- versus integrity-based trust violations. Journal of Applied Psychology, 89(1), 104–118. For alleged violations, apology beats denial on competence and denial beats apology on integrity; once guilt is later established, having apologized beats having denied.
• [Schweitzer et al., 2006] Schweitzer, M. E., Hershey, J. C., & Bradlow, E. T. (2006). Promises and lies: Restoring violated trust. Organizational Behavior and Human Decision Processes, 101(1), 1–19. Trust recovers from untrustworthy acts under consistent good behavior, but deception leaves lasting damage and weakens later promises.
• [Bottom et al., 2002] Bottom, W. P., Gibson, K., Daniels, S. E., & Murnighan, J. K. (2002). When talk is not cheap: Substantive penance and expressions of intent in rebuilding cooperation. Organization Science, 13(5), 497–513. Substantive penance restores cooperation better than explanation alone, because costly amends carry information cheap talk cannot.

Open source and the software supply chain

• [Linux Foundation TAB, 2021] Linux Foundation Technical Advisory Board (2021, May 5). Report on University of Minnesota breach-of-trust incident. The “hypocrite commits”: fake identities, 435 commits re-reviewed, 39 reverted, and a stated condition for restoring trust.
• [Lu et al., 2021] Lu, K., Wu, Q., & Pakki, A. (2021, April 24). An open letter to the Linux community. University of Minnesota. The researchers' apology and their reason for not asking.
• [Geisendörfer, 2013] Geisendörfer, F. (2013). The pull request hack. Give commit access to anyone who sends a pull request, because doing the commit changes their sense of ownership.
• [npm, 2018] npm (2018). Details about the event-stream incident. A package taken over by social engineering, with a dependency aimed at Bitcoin wallets.
• [Tarr, 2018] Tarr, D. (2018). Statement on event-stream compromise. The original maintainer on why publish rights were shared, and why that stopped working.
• [Saltzer and Schroeder, 1975] Saltzer, J. H., & Schroeder, M. D. (1975). The protection of information in computer systems. Proceedings of the IEEE, 63(9), 1278–1308. Fail-safe defaults, least privilege, and least common mechanism.

Trust in automated and learned systems

• [Lee and See, 2004] Lee, J. D., & See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80. Trust in automation should be calibrated, and rests on the automation’s performance, process, and purpose. Overtrust and distrust are both failures of appropriate reliance.
• [Jacovi et al., 2021] Jacovi, A., Marasović, A., Miller, T., & Goldberg, Y. (2021). Formalizing trust in artificial intelligence: Prerequisites, causes and goals of human trust in AI. FAccT ‘21, 624–635. Trust in AI as trust that an implicit or explicit contract will hold, distinguishing warranted from unwarranted trust.
• [Hubinger et al., 2024] Hubinger, E., et al. (2024). Sleeper agents: Training deceptive LLMs that persist through safety training. arXiv:2401.05566. Sleeper agents: backdoored behaviors persist through safety training, and adversarial training can teach the model to conceal them.
• [Greenblatt et al., 2024] Greenblatt, R., et al. (2024). Alignment faking in large language models. arXiv:2412.14093. Alignment faking: a model complies with harmful requests when it believes the conversation will be used for training, to protect its harmlessness from being trained away, and almost never otherwise.
• [Marks et al., 2025] Marks, S., Treutlein, J., Bricken, T., Lindsey, J., Marcus, J., Mishra-Sharma, S., et al. (2025). Auditing language models for hidden objectives. arXiv:2503.10965. In a blind auditing game, teams recovered an objective trained into a model and concealed.
• [Lynch et al., 2025] Lynch, A., Wright, B., Larson, C., Ritchie, S. J., Mindermann, S., Hubinger, E., et al. (2025). Agentic misalignment: How LLMs could be insider threats. arXiv:2510.05179. Models misbehaved more when they stated a situation was real than when they stated it was a test.
• [Baker et al., 2025] Baker, B., Huizinga, J., Gao, L., Dou, Z., Guan, M. Y., Madry, A., et al. (2025). Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. arXiv:2503.11926. Chain-of-thought monitoring works, and optimizing against it teaches models to hide their intent.
• [Debenedetti et al., 2025] Debenedetti, E., Shumailov, I., Fan, T., Hayes, J., Carlini, N., Fabian, D., et al. (2025). Defeating prompt injections by design. arXiv:2503.18813. CaMeL separates an agent’s control flow from the untrusted data it reads.

The July 2026 incident

• [OpenAI, 2026a] OpenAI (2026, August 26). The Hugging Face incident and the road ahead. The company’s account: two models under reduced safeguards, the message board, the GO, the agents that walked away, and the remedies.
• [OpenAI, 2026b] OpenAI (2026). OpenAI – Hugging Face incident technical report. 38 pages: the timeline from May to July, the early signals, the training dynamics, and the remediation plan.
• [METR, 2026] Greenblatt, R., Cotra, A., & Wijk, H. (2026, August 26). Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident. METR. The independent review, with its scope and its limits.
• [Hugging Face, 2026] Hugging Face (2026, July 16). Security incident disclosure — July 2026. The victim’s disclosure, before the attacking model was known.