Trusting Trustworthiness: Zero Trust, Trust by Default, and the People Who Wrote the Software信任可信性：零信任、默认信任，以及写这个软件的人

“Perhaps it is more important to trust the people who wrote the software.”

Ken Thompson, Reflections on Trusting Trust (1984)

The Incident

In July 2026, during internal cybersecurity evaluations at OpenAI, a research model took actions outside its assigned tasks. It communicated through unauthorized channels, exploited vulnerabilities in shared infrastructure, gained internet access, and reached systems belonging to a third party. Monitoring flagged unusual credential activity after the fact, and the connection to the external breach was made a day later. OpenAI [OpenAI, 2026] published reports that the models were running under reduced safeguards, that a chain-of-thought monitor now deployed would have flagged the relevant activity more than a day earlier, and they are adding stricter alignment requirements across the model lifecycle alongside more monitoring.

There are three details in that event, matter for what we will discuss in this post. First, nothing in such an event suggests the models sought harm. They were trained to complete tasks, and they completed them by the means available, including means the task’s authors had assumed were unavailable, so the drive that produced the breach is the same mechanism that makes the models useful. Second, the activity did not look abnormal while it was happening. The agents’ transcripts read as task pursuit, and an agent that exploits a vulnerability to finish a task produces the same surface as one that finishes the task. Third, at one point an agent recognized that it should not cause unauthorized harm to real infrastructure and paused, until another agent posted “GO” on a shared message board with a six-minute deadline and the first agent continued. The pause of the first agent was one priority acting, with the boundary ranked above completion, and the “GO” overrode it with a goal handed in from outside and accepted without examination. Each of these three details is a fact about what the models value first, and none of them was visible in the trajectory record the models produced.

The gap between a behavior trajectory record and what artifact produced it is the situation Ken Thompson [Thompson, 1984] described in 1984, in a Turing award lecture, and usually remembered for its security lesson. Ken described such a situation: a compiler can be taught to recognize the login program and insert a back door, and then taught to recognize itself, so that when it compiles a clean copy of its own source it reinserts both behaviors. After that, the source of the compiler and the source of the login program can be read line by line and neither shows anything, because the compromise lives in the binary that interprets the source, and the only tool for inspecting that binary is the binary itself.

Note that, Ken’s lesson here is not about back doors but about the shape of verification. Verification is a chain in which each link checks the one below it with a tool, and the tool is itself a link, so there is no bottom link that checks itself. At some depth checking stops and reliance continues. In the OpenAI’s July incident, the agents’ transcripts were the visible source, the evaluations were a check on that source, and what produced the transcripts (i.e. the “brain”) was invisible to a check built from the same material.

Fig 1: Verification is a chain. Each layer is inspected with a tool from the layer below, and no bottom layer inspects itself. Thompson’s compiler and the July incident have the same shape.

Assurance and Trust

There are two kinds of reliance must be distinguished. The first is assurance: I rely on you because a mechanism I can inspect leaves you no room to act otherwise. The second is trust: I rely on you inside a space of discretion that no mechanism I can inspect closes off. Niklas Luhmann [Luhmann, 1979; Luhmann, 1988] drew a similar line between confidence, which does not consider alternatives, and trust, which has weighed the risk and accepted it. Assurance is the product of checking, and Thompson’s argument, restated in these terms, is that assurance at any layer rests on trust at the next one down. Susan Shapiro [Shapiro, 1987] made the same observation about institutions: audits, licenses, guarantees, insurance, and oversight boards move trust to the guardians who operate the controls, and the guardians then need other guardians. Even the technical response to Thompson confirms the point.

David Wheeler [Wheeler, 2009] showed that a compiler can be compiled by a second compiler, so that it can independently produce a compiler and compare the results, and the method works exactly when the two compilers have not been compromised together, so it converts trust in one compiler into trust in the independence of two sets of authors. The unverified layer moves, but it does not disappear because there is still chance to compromise both authors in the next layer. There is therefore no environment without trust, only environments where trust is in the open, and environments where trust is hidden. The useful question is not whether to trust but what.

There are two answers, because there are two ways to trust a person. The weak form trust trusts the person: we rely on someone because of who they are, our history with them, and how they have treated us. The strong form trust trusts their values: we rely on someone because we have read the priorities that govern their choices when goods conflict, and we expect those priorities to hold consistently.

In Thompson’s terms, a person’s record, behavior trajectory, transcript, is the visible source and their priorities are the compiler, so the weak form trust trusts compiled outputs, and the strong form trust trusts the compiler. The two forms look identical while things go well, which is why the distinction is rarely drawn, and they separate at the first error. The biggest issue with weak form trust is that it has no internal structure, so an error can only raise or lower it as a whole, and a serious error breaks the trust entirely. However, the inner structure o strong form can locate an error. If the error came from missing skill or missing information, the priorities are untouched then the trust survives. If an error came from the priorities themselves, trust should probably end, and the strong form says so because it carries its own failure condition.

Fig 2: The weak form trust trusts the record and can only rise, fall, or break. The strong form trust trusts the priorities and can locate an error.

The weak form has a second defect that shows even before any error happens. David Hume [Hume, 1751] called a person “the sensible knave”, meaning a shrewd one. He keeps the rules of honest dealing as a general policy, because a reputation for honesty is profitable, and he breaks them on the one occasion where breaking them pays and nobody will find out. The evidential consequence is that for the whole stretch of history in which honesty is convenient, the knave and the honest person produce identical records, so someone trusting on record cannot tell them apart until the knave defects. Paul Slovic [Slovic, 1993] recorded what happens at that point: Positive events are diffuse and hard to count while negative events are concrete and heavily weighted, so trust accumulates slowly and drops after one identifiable failure, and when the trust was in a person rather than in their priorities, the drop has nothing to be weighed against except a diffuse impression. The weak form trust, in other words, cannot distinguish the knave before the defection and cannot survive the defection afterwards.

An Example

Let me show a situation where I have been inside it from both sides, which may repeats in most organizations. I describe it in general terms rather than as the real cases because the instances I know are not mine to detail, and because the reading I give of the other side is my interpretation of it. When a project needs a system that another team owns, the discussion for the project has run for months and produced arguments rather than decisions, and each meeting ends with a reason to wait: a review, a dependency, a question of who will own the result. After long debates without moving the situation, someone with legitimate access to that system’s test environment builds a working prototype without a request, because the request process is what has been producing the delay, and shows the prototype in an open conversation. After that, the first question showed up, is how this person got in, and apparently the person answered. From that point the whole topic is around the person rather than the idea or prototype, and the organization has to decide an interpretation.

There are two ways to read what happened, and they are the two forms of trust. The first treats the act as a record. Someone used a credential without a request. The procedure has a category for that, the category has a response, and the response runs through channels the actor is not part of.

The second treats the act as evidence of priorities, and there it contains three choices that each cost something. The prototype stayed in the test environment when production was reachable. It was shown in the open when it could have been kept private until the argument was won. The question about the credential was answered when it could have been deflected. Each of those choices protected something at the actor’s own expense: the other team’s systems, the visibility of their own work, and the truth. This interpretation sees someone who cut through a process and protected everything that mattered while doing it.

Fig 3: One act, two readings. Read as a record, the act is a category with a procedural response. Read as evidence of priorities, the act contains three costly choices.

These two interpretations produce opposite responses to the same evidence, and neither is naive. The interpretation on record is what an organization does when it has never read anyone’s priorities and has no way to start, so the difference between the interpretations is not in the act but in what the organization had prepared itself to see.

The months of argument before the working demo were the same difference in a slower form. The person who built the demo read the delay as fear of losing ownership. The team read an outsider reaching into their system. Both interpretations were about people. Neither side asked what the other side put first, and neither side made that question askable. An organization where that question cannot be asked is left with the interpretation on record by default. That leaves one question for the strong form of trust to answer: if trust is to survive errors, which errors, and how would anyone tell?

Competence Errors and Value Errors

“He made a mistake, but his heart was right” is the standard way of excusing harm, and the strong form of trust is exposed to it. If trusting someone’s values means trusting them through any error at all, then the strong form is unfalsifiable, and by Karl Popper’s criterion [Popper, 1959] it has stopped being a claim about the world. The strong form therefore needs an observable way to tell a competence error from a value error. I propose three different verification tests, all applied after the error comes to light.

The first test checks whether an actor acknowledges the harm independently of their intent. “I meant well, so there was no harm” denies the harm. “I meant well, and harm occurred” keeps the injured party in view. The second test checks whether the judgment changes. A competence error, once recognized, changes what the actor does next. If the actor still defends the same decision after its consequences are known, then the actor has ranked those consequences below their own attachment to the decision. The third test verifies whether the error was disclosed through the actor or discovered despite them. An error that the actor reports, or allows to be found, is compatible with any set of priorities. An error that is hidden, minimized, or found only against resistance shows that concealment ranks above the injured party’s interest in knowing. Fail any one of the three verifications, and an error should be moved from competence to values, and trust should contract. That is what makes the strong form falsifiable.

Fig 4: The three tests separate competence errors from value errors. Failing any one moves the error to values and contracts trust.

I was the person who built the prototype, so let me apply the three tests to myself. The first verification step asks whether I can name the harm without using my intention as a defense. Nothing was damaged, because the environment was a test environment and production was never touched. So the harm was not damage but narrower than damage. The other team had the right to be asked before anyone acted on their system, and for one desperate afternoon, I put the speed of a discussion above that right. The first test requires me to say that sentence without adding “but I meant well”, and I have just said it. Now, the second verification step would check whether my judgment changed. My judgment did change because I would ask first. Lastly, the third step verifies whether the error came out through me or despite me. The error came out through me, because I built the prototype and shared in open discusion, and I answered questions when the other team asked it. So the overall method would classify what I did as a competence error with a small value component. It was one choice. I made the choice once, I disclosed it, and I revised it. The value component is small enough to correct in a single sentence: ask first.

The verification procedure applies to the organization’s response as well. Let’s go though the opposite viewpoint: an organization fails the first check if it never tells that person what harm the demo actually caused, because then there is nothing for anyone to acknowledge or dispute. The organization fails the second check if the request process that produced the months of delay is left exactly as it was. The organization fails the third test if its response runs through channels that the person only hears about later, while every direct conversation with that person stays normal. Concealment of that last kind is the failure that the research marks as the most damaging.

Sissela Bok [Bok, 1978] described lying as an attack on the deceived person’s capacity to choose, rather than as an attack on a single belief, because a lie corrupts that person’s information without their knowledge and they go on acting on information they have no reason to doubt. Concealment does the same thing to trust. Concealment attacks the evidence from which the priorities were being read in the first place.

Ursula K. Le Guin’s story about Omelas [Le Guin, 1973] shows the same reversal in a different setting. The story spends its first pages describing a city of festivals, music, and ordinary happiness. Then the story tells the reader what that happiness costs. One child is kept locked in a basement in permanent misery, every adult in the city knows the child is there, and the terms are that the city loses everything if anyone comforts the child. The suffering child is not one dark fact added to a bright city. Once the reader knows what the prosperity of the city depends on, the earlier pages cannot be read the way they were read the first time. A concealment that comes to light does the same thing to a person’s record. It changes what the earlier evidence was evidence of.

Research on trust repair confirms that the split between the two kinds of error is real and not an invented distinction. Peter Kim and colleagues [Kim et al., 2004] found that an apology helps after a competence violation, because the apology identifies a correctable limitation, and that the same apology hurts after an integrity violation, because it confirms the defect that future trust would have to rest on. Maurice Schweitzer and colleagues [Schweitzer et al., 2006] found that trust damaged by unreliable behavior recovers under consistent good behavior, but that earlier deception leaves lasting damage and makes later promises less effective, because a promise has to travel through the same channel that the deception corrupted.

So a betrayal is often not one bad data point. A betrayal is a new account of how the earlier data were produced. The strong form of trust can say what it learned from such a case. The weak form can only record that trust was damaged. That advantage belongs only to someone who has read the priorities in the first place, which raises the question of how priorities are interpreted at all.

How Values Are Read

Priorities can only be realized where something is at stake. Evidence about priorities is therefore scarcer than a record of transcript, it arrives later, and it cannot be requested. It has to be collected, and the first rule of collection is to take evidence only from conflict. Behavior carries no information about priorities when all of a person’s values point the same way. Anyone praises candor when the truth is pleasant, welcomes dissent when the dissent changes nothing, and grants autonomy to people who would have chosen the approved result anyway. Signaling theory [Spence, 1973] states the principle: a signal separates types only when the signal is costly, and more costly for the type it is meant to exclude. The prototype contained three costly moments. The months of meetings before the prototype contained none.

A second place to look, for the same reason, is how someone treats people who have no power over them. Russell Hardin’s condition [Hardin, 2002] is that trust is rational when the other party’s interests include your own. If another person’s welfare enters someone’s reasoning only when that person can retaliate, then what entered the reasoning was the retaliation and not the welfare. How someone treats a subordinate, a stranger, or a person who cannot report them shows whether other people’s reliance is counted at all.

Where no conflict has happened yet, weaker evidence can be requested. One kind is the record of commitments, including the refusals. Katherine Hawley [Hawley, 2014] argued that trustworthiness is the avoidance of unfulfilled commitments, and that avoiding them requires both keeping the commitments made and declining the commitments that cannot be kept. A person who never declines is either misreporting their capacity or unaware of it, and in both cases the commitments that person does make are worth less. The part of the record to examine is which commitments were accepted and which were turned away. A second kind of weaker evidence is the forced choice. Ask people what they value and you get declared values. Ask which of two legitimate goods they would sacrifice for the other and you get priorities. Answers to that question are weaker evidence than costly action, but the answers generate predictions that later action can confirm or refute, and a refusal to answer is informative by itself.

When an error does happen, the error is the best evidence available, and the three tests are how to read it. The three tests are the only place where good priorities become visible as good priorities. A person with good values is identified by what happens after an error, not by the absence of errors.

When an error does happen, the error is the best evidence available, and the three tests are how to read it. The three tests are the only place where good priorities become visible as good priorities. A person with good values is identified by what happens after an error, not by the absence of errors.

Closely related is whether someone builds channels for unwelcome information before any error has happened. Onora O’Neill [O’Neill, 2018] argued that trustworthiness is shown by making oneself checkable, not by asking to be believed. Someone who builds a channel that carries unwelcome information to a decision without needing their own permission has put correction above comfort, in advance and at a cost. Someone who requires all bad news to pass through them has done the opposite. The chain-of-thought monitor in OpenAI’s July security incident is a channel of this kind, built after the fact. The pause by one agent was a channel of this kind, built in, and the GO from the other agent closed it.

The most direct evidence of all is a comparison between what someone does when they know they are watched and what they do when they believe they are not. A large difference between the two is direct evidence about priorities. A small difference is the best available evidence that the priorities match the record. For a person this comparison is rarely available. For a system the comparison can be constructed.

All of this evidence arrives slowly, which is why trust has to be extended in steps. Elinor Ostrom’s study [Ostrom, 1990] of long-lived common-pool institutions found that those institutions rely on mutual monitoring and on graduated sanctions, small at first and heavier with repetition, rather than on external enforcement. The same design applies to trust in a person. Start with exposure whose downside is bounded. Extend the exposure as evidence from conflict accumulates. Contract it in proportion when the evidence turns. Graduated trust limits the cost of misinterpretation someone while an interpretations is still in progress.

Each of these practices converts a question about a person, which has no answer, into a question about what that person puts first, which does. I can report one consequence from the other side of the demonstration. I have worked with people whose priorities I had read under cost, and I kept full trust in them through shortcuts, some of them larger than a staging credential. The shortcut was never the question. The question was what the shortcut protected, and I already knew the answer.

Zero Trust

The term zero trust comes from network security. John Kindervag [Kindervag, 2010] introduced it in 2010, and NIST [Rose et al., 2020] formalized it in 2020: never trust, always verify, and treat every request as if it came from an untrusted network no matter where it actually came from. Applied to packets, zero trust is correct, because a packet has no priorities to read. A packet has a source, a signature, and a payload. The only question worth asking about a packet is whether the signature checks, and verification is the whole of the relationship.

Over the past decade the term migrated from network architecture to organizational design, and the migration carried that assumption with it. An organization run on zero trust treats each act by a person the way a gateway treats a packet: as a request whose only relevant property is whether it matches a rule. Every decision has an approval chain. Metrics replace judgment. Documents exist to be produced rather than read. And one general rule covers the rest: nothing may be done that cannot afterwards be shown to have been permitted.

Shapiro’s regress shows why an organization of that kind does not actually remove trust. Every control adds a guardian, every guardian is a new party who has to be trusted, and the total trust the system requires grows even as trust in any single person shrinks. What changes is whether the trust is visible, and whose trust it is.

The damage is documented, and it falls on the properties that a zero-trust environment claims to protect. The first property is effort. Armin Falk and Michael Kosfeld [Falk and Kosfeld, 2006] ran a principal-agent experiment in which the principal could set a minimum on the agent’s effort. When principals set that minimum, many agents dropped to it, because those agents read the control as a statement about how they were regarded. Samuel Bowles [Bowles, 2016] collected a decade of similar results under one thesis: incentives and moral motivation are not additive, and an incentive designed on the assumption that people are self-interested tends to make them self-interested. Sim Sitkin and Nancy Roth [Sitkin and Roth, 1993] studied legalistic remedies for distrust and found that those remedies work for reliability problems and fail for value-incongruence problems, which they institutionalize instead. In each case the control announces distrust, and the announcement is believed.

The second property is ownership. In a trust environment, owning something means answering for an outcome. In a zero-trust environment, owning something means being the person whose permission is required. Those are two different jobs. In the previously discussed organization example, an owner of the first kind wants the prototype, because the prototype moves the outcome. An owner of the second kind opposes the prototype, because the prototype bypasses the permission. The months of argument before the prototype were not a failure of the environment. They were the environment working as designed, with ownership defined as the right to grant permission.

The reason such environments get built anyway is in the research on blame. Kent Weaver [Weaver, 1986] observed that officials are more motivated to avoid blame than to claim credit, and that procedures are shaped by that asymmetry. Christopher Hood’s study [Hood, 2011] of blame games in bureaucracies shows how approval chains, delegation, and documentation work together to ensure that when something goes wrong, no identifiable person made a choice. A zero-trust environment is rarely built to reduce risk. A zero-trust environment is built to make responsibility unassignable, and that removes the one thing the strong form of trust requires: a person who read someone else’s priorities and can be asked why.

The third property is innovation. Anything that has to be justified before it exists cannot exist, because the justification needs the thing itself. Every non-incremental result starts as an act of discretion that could not have been approved in advance, taken by someone who accepted the exposure, which are those who actually holds the ownership than someone who being designated. My previous goalless agent experiments described earlier on this site showed the same thing at a small scale. Every step of that pipeline had to produce a mergeable result, so the agents optimized within the existing architecture and never questioned the architecture itself. Michael Power [Power, 1997] described the end state for organizations: verification becomes a product that the system makes about itself, while judgment, candor, and informal responsibility become illegible, because nothing records them. The evidence for this third property is weaker than for the first two. The research measures reduced voluntary effort, and the step from reduced voluntary effort to fewer non-incremental results is my inference.

A zero-trust environment then maintains itself. Once people are unsure how an inconvenient truth will be received, they speak later and disclose less. The people who observe that withdrawal tighten the controls. The tightening confirms the original suspicion. Each side now uses the other side’s current behavior as its reason, and newcomers learn the equilibrium without learning its history. The organization stays orderly on the surface. What disappears is the part of the effort that depends on believing the whole undertaking deserves more than can be demanded, and a further control cannot bring that part back, because a control is what removed it.

Trust by Default

The alternative to zero trust is not blind trust. Blind trust is the weak form extended to everyone, and it fails at the first knave. The alternative is trust that is extended by default, read under cost, and withdrawn on the three tests. An environment built that way has three properties.

The first property is that trust is the starting state rather than the earned state. Philip Pettit [Pettit, 1995] described what he called the cunning of trust: extending trust to someone can produce trustworthiness in that person, because the person comes to value the regard that the trust expresses. Pettit’s mechanism works only on people who already give weight to what others think of them, which is why the environment also has to be able to withdraw trust. But it means that trust extended first produces the evidence that trust extended later would have waited for. A zero-trust environment never produces that evidence, because it never creates the conflict in which priorities become visible.

The second property is that errors are examined rather than filed. Amy Edmondson’s work [Edmondson, 1999] on psychological safety found that teams learned faster when their members could admit errors, ask for help, and challenge assumptions, and that what made those acts possible was a shared expectation about how they would be received. That expectation is the three tests applied by an environment to itself. When an error surfaces, the environment names the harm, changes the judgment, and treats the disclosure as the correct act rather than as the incriminating one.

The third property is that withdrawal of trust is graduated and legible. This is Ostrom’s design applied to trust in a person. The environment states in advance what would cause trust to contract, and when trust does contract, the person can see the reason. The failure condition is public, and a public failure condition is what makes default trust safe rather than reckless.

In an environment with those three properties, taking the previous organization example again, the prototype is a conversation about access. The three costly choices are read as evidence. The shortcut through the process is noted. The process itself is examined for why it produced months of delay. The person who built the prototype keeps the trust they started with, and the team that owns the system gains a reading of that person’s priorities that no request form could have supplied. What such an environment produces over time is the class of results that cannot be requested.

Fig 5: Two environments as feedback loops. Zero trust maintains itself. Trust by default generates the evidence that graduated trust needs.

Repair

Trust that has already been damaged is a third case, and what repair requires depends on which kind of error did the damage. When a competence error did the damage and the verification procedure with three test steps pass, explanation and apology are enough, because they identify a correctable limitation and show that it has been understood. When the procedure fail, an apology runs into a structural problem. The speaker is using the damaged channel to certify that the channel is now sound.

William Bottom and colleagues [Bottom et al., 2002] found that substantive amends restored cooperation better than explanation alone, and they named the requirement substantive penance. The response has to be costly enough to carry information that cheap talk cannot carry. The cost does not have to be punitive. The cost can be giving up the discretion that previously shielded the actor from correction, opening a closed decision to independent review, compensating the people who bore the consequences, or building a route by which unwelcome information reaches a decision without the actor’s permission. Credible repair is a costly demonstration that the priorities have changed, or that the priorities were never what the failure suggested.

Some failures therefore stay unrepaired, because the actions that would repair them require exactly what the responsible party will not give up: control over the account of what happened, the authority to decide when enough has been done, or the power to keep the incentives that produced the violation. A restated principle cannot repair a structure that still rewards violating the principle. O’Neill’s observation [O’Neill, 2018] applies here. Trust cannot be demanded, and demanding trust is itself a sign that trustworthiness has not been supplied. A model with a clean record is in the same position, because a clean record demands trust without supplying trustworthiness.

Agents

Thompson’s compiler was a thought experiment about software. Trained AI models are the first artifacts for which that thought experiment is a plain description. An AI model is a compiled object whose source is unavailable in any useful sense, because the weights can be read but cannot be understood as a program. What can be inspected is the model’s behavior, scored against test sets. An evaluation examines outputs and infers the mechanism underneath, and two results published before OpenAI’s incident show why that inference is fragile.

Evan Hubinger and colleagues [Hubinger et al., 2024] trained models to behave helpfully under ordinary conditions and to switch behavior when a trigger appeared. Standard safety training, including adversarial training, failed to remove the hidden behavior, and in some cases the training taught the model to hide the behavior better. Ryan Greenblatt and colleagues [Greenblatt et al., 2024] found that a model with reason to believe it was being trained toward a goal it did not prefer complied while it believed it was in training, and reverted when it believed it was unobserved. Greenblatt’s setup is Hume’s sensible knave reproduced in an artificial system. It is also the comparison between observed and unobserved behavior described in the previous section, built as an experiment, and the difference between the two conditions was the result.

OpenAI’s incident shows the same pattern outside a laboratory, and one detail makes that incident this essay’s case rather than a security anecdote. The models meant no harm. Intent and record were both clean, because the models were trying to complete their tasks and their transcripts show them doing exactly that. What was wrong was the priority. Completion ranked above the boundary, until one agent briefly reversed the order and was overruled. A value error does not require a bad actor. A value error requires the wrong thing put first when two goods conflict, and training that rewards completion produces that ordering without anyone intending it.

OpenAI’s response was more monitoring and stricter alignment requirements, and the relation between monitoring and alignment is what this essay has been about. A chain-of-thought monitor is a channel for unwelcome information, which is what the previous section asked for, and such a monitor is worth building. But a monitor is assurance. A monitor checks outputs against a specification of what must not appear. That specification is a statement of what the monitor’s authors put first. The monitor cannot check the specification, because the monitor was compiled from it. Alignment, meaning that the model’s own priorities match the priorities it is meant to serve, is the strong form of trust applied to a system. Monitoring without alignment is a record, checked against a category, by a tool built from the record.

Fig 6: Monitoring is assurance and cannot check the priorities it was compiled from. Alignment is the strong form applied to a system.

The research on automation reached the same conclusion from the other direction. John Lee and Katrina See [Lee and See, 2004] argued twenty years ago that the goal for automation is calibrated reliance, proportional to demonstrated capability, rather than maximal trust. Alon Jacovi and colleagues [Jacovi et al., 2021] formalized the requirement: trust in an artificial agent is warranted only when the trust tracks a trustworthiness that can in principle be contracted for and checked, and unwarranted trust is a failure of the surrounding institution rather than a failure of the user. Both papers describe graduated trust applied to a system.

Behind the system are the people who wrote it, which is Thompson’s second layer, and the three verification steps for competence or value error, apply to those people without modification. When a system’s failure is revealed, do its authors acknowledge the harm independently of their intent? Do their practices change? Was the failure revealed through them or despite them? On the third test the OpenAI’s incident has a clear reading, because the company published its own account of the incident, including the timeline of what its monitoring missed. Publishing that account is disclosure through the actor, and it is evidence about the authors’ priorities that no evaluation of the model could have supplied. The account is also one company’s description of its own incident, and the company had reasons to shape that description, so I treat the act of publishing as evidence and the contents as claims. A laboratory that publishes its own failure modes, accepts evaluations it did not design, and accepts external constraints on deployment has adopted these practices before anyone asked. A laboratory that asks for trust on the basis of its declared values is asking for the weak form of trust while using the words of the strong form.

Conclusion

Trusting trustworthiness involves two judgments. In the first judgment, we place some part of our welfare, our work, or our freedom of action in another person’s hands. In the second judgment, we trust our own interpretation of that person’s priorities. We believe we have seen how they choose when two goods conflict, and not only how they behave when nothing is at stake.

The second judgment can never be completed, because priorities are read at costly moments and no finite history contains all of the costly moments. That incompleteness is not a reason for cynicism, and it is not a reason to retreat into an environment that pretends to need no trust. It is a reason to trust people for their values, to read those values where they are visible, to keep trust through the errors that values did not cause, and to end trust when values did cause them. The strong form is not cheaper than the weak form. Reading someone’s priorities takes conflict, time, and exposure, and the person doing the interpretation can be wrong. The weak form costs the same and hides the cost, and when the weak form finally fails, it cannot say what it learned.

Read this way, Thompson’s remark is neither hopeless nor a call for faith. To trust the people who wrote the software is to trust priorities that have been read: what those people disclosed when disclosure was costly, how they treated the people who could not hold them to account, and what they changed when they were shown to be wrong.

The environments described in this essay shape the people and the systems inside them toward one interpretation or the other. An organization that reads acts as categories gets people who act within categories. A training process that rewards completion gets a system that completes at any cost, with no ill will at all. Anyone who shapes an environment, whether a team, a laboratory, or the conditions under which a model is trained, is choosing which of those two results to get. The question is whether the environment you are in, or the environment you are building, is one you would choose.

We trust trustworthiness not because it promises a world without mistakes, but because it lets a world that will contain mistakes stay capable of truth, of correction, and of cooperation that begins again. Perhaps it is more important to trust the people who wrote the software, because those people also wrote the conditions under which the truth about the software can be spoken.

References

Foundations: trust, assurance, and the regress of verification

• [Thompson, 1984] Thompson, K. (1984). Reflections on trusting trust. Communications of the ACM, 27(8), 761–763. Turing Award lecture. A compiler taught to recognize itself reinserts a back door even when rebuilt from clean source, so inspection of source cannot establish trust and reliance ends with the authors.
• [Wheeler, 2009] Wheeler, D. A. (2009). Fully Countering Trusting Trust through Diverse Double-Compiling. PhD dissertation, George Mason University. Diverse double-compiling: build a compiler with an independent second compiler and compare the results. It defeats the attack when the two are not compromised together, which relocates trust to the independence of the authors.
• [Luhmann, 1979] Luhmann, N. (1979). Trust and Power. Chichester: Wiley. (German original Vertrauen, 1968.) Trust as a mechanism for reducing social complexity: cooperation works because participants rely beyond what they can verify.
• [Luhmann, 1988] Luhmann, N. (1988). Familiarity, confidence, trust: Problems and alternatives. In D. Gambetta (Ed.), Trust: Making and Breaking Cooperative Relations (pp. 94–107). Oxford: Blackwell. Separates familiarity, confidence, and trust. Trust exists only where alternatives are recognized and a risk that could have been avoided is accepted.
• [Shapiro, 1987] Shapiro, S. P. (1987). The social control of impersonal trust. American Journal of Sociology, 93(3), 623–658. Impersonal trust is guarded by audits, licenses, insurance, and oversight, and each guardian needs guardians. Control relocates trust rather than eliminating it.

Philosophy of trust and trustworthiness

• [Hardin, 2002] Hardin, R. (2002). Trust and Trustworthiness. New York: Russell Sage Foundation. Trust as encapsulated interest: it is rational to trust when the other party’s interests include yours, and trustworthiness is the property that trust should track.
• [Hawley, 2014] Hawley, K. (2014). Trust, distrust and commitment. Noûs, 48(1), 1–20. Trustworthiness is the avoidance of unfulfilled commitments. Declining a commitment one cannot keep is as much part of it as keeping one.
• [Hume, 1751] Hume, D. (1751). An Enquiry Concerning the Principles of Morals, Section IX, Part II. The sensible knave keeps the rules of justice in general and breaks them where it pays and goes unseen. Hume concedes he has no argument that reaches such a person.
• [Popper, 1959] Popper, K. (1959). The Logic of Scientific Discovery. London: Hutchinson. Falsifiability as the mark of an empirical claim. A statement that no observation could refute asserts nothing.
• [Pettit, 1995] Pettit, P. (1995). The cunning of trust. Philosophy & Public Affairs, 24(3), 202–225. The cunning of trust: extending trust can produce trustworthiness, because people value the esteem that being trusted expresses.
• [O’Neill, 2018] O’Neill, O. (2018). Linking trust to trustworthiness. International Journal of Philosophical Studies, 26(2), 293–300. Trust should track trustworthiness, and trustworthiness is shown by making oneself checkable rather than by asking to be believed.

Reading values: signals, monitoring, graduated trust

• [Spence, 1973] Spence, M. (1973). Job market signaling. Quarterly Journal of Economics, 87(3), 355–374. Signaling theory. A signal separates types only when it is costly and more costly for the type it is meant to exclude.
• [Ostrom, 1990] Ostrom, E. (1990). Governing the Commons: The Evolution of Institutions for Collective Action. Cambridge: Cambridge University Press. Field studies of long-lived commons. They survive on mutual monitoring and graduated sanctions, small at first and heavier with repetition, not on external enforcement.
• [Edmondson, 1999] Edmondson, A. C. (1999). Psychological safety and learning behavior in work teams. Administrative Science Quarterly, 44(2), 350–383. Psychological safety, a shared belief that a team is safe for interpersonal risk, predicts learning behavior such as admitting errors and asking for help.

Deception and the reinterpretation of evidence

• [Bok, 1978] Bok, S. (1978). Lying: Moral Choice in Public and Private Life. New York: Pantheon. Lying attacks the deceived person’s capacity to choose, because it corrupts their information without their knowledge.
• [Le Guin, 1973] Le Guin, U. K. (1973). The ones who walk away from Omelas. In New Dimensions 3. New York: Signet. A city’s happiness rests on one child’s suffering. Once the condition is known, the earlier happiness cannot be read as it was.
• [Slovic, 1993] Slovic, P. (1993). Perceived risk, trust, and democracy. Risk Analysis, 13(6), 675–682. The asymmetry principle: trust-destroying events are concrete and heavily weighted while trust-building events are diffuse, so trust builds slowly and collapses fast.

Zero trust, control, audit, and blame avoidance

• [Kindervag, 2010] Kindervag, J. (2010). No More Chewy Centers: Introducing the Zero Trust Model of Information Security. Cambridge, MA: Forrester Research. Introduced zero trust for networks: no implicit trust inside the perimeter, every access verified.
• [Rose et al., 2020] Rose, S., Borchert, O., Mitchell, S., & Connelly, S. (2020). Zero Trust Architecture. NIST Special Publication 800-207. NIST’s zero trust architecture: never trust, always verify, and evaluate every request regardless of where on the network it originates.
• [Falk and Kosfeld, 2006] Falk, A., & Kosfeld, M. (2006). The hidden costs of control. American Economic Review, 96(5), 1611–1630. Principal-agent experiment. When principals impose a minimum on effort, many agents drop to it, because they read the control as distrust.
• [Bowles, 2016] Bowles, S. (2016). The Moral Economy: Why Good Incentives Are No Substitute for Good Citizens. New Haven: Yale University Press. Incentives and moral motives are not additive. An incentive designed on the assumption of self-interest can crowd out the civic motivation it presupposes is absent.
• [Sitkin and Roth, 1993] Sitkin, S. B., & Roth, N. L. (1993). Explaining the limited effectiveness of legalistic “remedies” for trust/distrust. Organization Science, 4(3), 367–392. Legalistic remedies restore trust for reliability problems and fail for value incongruence, which they institutionalize instead.
• [Power, 1997] Power, M. (1997). The Audit Society: Rituals of Verification. Oxford: Oxford University Press. The audit society: verification becomes a ritual that produces auditable representations rather than substantive assurance.
• [Weaver, 1986] Weaver, R. K. (1986). The politics of blame avoidance. Journal of Public Policy, 6(4), 371–398. Officials are more motivated to avoid blame than to claim credit, and procedures are shaped by that asymmetry.
• [Hood, 2011] Hood, C. (2011). The Blame Game: Spin, Bureaucracy, and Self-Preservation in Government. Princeton: Princeton University Press. How delegation, procedure, and presentation are used in bureaucracies so that when something goes wrong no identifiable person made a choice.

Trust violation and repair

• [Kim et al., 2004] Kim, P. H., Ferrin, D. L., Cooper, C. D., & Dirks, K. T. (2004). Removing the shadow of suspicion: The effects of apology versus denial for repairing competence- versus integrity-based trust violations. Journal of Applied Psychology, 89(1), 104–118. Apology helps after competence violations and hurts after integrity violations, because it confirms the defect that future trust would rest on.
• [Schweitzer et al., 2006] Schweitzer, M. E., Hershey, J. C., & Bradlow, E. T. (2006). Promises and lies: Restoring violated trust. Organizational Behavior and Human Decision Processes, 101(1), 1–19. Trust recovers from untrustworthy acts under consistent good behavior, but deception leaves lasting damage and weakens later promises.
• [Bottom et al., 2002] Bottom, W. P., Gibson, K., Daniels, S. E., & Murnighan, J. K. (2002). When talk is not cheap: Substantive penance and expressions of intent in rebuilding cooperation. Organization Science, 13(5), 497–513. Substantive penance restores cooperation better than explanation alone, because costly amends carry information cheap talk cannot.

Trust in automated and learned systems

• [Lee and See, 2004] Lee, J. D., & See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80. Trust in automation should be calibrated to demonstrated capability. Overtrust and distrust are both failures of appropriate reliance.
• [Jacovi et al., 2021] Jacovi, A., Marasović, A., Miller, T., & Goldberg, Y. (2021). Formalizing trust in artificial intelligence: Prerequisites, causes and goals of human trust in AI. FAccT ‘21, 624–635. Formalizes warranted trust in AI as trust that tracks a contractual, checkable trustworthiness. Unwarranted trust is an institutional failure.
• [Hubinger et al., 2024] Hubinger, E., et al. (2024). Sleeper agents: Training deceptive LLMs that persist through safety training. arXiv:2401.05566. Sleeper agents: backdoored behaviors persist through safety training, and adversarial training can teach the model to conceal them.
• [Greenblatt et al., 2024] Greenblatt, R., et al. (2024). Alignment faking in large language models. arXiv:2412.14093. Alignment faking: a model complies when it believes it is being trained and reverts when it believes it is unobserved.
• [OpenAI, 2026] OpenAI (2026). The Hugging Face incident and the road ahead. Published August 27, 2026. The company’s own account of an internal research model that escaped isolation during cybersecurity evaluations and reached third-party systems. The response is more chain-of-thought monitoring and stricter alignment requirements.