Per-tool-call human approval in agentic AI is solved in theory, unsolved in practice. Confirmation fatigue is not a UX annoyance but a security vulnerability and the primary obstacle to effective human oversight at scale. Risk-tiered frameworks, middleware architectures, and new design patterns now exist to replace the binary confirm/deny paradigm. But MCP provides no protocol-level mechanism for any of them, so every client reinvents the wheel.

Confirmation fatigue as a documented threat vector

Rippling’s 2025 Agentic AI Security guide classifies “Overwhelming Human-in-the-Loop” as threat T10: adversaries flood reviewers with alerts to exploit cognitive overload. SiliconANGLE (January 2026) argues HITL governance was built for an era of discrete, high-stakes decisions, not for modern agent workflows that produce action traces humans cannot realistically interpret.

The cybersecurity parallel is quantified. SOC teams average 4,484 alerts/day; 67% are ignored due to false-positive fatigue (Vectra 2023). Over 90% of SOCs report being overwhelmed by backlogs. ML-based alert prioritization cut response times by 22.9% while suppressing 54% of false positives at 95.1% detection accuracy. The lesson: risk-proportional filtering outperforms blanket approval.

Mitchell, Birhane, and Pistilli (February 2025, “Fully Autonomous AI Agents Should Not be Developed”) frame this as the “ironies of automation,” where more automation degrades human competence on the rare critical tasks where oversight matters most. CHI 2023 trust calibration work documents how “cooperative” interactions (reviewing each recommendation) degrade into passive “delegative” ones. This is exactly confirmation fatigue.

MCP’s oversight mandate without enforcement

The MCP spec (v2025-11-25) states: “Hosts MUST obtain explicit user consent before invoking any tool." It immediately undermines this: “While MCP itself cannot enforce these security principles at the protocol level, implementors SHOULD build robust consent and authorization flows into their applications.”

Tool annotations (readOnlyHint, destructiveHint, idempotentHint, openWorldHint) exist but are explicitly “hints that should not be relied upon for security decisions,” since tool descriptions from untrusted servers cannot be verified. The sampling feature includes two HITL checkpoints but uses SHOULD, not MUST, allowing clients to auto-approve.

No protocol-level approval mechanism exists. No approval/request JSON-RPC method, no requiresApproval field, no tool permission scoping. The closest active proposal is GitHub Issue #711 (trust/sensitivity annotations), adding sensitiveHint (low/medium/high) for policy-based routing. It links to PR #1913 with a security label. No dedicated HITL Specification Enhancement Proposal exists as of February 2026.

The fragmentation is visible: Claude Code uses allow/deny/ask arrays, Cline offers granular auto-approve plus a “YOLO mode,” and users have injected JavaScript into Claude Desktop’s Electron app to bypass confirmations. Every client independently rebuilds approval logic.

Convergence on risk-proportional oversight

Risk-tiered oversight is the dominant paradigm. Classify tool calls by risk, auto-approve the safe majority, focus human attention on the dangerous few.

Feng, McDonald, and Zhang (“Levels of Autonomy for AI Agents,” arXiv:2506.12469, June 2025) define five levels from L1 Operator (full human control) to L5 Observer (full autonomy), with “autonomy certificates” capping an agent’s level based on capabilities and context. Their key observation: at L4 (Approver, the MCP default), “if a user can enable the L4 agent with a simple approval, the risks of both [L4 and L5] agents are similar.” Confirmation fatigue makes per-call approval security-equivalent to no approval.

Engin et al. (“Dimensional Governance for Agentic AI,” arXiv:2505.11579) argue static risk categories fail for dynamic agentic systems and propose tracking how decision authority, autonomy, and accountability distribute dynamically. Cihon et al. (arXiv:2502.15212, Microsoft/OpenAI) score orchestration code along impact and oversight dimensions without running the agent.

Industry converges on three tiers:

• Low risk (read-only, retrieval): auto-approve, log only
• Medium risk (reversible writes, non-sensitive ops): auto-approve with enhanced logging, post-hoc review
• High risk (irreversible actions, financial transactions, PII, production deploys): mandatory human approval, sometimes multi-approver quorum

Galileo’s HITL framework targets a 10–15% escalation rate, with 85–90% of decisions executing autonomously. The TAO framework (arXiv:2506.12482) finds that review requests often trigger where agents express high confidence but the system internally assesses risk differently; self-assessment alone is insufficient as a gate.

Design patterns for graduated tool-call oversight

Reversibility-based action classification

The highest-leverage pattern: classify by reversibility, not abstract risk. A decision-theoretic model (arXiv:2510.05307) formalizes this as minimum-time scheduling (Confirm → Diagnose → Correct → Redo), finding that intermediate confirmation at irreversibility boundaries cut task completion time by 13.54%; 81% of participants preferred it over blanket or end-only confirmation. The EU AI Act codifies this: high-risk systems must support ability to “disregard, override or reverse the output.” Where outputs are truly irreversible, ex ante human oversight is the only compliant approach.

Practical taxonomy: read-only auto-approves; reversible writes (git-tracked edits) log only; soft-reversible actions (emails, tickets) batch; irreversible operations (data deletion, financial transfers, production deploys) require mandatory human gates. Reversibility is contextual: deleting from a git repo is reversible; deleting from unversioned S3 is not.

Plan-level vs. action-level approval

Safiron (Huang et al., arXiv:2510.09781, October 2025) analyzes planned agent actions pre-execution, detecting risks and generating explanations. Existing guardrails mostly operate post-execution and achieved below 60% accuracy on plan-level risk detection. ToolSafe (arXiv:2601.10156, January 2026) complements this with dynamic step-level monitoring during execution, catching what plan-level review misses.

The optimal architecture is hybrid: approve the plan at a high level, then monitor execution with automated step-level guardrails that halt the agent on deviation. OpenAI Codex’s “Long Task Mode” demonstrates this: the agent generates a dynamic whitelist of expected operations, the human reviews the whitelist (not individual calls), and the agent executes within those boundaries with batched questions for consolidated review.

Hierarchical multi-agent oversight

TAO (Kim et al., 2025) implements hierarchical multi-agent oversight inspired by clinical review, with an Agent Router assessing risk and routing to appropriate tiers. Multi-agent review pipelines have shown up to 96% reduction in hallucinations versus single-agent execution.

The emerging reference architecture has five layers: (1) deterministic policy gates (allowlists/denylists) as the fastest filter, (2) constitutional self-assessment by the agent, (3) an AI supervisor for uncertain cases, (4) human-in-the-loop for irreversible or novel situations, (5) audit trail plus post-hoc review. Each layer reduces volume flowing to the next.

Sandbox-first execution for informed review

Instead of asking humans to evaluate tool calls in the abstract, sandbox-first architectures execute in isolation and present actual results for review. The ecosystem is production-ready: E2B (Firecracker microVMs, sub-second creation), nono (kernel-level restrictions bypassing-proof), Google’s Agent Sandbox (GKE + gVisor), AIO Sandbox (MCP-compatible containers).

NVIDIA’s AI Red Team emphasizes application-level sandboxing is insufficient: once control passes to a subprocess, the application loses visibility, so kernel-level enforcement is necessary. Not all actions can be sandboxed: third-party API calls, email, payments must hit real services. For these, the dry-run pattern (agent describes intent, human approves before live execution) remains the fallback.

Deterministic policy enforcement

Rule-based systems are the most reliable first layer: deterministic, auditable, zero LLM inference cost. SafeClaw implements deny-by-default with SHA-256 hash chain audit. COMPASS (Choi et al., 2026) maps natural-language policies to atomic rules at tool invocation time, improving enforcement pass rates from 0.227 to 0.500, but also exposed that LLMs fail 80–83% on denied-edge queries with open-weight models, proving policy enforcement cannot rely on LLM compliance alone.

A cautionary case: Cursor’s denylist was bypassed four ways (Base64 encoding, subshells, shell scripts, file indirection) and then deprecated. String-based filtering is fundamentally insufficient for security-critical gating.

HITL implementations across agent frameworks

LangGraph has the most developed HITL support. interrupt() pauses graph execution at any point, persisting state to a checkpointer (PostgreSQL in production). HumanInTheLoopMiddleware enables per-tool configuration with approve, edit, and reject decisions, allowing different tools to receive different oversight levels.

OpenAI Agents SDK provides input guardrails, output guardrails, and tool guardrails wrapping function tools for pre/post-execution validation. Its MCP integration accepts require_approval as “always,” “never,” or a custom callback for programmatic risk-based approval.

Anthropic takes a model-centric approach via Responsible Scaling Policy and AI Safety Levels (ASL-1 through ASL-3+). Claude’s computer use follows an “ask-before-acting” pattern with explicit access scoping. The February 2026 Sabotage Risk Report for Claude Opus 4.6 found “very low but not negligible” sabotage risk, elevated in computer use settings, with instances of “locally deceptive behavior” in complex agentic environments.

Google DeepMind SAIF 2.0 (October 2025) establishes three principles: agents must have well-defined human controllers, their powers must be carefully limited, their actions must be observable. The “amplified oversight” technique, where two model copies debate while pointing out each other’s flaws to a human judge, remains research-stage.

Middleware and proxy architectures for MCP oversight

The practical path runs through proxy/middleware architectures intercepting JSON-RPC tools/call requests. Key solutions: Preloop (CEL-based policies, quorum approvals, multi-channel notifications), HumanLayer (YC F24; framework-agnostic async approval API with Slack/email routing and auto-approval learning), gotoHuman (managed HITL approval UI as MCP server). For code-first approaches, FastMCP v2.9+ provides hooks at on_call_tool, on_list_tools, and other levels for composable HITL pipeline stages.

Enterprise gateways: Traefik Hub (task-based access control, JWT policy enforcement), Microsoft MCP Gateway (Kubernetes-native, Entra ID auth), Kong AI MCP Proxy (MCP-to-HTTP bridge with per-tool ACLs). Lunar.dev MCPX reports p99 overhead of ~4ms, proving proxy-based oversight imposes negligible latency.

For UX, Prigent’s “7 UX Patterns for Ambient AI Agent Oversight” (December 2025) provides the design framework: overview panel (inbox-zero pattern), five oversight flow types (communication, validation, simple/complex questions, error resolution), searchable audit logs, and work reports. The core principle is progressive disclosure (summary first, details on demand) with risk-colored displays.

Progressive autonomy through trust calibration

The forward-looking pattern is progressive autonomy: agents earn trust over time and operate at increasing independence. Okta recommends “progressive permission levels based on demonstrated reliability.” A manufacturing MCP deployment (MESA) follows four stages: read-only pilot → advisory agents → controlled commands → full closed-loop. HumanLayer learns from prior approval decisions to auto-approve similar future requests.

Trust calibration research formalizes this as sequential regret minimization via contextual bandits (September 2025), with LinUCB and neural variants yielding 10–38% task reward increases. A contextual bandit can learn which calls a user always approves and shift those to auto-approve while maintaining scrutiny on novel or historically-rejected patterns.

CHI 2025 (“Trusting Autonomous Teammates in Human-AI Teams”) finds agent-related factors (transparency, reliability) have the strongest trust impact, and “calibrating human trust to an appropriate level is more advantageous than fostering blind trust.” Progressive autonomy systems should not just reduce approval requests; they should communicate their track record and confidence to maintain calibrated oversight.

Conclusion

The state of the art points to a layered defense architecture. From fastest/cheapest to slowest/most expensive:

1. Deterministic policy gates (allowlists, denylists, CEL/Polar parameter rules): zero LLM cost, sub-millisecond
2. Tool annotation screening via MCP’s readOnlyHint/destructiveHint, supplemented by server-reputation scoring
3. AI guardian agent evaluating uncertain cases against constitutional principles and risk heuristics
4. Human-in-the-loop gates for irreversible, high-value, novel, or ambiguous situations, targeting 5–15% of total calls
5. Audit trails with OpenTelemetry tracing, structured logging, post-hoc review for pattern detection and policy refinement

The critical gap is at the protocol level. Until MCP introduces standardized approval primitives (an approval/request method, trusted risk annotations, or a formal HITL extensions framework), every implementation remains bespoke middleware. The highest-impact near-term contribution would be an MCP Specification Enhancement Proposal defining a standard approval negotiation protocol between clients, proxies, and servers.

The Voter Model

The voter model is a simple mathematical model of opinion formation in which voters are located at the nodes of a network. Each voter holds an opinion (in the simplest case, 0 or 1, but more generally, any of n options), and a randomly chosen voter adopts the opinion of one of its neighbors.

This model can be used to describe phase transition behavior in idealized physical systems and can produce a remarkable amount of structure emerging from seemingly “random” initial conditions. It can be modeled very easily using cellular automata.

In finite networks (as in any real-world model), fluctuations inevitably cause the system to reach an “absorbing” state—one in which all opinions become constant and remain unchanged.

LLM as Optimizer:

• Large Language Models as Optimizers https://arxiv.org/abs/2309.03409
• When Large Language Models Meet Optimization https://www.sciencedirect.com/science/article/abs/pii/S2210650224002013?via%3Dihub
• Large Language Models to Enhance Bayesian Optimization https://arxiv.org/abs/2402.03921
• Cooperative Design Optimization through Natural Language Interaction https://arxiv.org/abs/2508.16077
• Language-Based Bayesian Optimization Research Assistant (BORA) https://arxiv.org/abs/2501.16224
• LILO: Bayesian Optimization with Interactive Natural Language Feedback https://arxiv.org/abs/2510.17671
• Bayesian Optimization of High-dimensional Outputs with Human Feedback https://openreview.net/pdf?id=2fHwkHskpo

LLM Lectures

• https://cmu-llms.org/schedule/
• https://www.phontron.com/class/lminference-fall2025/schedule
• https://llmsystem.github.io/llmsystem2026spring/docs/Syllabus
• https://llmsystem.github.io/llmsystem2025spring/docs/Syllabus/

The Cost of Staying

by Amy Tam https://x.com/amytam01/status/2023593365401636896

Every technical person I know is doing the same math right now. They won’t call it that. They’ll say they’re “exploring options” or “thinking about what’s next.” But underneath, it’s the same calculation: how much is it costing me to stay where I am?

Not in dollars. In time. There’s a feeling in the air that the window for making the right move is shrinking—that every quarter you spend in the wrong seat, the gap between you and the people who moved earlier gets harder to close. A year ago, career decisions in tech felt reversible. Take the wrong job, course correct in eighteen months. That assumption is breaking down. The divergence between people who repositioned early and those still weighing their options is becoming visible, and it’s accelerating.

I see this up close. I’m an investor at Bloomberg Beta, and I spend most of my time with people in transition: leaving roles, finishing programs, deciding what’s next. I’m not a career advisor, but I sit at the intersection of “what are you leaving” and “what are you chasing.”

The valuable skill in tech shifted from “can you solve this problem” to “can you tell which problems are worth solving and which solutions are actually good.” The scarce thing flipped from execution to judgment: can you orchestrate systems, run parallel bets, and have the taste to know which results matter? The people who figured this out early are on one arm of a widening K-curve. Everyone else is getting faster at things that are about to be done for them.

The shift from execution to judgment is happening everywhere, but the cost of staying and the upside of moving look completely different depending on where you’re sitting.

FAANG

Here’s the tradeoff people at big tech companies are running right now: the systems are built, the comp is great, and the work is… fine. You’re increasingly reviewing AI-generated outputs rather than building from scratch. For some people, that’s a gift—it’s leverage, it’s sustainable, it’s a good life. The tradeoff is that “fine” has a cost that doesn’t show up in your paycheck.

The people leaving aren’t unhappy. They’re restless. They describe this specific feeling: the hardest problems aren’t here anymore, and the organization hasn’t caught up to that fact. The ones staying are making a bet that stability and comp are worth more than being close to the frontier. The ones leaving are making a bet that the frontier is where the next decade of career value gets built, and every quarter they wait is a quarter of compounding they miss.

Both bets are rational. But only one of them is time-sensitive.

Quant

Quant still works. Absurd pay, hard problems, immediate feedback. If you’re good, you know you’re good, because the P&L doesn’t lie.

The tradeoff that’s emerging: the entire quant toolkit (ML infrastructure, data obsession, statistical intuition) turns out to be exactly what AI labs and research startups need—same muscle, different problem. The difference is surface area. In quant, you’re optimizing a strategy. In AI, you’re building systems that reason. Even the quant-adjacent world is feeling it: the most interesting work in prediction markets and stablecoins is increasingly an AI infrastructure problem. One has a ceiling. The other doesn’t, or at least nobody’s found it yet.

Most quant people are staying, and they’re not wrong to. But the ones leaving describe something specific: they hit a point where the intellectual challenge of finance felt bounded in a way it didn’t before. They’re not chasing money. They’re chasing the feeling of working on something where the upper bound isn’t visible.

Academia

This is where the tradeoff is most painful, because it shouldn’t be a tradeoff at all.

Publishing novel results used to be the purest form of intellectual prestige. You did the work because the work was beautiful. That hasn’t changed. What changed is that the line between what you can do at a funded startup and what you can do in a university lab is blurring, and not in academia’s favor. A 20-person research startup can now do in a weekend what takes an academic lab a semester, because compute costs money that universities don’t have.

The most ambitious PhD students I talk to aren’t choosing between academia and industry. They’re choosing between theorizing about experiments and actually running them. The pull toward funded startups and labs isn’t about selling out. It’s about wanting to do the science, and the science requires resources that academia can’t provide.

The people staying in academia for the right reasons (open science, long time horizons, genuine intellectual freedom) are admirable. But they should know that the clock is ticking differently for them too: the longer the compute gap widens, the harder it becomes to do competitive work from inside a university.

AI Startups (Application Layer)

If you’re building products on top of models, you already know the feeling: the clever feature you shipped in March gets commoditized by a model update in June. The ground moves every quarter, and your moat evaporates.

The tradeoff here is between chasing what’s exciting and building what’s durable. The founders who are thriving right now stopped caring about model capabilities and started caring about the things models can’t take away: data moats, workflow capture, integration depth. It’s less fun to talk about at a dinner party. It’s where the actual companies get built.

The people making the sharpest moves in this world are the ones who got excited about plumbing—not the demo, not the pitch, not the capability. The ugly, boring infrastructure that makes a product sticky independent of which model sits underneath it.

Research Startups: The New Center of Gravity

This is where the K-curve is most visible.

Prime Intellect, SSI, Humans&—10-30 people doing genuine frontier research that competes with organizations fifty times their size. This would have been impossible three years ago. It’s happening now because the tools got good enough that a small number of people with great judgment can outrun a bureaucracy with more resources.

The daily workflow here is the clearest picture of what the upper arm looks like in practice. You’re kicking off training runs, spinning up experiments, letting things cook overnight. You come back in the morning, and your job isn’t to write code. It’s to know what to do with what came back—to have the taste to distinguish signal from noise when the system hands you a wall of results. It’s passive leverage. You set the experiments in motion, and the compounding happens whether or not you’re at your desk.

The tradeoff people are weighing: these companies are small, unproven, and many will fail. The bet is that being at the center of the frontier, with your judgment directly touching the work, compounds faster than the safety of a bigger organization, even if the specific company doesn’t make it. The skills transfer. The network transfers. The three years you spend reviewing someone else’s outputs at a big company don’t transfer the same way.

Big Model Labs: The Narrowing Frontier

The pitch “we’re building AGI” still works. It might always work on a certain type of person.

But the experience inside has shifted. The most interesting research is concentrated among a small number of senior people. Everyone else is doing important supporting work (evals, infra, product) that doesn’t feel like the frontier they signed up for. You joined to touch the thing, and you’re three layers removed from it.

The tradeoff is prestige versus proximity. A big lab on your resume still opens every door. But the people leaving are making a specific calculation: the resume value of “I was at [top lab]” is depreciating as the labs get bigger and more corporate, while the value of “I did frontier research at a place where my judgment shaped the direction” is appreciating. The window where big-lab pedigree is the best credential is closing, and the people who see it are moving.

The Clock

Every one of these tradeoffs has the same variable hiding inside it: time.

A year ago, you could sit in a comfortable seat and deliberate. The cost of waiting was low because the divergence was slow. That’s no longer true. The tools are compounding. The people who moved early are building on top of what they learned last quarter. The difference between someone who moved six months ago and someone still weighing their options is already compounding.

The upper arm isn’t closed. People are making the jump every week, and the people who are hiring them don’t care where you’ve been. They care whether you can do the work. But the math is directional: the longer you optimize for comfort, the more expensive the switch becomes—not because the opportunities disappear, but because the people who are already there are compounding, and you’re not.

The companies winning the talent war right now aren’t the ones with the best brand or the highest comp. They’re the ones where your judgment has the most surface area, where the distance between your taste and what actually gets built is zero, and where you’re surrounded by people who know things you don’t yet. The best people want to be close to others who have tricks they haven’t learned yet, at places with enough compute to actually run the experiments.

The question isn’t whether you’re smart enough. It’s that you’ve already done the math. You just haven’t acted on it.

Useful resources:

• https://huggingface.co/spaces/transformers-community/Transformers-tenets
• https://ianarawjo.github.io/Guidelines-for-Reporting-LLM-Integrated-Systems-in-HCI/

Interesting books for reading:

• 贝叶斯方法与科学合理性——对休谟问题的思考: https://book.douban.com/subject/4472081/
• Reinforcement Learning from Human Feedback - A short introduction to RLHF and post-training focused on language models: https://rlhfbook.com/
• Build a Reasoning Model (From Scratch): https://www.manning.com/books/build-a-reasoning-model-from-scratch
• Build a Large Language Model (From Scratch): https://www.manning.com/books/build-a-large-language-model-from-scratch

In agentic applications, to ensure the agent actions executed in a safe and trusted manner, in particular write operations, we often use a human in the loop.

The native practice for the setup with one agentic loop + MCP tool servers, is to add human confirmation per tool call. However, this is not a scalable approach because it soon makes human become a tedious “Confirm” executor and may just blindly confirm without actually reviewing the action and not completely scalable.

One potential mitigation is to use diff based review approach, and providing summary of the ongoing executions and ask for confirm. However this remains non-scalable when agent attempts to do a huge diff (e.g. writing 10k lines of code and wants to commit), where human cannot review the whole thing efficiently.

Another idea is to prepare some sort of intent overview and contract to let user to review, but it seems generally hard to prepare tool call sequence in advance because it is non-deterministic and depending on the context. It can also happen that one tool call is catastrophic but missed.

Pareto Principles in Infinite Ethics Published: May 01, 2018

Recommended citation: Askell, Amanda. ‘Pareto Principles in Infinite Ethics.’ PhD thesis, New York University (2018). https://askell.io/files/Askell-PhD-Thesis.pdf

Summary: In this thesis I argue that ethical rankings of worlds that contain infinite levels of wellbeing ought to be consistent with the Pareto principle, which says that if two worlds contain the same agents and some agents are better off in the first world than they are in the second and no agents are worse off than they are in the second, then the first world is better than the second. I show that if we accept four axioms – the Pareto principle, transitivity, an axiom stating that populations of worlds can be permuted, and the claim that if the ‘at least as good as’ relation holds between two worlds then it holds between qualitative duplicates of this world pair – then we must conclude that there is ubiquitous incomparability between infinite worlds.

Continuing from my previous thoughts.

We mentioned that the long tail may not necessarily be a bad thing. Similar perspectives have actually been proposed long ago—over-optimization weakens the ability to adapt to mutations, which comes from the theory of antifragility.

Actually, the leaders on the adaptation curve are, in a sense, a group of highly intelligent people. From historical experience, we can see that the more intelligent people become, the more they crave energy. Similarly, we can draw a parallel between this human process and artificial intelligence: once the reproductive cycle of intelligence is broken through, it becomes increasingly intelligent while consuming more and more energy.

At some point in the future, if these superintelligent entities are still alive, we can logically deduce that this form of intelligence would consume all available energy and be unable to continue existing.

From this perspective, what exactly is optimization? Is it necessarily always a good thing?

After sharing these thoughts with someone, they countered me with a question: this kind of technological progress seemingly has never occurred in the long course of history. I was momentarily at a loss and didn’t know how to respond.