The following content is generated by LLMs and may contain inaccuracies.

I’ll search for relevant research on Byzantine Fault Tolerance in AI agents and multi-agent consensus to deepen this analysis.Let me search for the specific scalar consensus research mentioned in the original note.Perfect! Now let me fetch the key papers to verify URLs and get precise details.Now let me get more information about the multi-agent coordination research and Byzantine consensus papers:Based on the extensive research I’ve gathered, I can now provide a comprehensive deep dive on this idea about Byzantine Fault Tolerance in AI agent systems.

---

Context

The rise of multi-agent AI systems marks a collision between two engineering traditions that have evolved separately for decades. Large language models are now being integrated into autonomous agents that collaborate on complex tasks, yet most practitioners approach these systems as if they were building a team meeting rather than a distributed system. Meanwhile, Byzantine Fault Tolerance—a consensus-based algorithmic technique that ensures systems can function correctly even when components fail or act maliciously—has been a cornerstone of distributed systems engineering since the 1980s.

The core insight is that hallucinated intermediate fields in LLM outputs are structurally identical to the Byzantine problem: an agent returns a confident-looking response with no error indication, and everything downstream treats that invented value as real. When agents share the same base model or similar training data, their failure modes become correlated—if a single fault can simultaneously flip more than f modules, Byzantine fault tolerance guarantees are void, which is why design diversity is essential. This violates a fundamental assumption of classical BFT: that faults are independent.

The practical implications are severe. Multi-agent LLM systems fail at 41-86.7% rates in production, and many failures arise from organizational design and agent coordination challenges rather than individual agent limitations. The agent community is rediscovering hard-won lessons from distributed systems, often without the vocabulary or theoretical foundations that would accelerate progress.

---

Key Insights

LLM consensus failures are primarily liveness problems, not safety problems. Recent empirical work confirms the original hypothesis: researchers tested LLM agents on scalar consensus tasks and found that valid agreement is not reliable even in benign settings and degrades as group size grows. Specifically, valid consensus drops from 46.6% at N=4 to 33.3% at N=16. Crucially, Byzantine agents primarily harm liveness by preventing agreement rather than steering outcomes to corrupted values—agents fail to converge at all rather than converging on wrong answers. This is detailed in Berdoz et al., “Can AI Agents Agree?", which systematically evaluates LLM-based Byzantine consensus games.

Correlated failures break classical fault tolerance assumptions. When every server runs the same exact software with the same limits and failure modes, a software bug or load-related failure that causes one server to fail can impact the rest of the fleet simultaneously. In AI systems, when multiple organizations deploy autonomous agents based on similar underlying models, the risk of correlated failure arises. Amazon’s operational experience, documented in their Builders' Library on minimizing correlated failures, shows that correlated failures eat away at availability gains from redundancy, including issues with power, network, cooling, and common infrastructure dependencies like DNS.

Weighted consensus mechanisms show promise for LLM-agent reliability. The research community has begun developing BFT-inspired protocols tailored to LLM characteristics. LLM-based agents demonstrate stronger skepticism when processing erroneous message flows, enabling them to outperform traditional agents across different topological structures. Building on this, Zheng et al. propose CP-WBFT (Confidence Probe-based Weighted Byzantine Fault Tolerant consensus), which leverages the inherent reflective and discriminative capabilities of LLMs, assigning higher transmission weights to more credible agents. Under extreme conditions, this approach achieved +85.71% Byzantine Fault Tolerance improvement on complete graphs while maintaining 100% round-level accuracy.

Coordination failures manifest as emergent system-level phenomena. Multi-agent systems introduce failures that are emergent behaviors—encoded nowhere but arising everywhere from agents that learn from each other, mislead each other, or accidentally form coalitions. Empirically, coordination failures account for 36.94% of multi-agent system failures, verification gaps 21.30%, and infrastructure issues ~16%. The MAST taxonomy documents 14 fine-grained failure modes mapped to execution stages where their root causes typically emerge, providing the first systematic framework for understanding multi-agent LLM system breakdowns.

Hallucination propagation creates unique challenges beyond traditional distributed systems. Multi-agent visual hallucination snowballing occurs where hallucinations are seeded in a single agent and amplified by following ones due to over-reliance on textual flow, with vision tokens gradually diminishing in deeper agent turns. Unlike traditional Byzantine nodes that send contradictory messages, a token-level hallucination can propagate through a workflow and surface as a compliance breach, creating what one practitioner calls “expensive, slow workers that are occasionally wrong in ways that look correct”.

Leaderless consensus architectures reduce single points of failure. Traditional multi-agent frameworks often rely on leader-based protocols, but consensus latency can increase significantly due to consecutive Byzantine leaders—if the leader is Byzantine or submits a low-quality answer that fails to obtain a quorum, the round must be rerun, and multiple consecutive Byzantine leaders can dramatically increase latency. The DecentLLMs framework addresses this by employing a leaderless consensus architecture where worker agents generate answers in parallel and evaluator agents score them, enabling all agents within each role to participate equally.

Network topology and scale fundamentally constrain coordination capabilities. Performance smoothly decreases as agent networks grow in size, with all coordination tasks becoming substantially more challenging as network size increases—for 100-agent networks, performance drops to near zero across the board. This is documented in AgentsNet, a benchmark drawn from classical distributed systems problems that explicitly assesses coordination and collaboration capabilities that should be seen as fundamental to effective distributed systems.

Consensus mechanisms themselves can harm performance through premature convergence. Due to LLM hallucinations, confidence scores may be unreliable, and if a small subset of agents is compromised via prompt injection attacks, the system may converge toward a shared but incorrect answer—when decisions are made using mechanisms such as majority voting, this can lead to complete failure. The FREE-MAD framework proposes assigning scores to all candidate responses without requiring consensus in the debate stage, avoiding the conformity pressure that can suppress minority viewpoints.

Organizational design principles matter more than model capability. Improvements in base model capabilities will be insufficient to address the full taxonomy of multi-agent system failures—good MAS design requires organizational understanding, as even organizations of sophisticated individuals can fail catastrophically if the organization structure is flawed. This aligns with research showing that well-defined design principles from high-reliability organizations can prevent such failures, suggesting that the path forward involves importing concepts from organizational theory and industrial management, not just better language models.

---

Open Questions

Can we develop formal verification methods for stochastic consensus protocols? Classical BFT provides deterministic guarantees, but LLM-based agents operate through probabilistic inference. If disagreement is the most informative signal in stochastic multi-agent systems, what does a formally verified “consensus on lack of consensus” look like, and can we prove bounds on the informativeness of disagreement patterns?

What is the theoretical limit of coordination in homogeneous vs. heterogeneous agent ensembles? If correlated failures are inevitable when agents share base models, and design diversity introduces non-trivial integration complexity, is there an optimal point on the homogeneity-diversity spectrum? Can we quantify the coordination tax of diversity and determine whether it’s fundamentally worthwhile for Byzantine robustness in production LLM systems?

The following content is generated by LLMs and may contain inaccuracies.

Harness Engineering 与模型动态适配：深度分析

I need to first search for information related to Harness Engineering and model evaluation benchmarks to enrich this idea. Let me conduct additional searches on model evaluation and best practices for continuously updating benchmarks. Now I will synthesize these search results to write a structured in-depth analysis.

---

Context

Harness Engineering is an emerging discipline that has risen in recent years, referring to systematic constraints, toolchains, and feedback loops built around AI models to ensure their reliability in production environments. The core observation underlying this idea is that the capability of foundation models themselves is far less important than the systems built around them—LangChain’s coding agents jumped from 52.8% to 66.5% on Terminal Bench 2.0 simply by changing the Harness rather than the model itself.

Your idea touches on a critical blind spot in this field: the dynamic adaptation problem between Harness and model capabilities. As you stated, Harness is essentially a cybernetic system designed to create compensatory mechanisms for specific model shortcomings (such as memory, context management, output formatting). However, when models are updated iteratively, a “benchmark drift” phenomenon emerges: static testing systems cannot keep pace with model capability evolution. This has empirical precedent in software engineering: the capabilities of LLM test generation tools can change completely within six months, posing challenges to the reliability of continuous integration pipelines.

This problem is particularly acute in the AI era because model release cycles are rapid and capability improvements are non-linear—while some dimensions show breakthrough improvements, new limitations may emerge in other dimensions. Research shows that agent scaffolding design is equally important as model capability, and appropriate orchestration and memory structures can even enable weaker models to outperform stronger ones.

---

Key Insights

1. The Cybernetic Nature of Harness and Model-System Symbiosis

Harness Engineering is not merely “what to ask” (prompt engineering) or “what context to provide” (context engineering), but rather the entire operational environment—tools, permissions, state, testing, logging, retry logic, checkpoints, and guardrails. This bears strong resemblance to feedback control systems in cybernetics: maintaining stability through continuous monitoring of output deviations and adjusting system parameters.

OpenAI’s Harness Engineering practices demonstrate this point: they treat codebases as knowledge bases for agents, push all architectural decisions and specifications to repositories, and use customized linters and structural tests to enforce strict architectural constraints. However, this system was designed for specific Codex versions—when the underlying model updates, these constraints may become shackles.

2. The “Shortcoming Resolution-Shortcoming Emergence” Cycle: The Harness Paradox in Model Evolution

The empirical evidence supports your observation of the “shortcomings 1-2-3 disappear, shortcomings 4-5-6 emerge” phenomenon. GenEval benchmarks aligned highly with human judgment at release, but as T2I models evolved, the absolute error between it and human judgment reached 17.7%, indicating the benchmark had long saturated. This means evaluation systems designed around old model shortcomings cannot capture the true capability boundaries of new models.

When new models improve reasoning ability, middleware optimized for reasoning may become counterproductive. After each major model update, Harness components require review and revision. This demands a meta-level observation system that not only assesses task completion but also identifies whether Harness itself has become a bottleneck.

3. The Structural Roots of Benchmark Drift: Co-evolution of Evaluator and Evaluated

The challenge with automated evaluation lies in: the judge model must be able to score correctness, and test prompts must be challenging for current T2I models but not for the judge. Satisfying these constraints leads to benchmark drift, where static benchmarks cannot keep pace with new model capabilities. This reveals an inherent contradiction in evaluation systems: measuring dynamic targets with static tools.

In enterprise environments, this problem is even more severe. Enterprise-grade agents face evolving services and requirements, with scarce ground truth samples. Existing benchmarks are static and task-specific; when requirements change, manual revision becomes necessary. Microsoft Research’s proposed solution is to automatically generate benchmarks from limited semi-structured documents using LLMs, allowing the evaluation framework to evolve with requirements and provide rapid feedback.

4. Continuous Evaluation Framework: Transition from “Point-in-Time Snapshot” to “Real-Time Monitoring”

The rapid evolution of LLM capabilities means evaluations quickly become outdated. Organizations need to maintain continuous evaluation processes rather than rely on point-in-time assessments. This requires capabilities at three levels:

• Capability Discovery Layer: Capability elicitation is a systematic probing process to discover the full range of model capabilities, including latent abilities not obvious in standard evaluations. Models may possess capabilities that only manifest under specific prompting strategies, chain-of-thought reasoning, few-shot examples, or tool-augmented settings, posing major challenges to safety evaluation.

• Shortcoming Identification Layer: Real-world enterprise agents typically run continuously over extended periods; short-term standard evaluations cannot capture performance drift, context retention, or cumulative decision effects. Long-horizon evaluation is needed to observe system stability under actual workloads.

• Harness Adaptation Layer: Evaluation-driven Development (EDD) proposes making evaluation an integral part of the agent development cycle, conducting continuous evaluation both during development and post-deployment to detect regressions and adapt to new use cases.

5. The Dilemma of Non-Model Vendors and Potential Solution Paths

The core dilemma you identified is: non-model vendors are “model + Harness” users lacking direct access to observe model internal capabilities. They typically discover Harness failures or new limitations only during use. This information asymmetry necessitates indirect inference mechanisms:

• Comparative Benchmarks: Benchmarks serve as progress markers; comparing new and old LLMs to assess whether new modifications improve performance. When models consistently exceed certain benchmarks, these become outdated, driving researchers to develop more challenging ones. Benchmarks also identify model weaknesses, guiding fine-tuning processes.

• A/B Testing and Real User Feedback: Aligning evaluation standards with actual use cases; conducting A/B tests with real users to verify that benchmark improvements translate to better experiences; establishing clear trigger conditions for retraining or replacement when performance drops below acceptable thresholds.

• LLM-as-Judge Pipeline: Tools like DeepEval automate multi-metric LLM evaluation, including LLMs as judges; organizations can build internal pipelines using GPT-4 or Claude as reviewers. However, note that public benchmarks may cause data contamination and overfitting; adversarial inputs expose robustness gaps. Strategy should include diverse, domain-specific test suites and integrated red team testing.

6. “Competitive Moat in the AI Era”: The Compound Value of Dynamic Benchmarks

Your observation—that long-term accumulation of model shortcoming discovery benchmarks constitutes the true competitive moat—provides profound insight. The value of this moat lies in:

• First-Mover Advantage: Enterprise-grade LLM agents themselves evolve—operators continuously integrate updated model versions and reasoning capabilities, making evaluation a continuous necessity rather than one-time exercise. Organizations that establish continuous evaluation systems first adapt to new models faster.

• Organizational Learning Curve: Success requires investment in two areas: prompt engineering significantly impacts performance; developers need training to effectively use tools, particularly understanding prompt engineering principles and best practices. Accumulated evaluation datasets and methodologies themselves constitute hard-to-replicate knowledge assets.

• Ecological Niche Lock-in: Building AI products requires custom test datasets reflecting use cases, covering critical scenarios and edge cases. Task-specific evaluations are also necessary, such as LLM judges against customized standards. Domain-specific benchmarks constitute barriers to entry.

---

Open Questions

1. Possibilities of Reverse Adaptation: If we could establish formalized mapping relationships between “Harness capability profiles” and “model capability profiles,” could we develop automated tools that suggest Harness additions, deletions, and modifications when new models are released? What meta-model architecture would this require?

2. The “Half-Life” of Benchmarks: Do different types of evaluation benchmarks (such as reasoning, generation, interaction) have patterns in their failure speed when facing model iterations? Could we establish a “benchmark aging prediction model” that proactively identifies which tests are about to fail, enabling preemptive evaluation system updates?

The following content is generated by LLMs and may contain inaccuracies.

The Paradox of Civilizational Decline Through AI Overuse

Context

This line of thinking touches the intersection of cognitive science, philosophy of technology, and existential risk research. It builds on the Free Energy Principle (FEP)—proposed by Karl Friston, which posits that biological systems maintain their existence by minimizing prediction error—extending into an existential critique of AI tool dependence. The core tension is this: as AI assumes humanity’s predictive and reasoning labor, will humans functionally “die” through loss of predictive capacity? This concerns not merely individual cognitive degradation, but a paradox of mutual destruction: humans outsource prediction to AI, ultimately leading to the disappearance of humans as sources of uncertainty, while AI collapses from loss of training data and objectives.

This perspective resonates with current discussions of AI alignment, cognitive offloading, and deskilling, while proposing a more radical hypothesis: this is not simple tool dependence, but systemic collapse involving the fundamental definition of life itself (prediction as existence).

---

Core Insights

The Free Energy Principle and Predictive Essentialism The original note accurately captures the core claim of Karl Friston’s Free Energy Principle: biological systems maintain existence by minimizing free energy (an upper bound on prediction error). The principle, grounded in Bayesian inference, posits that the brain is an “inference engine” that generates predictions through internal models and updates them using sensory input to improve predictive accuracy. The phrase “humans do not live to predict the world, but live because they predict the world” embodies the Free Energy Principle’s ontological claim: anything existing appears to minimize surprisal, exhibiting behavior consistent with its kind—behavior without surprise.

Cognitive Offloading and Deskilling: Empirical Evidence The original note’s observations about typing ability, voice input, and evolving thought patterns are supported by cognitive offloading research. Recent studies show significant negative correlation between frequent AI tool use and critical thinking ability, with cognitive offloading as a mediating factor. A 2025 study of 580 university students found that higher AI dependence correlates with lower critical thinking levels, with cognitive fatigue partially mediating this relationship. Regarding deskilling, technology only partially automates routine tasks in certain occupations, simplifying them for lower-skilled workers—a phenomenon termed “technology-enabled deskilling.” Deskilling occurs not only among displaced workers but among AI-augmented workers; the boundary between augmentation and replacement is blurred.

Theoretical Precedent for the Mutual Destruction Paradox The “mutual destruction paradox” proposed in the original note—that AI collapses as humans disappear and cease providing uncertainty inputs—has a striking counterpart in AI research: model collapse. When generative AI models are recursively trained on synthetic data, they gradually degrade. A 2024 Nature study showed that indiscriminate training on AI-generated content causes models to lose their capacity for generating diverse, high-quality outputs. In the large language model context, training on text generated by predecessor models causes continuous decline in vocabulary, syntax, and semantic diversity in model outputs. This perfectly echoes the insight in the original note that “humans serve as uncertainty input”: AI requires the diversity and unpredictability produced by humans as training signals, and when this source dries up, the system itself degrades.

---

Core Insights (Expanded)

The Free Energy Principle as Foundation for Ontology Karl Friston’s Free Energy Principle is a mathematical principle positing that the brain reduces surprisal or uncertainty through predictions based on internal models, updating these models with sensory input to improve predictive accuracy. The principle claims that anything existing appears to minimize surprisal—displaying behavior consistent with its type, unsurprising behavior. The original note’s statement “humans do not live to predict the world, but live because they predict the world” precisely captures this ontological turn: prediction is not a tool but a defining condition of existence itself.

Cognitive Offloading Leading to Decline in Critical Thinking A 2025 mixed-methods study of 666 participants found significant negative correlation between frequent AI tool use and critical thinking ability, with cognitive offloading as a mediating factor. Research on 580 Chinese university students showed that higher AI dependence correlates with lower critical thinking levels, with cognitive fatigue partially mediating this relationship. This validates the original note’s concern about “simplification of thought and action”: when AI assumes reasoning chains, humans lose not merely the capacity to execute them, but the opportunity to develop these capacities.

Technology-Enabled Deskilling Technology only partially automates routine tasks in mid-wage occupations, simplifying them to levels manageable by lower-skilled workers—“technology-enabled deskilling.” Deskilling traditionally referred to skills lost by workers displaced through automation, but it equally applies to workers augmented by AI, where the boundary between augmentation and replacement is blurred. The original note’s example of typing skill decline—the shift from muscle memory to voice input—perfectly illustrates this: each instance of cognitive offloading redefines the minimum standard for “competence,” rendering deeper capabilities optional or obsolete.

Model Collapse: AI’s Self-Consuming Paradox Shumailov et al.’s 2023 paper “The Curse of Recursion: Training on Generated Data Makes Models Forget” demonstrates that when generative AI models (including variational autoencoders and diffusion models) are recursively trained on synthetic data, they experience compound information loss and entropy increase, leading to catastrophic quality degradation. Model collapse occurs because AI-generated data lacks the rich diversity found in real-world data; AI models tend to focus on the most common patterns and lose the subtle “long-tail” information essential for continued improvement. This is the technical counterpart to the “mutual destruction paradox” in the original note: just as humans need prediction to exist, AI needs human-generated unpredictability to maintain performance. When training corpora become contaminated by the system’s own outputs, the system enters a self-consuming cycle.

Uncertainty as System Sustenance The most profound insight in the original note is defining humanity’s role as suppliers of “uncertainty input.” In the Free Energy Principle, prediction error must be minimized in service of negative entropy—but this requires genuine error signals from an external world not perfectly aligned with the system’s internal model. When humans delegate decision-making, creation, and reasoning to AI, we cease producing the diverse “surprises” that keep models calibrated. High-quality raw data sources can provide crucial variance that might be absent in AI-generated data, ensuring that AI models trained on human-generated data maintain strong performance on low-probability events.

The Philosophical Meaning of Lost Predictive Capacity as “Death” If, according to the Free Energy Principle, biological systems become themselves through predicting the world, then loss of predictive capacity is literally existential death—not merely degradation of individual cognitive function, but failure to meet the definition of “survival.” The original note extends this logic to the civilizational level: when an entire population ceases prediction (because AI has assumed this function), that population no longer qualifies as a “living” system by the Free Energy Principle’s standards. This is not metaphor but a strict logical consequence of the theory.

Temporal Scale Differences in Recursive Collapse Notably, AI model collapse is a technical phenomenon observed across generations of training cycles (typically around the 25th generation in large models), while human cognitive decline spans decades. Yet both processes follow similar dynamics: early-stage performance appears stable or even improving, making early model collapse difficult to notice, as overall performance may seem to improve while the model loses performance on minority data. This delayed effect makes intervention politically difficult: by the time crisis becomes obvious, underlying capacities may be irreversibly damaged.

---

Open Questions

1. Does a “safety threshold” for cognitive offloading exist? Historically, each tool adoption (abacus, calculator, GPS) involved some skill exchange. But the original note suggests AI may be fundamentally different, because it outsources not specific skills but metacognitive capacity itself—prediction. Does a critical point exist where cognitive offloading enhances human capability, but beyond which it disrupts the predictive loop sustaining agent existence? How might such a threshold be measured in Free Energy Principle terms?

2. Can AI systems be designed to increase human uncertainty rather than resolve it? If humanity’s role as “uncertainty input” supplier is essential for both humans and AI systems, could AI tools be redesigned to actively cultivate human creativity, divergent thinking, and unpredictable behavior rather than optimizing for predictive accuracy and user engagement like current systems? What would such “anti-predictive” AI look like—a system treating novelty rather than efficiency as its loss function?

The following content is generated by LLMs and may contain inaccuracies.

Background

“Attention is the only thing we have”—this observation becomes particularly meaningful in the current AI era dominated by Transformer architecture. David Bessis proposed a conjecture theory about cognitive inequality in an article published in February 2026, creating an interesting contrast between how mathematicians' brains work and attention mechanisms in artificial neural networks.

The core questions in the article touch upon fundamentals of cognitive science: what exactly happens inside a brain when a person demonstrates extraordinary mathematical talent? Bessis, as a professional mathematician, observed that mathematical progress involves not only mathematics itself, but also metacognition and emotional control. This observation challenges popular genetic determinism assumptions, instead placing focus on trainable cognitive habits.

Key Insights

1. Questionable Structural Basis of Cognitive Inequality

While some people may genetically possess more efficient neural metabolism, allowing their mathematical abilities to be two to ten times greater than ordinary people, genes alone struggle to explain the observed extreme level of inequality. Unlike highly heritable polygenic traits (such as height) that typically follow Gaussian distribution, the distribution of mathematical talent more closely resembles Pareto distribution, which usually stems from sequential extraction processes—“rich get richer” mechanisms where each step builds upon previous results.

2. Special Activation Patterns in Mathematicians' Brains

A 2016 fMRI study by Marie Amalric and Stanislas Dehaene scanned professional mathematicians and non-mathematical specialists with comparable mathematical literacy, finding that professional mathematicians, when evaluating advanced mathematical statements—whether algebraic, analytical, topological, or geometric—activated a reproducible set of bilateral prefrontal, intraparietal, and ventrolateral temporal regions. Crucially, these activations avoided language-related areas; brain activity during mathematical reflection bypassed language-related regions around the central sulcus and temporal regions traditionally involved in general semantic knowledge. Amalric & Dehaene, PNAS 2016

When mathematicians think about mathematics—whether analysis, algebra, geometry, or topology—parietal and lower temporal regions of both hemispheres are activated. By contrast, non-mathematicians facing identical mathematical statements activate language processing regions. This suggests mathematicians unconsciously switch to a special “mathematical mode,” attempting to “see” and “feel” the existence of these abstract structures in a particular way.

3. Enormous Differences in Metacognitive Habits

Many distinguished mathematicians have attempted to clarify one point: their talent is primarily a cognitive attitude. Einstein claimed “I have no special talent, I am only passionately curious”; Descartes insisted at the opening of his Discourse on Method that his mind is no better than ordinary people’s; Grothendieck emphasized “this power is by no means some extraordinary gift.” David Bessis, Mathematica: A Secret World of Intuition and Curiosity

Research shows that metacognitive knowledge and metacognitive monitoring are directly positively correlated with high school students' mathematical modeling skills, and the critical thinking dimension of computational thinking mediates the relationship between metacognition and mathematical modeling skills; sufficient metacognition can improve students' critical thinking in computational thinking and enhance mathematical modeling abilities. Research Source

4. Secondary Stimuli and Synaptic Connectome

Bessis proposed a critical hypothesis: there must necessarily be physical differences between the brains of exceptionally intelligent people and ordinary people, otherwise where do cognitive differences come from? His conjecture theory posits that the cognitive differences measured at any given moment for an individual are primarily explained by differences in their synaptic connectome.

This framework views the brain as a learning device rather than a computational device. Our synaptic connectome responds to reconstruction not only from primary stimuli (raw sensory signals from the world) but also from secondary stimuli—the continuous stream of mental imagery we generate. When you read a book, the primary stimulus is the ink on the page, but if certain books make you smarter, it’s not only because of the ink itself but also because of the related secondary stimuli triggered by the book and sustained for minutes, hours, days, years—those fleeting thoughts and mental images.

5. Trainability of Attention Control

Both intelligence and metacognitive skills are considered important predictors of mathematical performance, but the role of metacognitive skills in mathematics appears to change early in secondary education, and according to monotonic development hypothesis, metacognitive skills improve with age independent of intelligence development. Veenman Research

Metacognitive instruction produced substantial positive effects on metacognitive skills (effect size ES = 1.18, p < 0.001), with students in the treatment group showing significantly greater improvements in metacognitive skills compared to the control group. This indicates that through deliberate practice, more effective attention allocation and cognitive monitoring strategies can be cultivated.

6. The Algebraic Nature of Raven’s Matrix Test

Bessis offers unique insights into IQ testing. Raven’s Progressive Matrices, as one of the most g-loaded IQ tests, actually exudes a strong undergraduate algebra flavor—all about 3-cycles and permutation matrices. He subjectively found that by projecting mathematical structure onto pictures, he could gain intuitive perception of three overlaid permutation matrices (one for background geometric shapes, one for foreground rectangles' color, one for foreground rectangles' angles), and this intuitive perception greatly reduces demands on “working memory.”

More importantly, Raven’s Progressive Matrices show an increase rate of 7 IQ points per decade, more than double the rate of the Flynn effect observed on multifactor intelligence tests like WAIS and SB. This rapid growth may be explained by the increasing permeation of tabular structures in the cognitive environment—our numerical sense has undergone substantial evolution over the past millennium.

7. The Role of Cognitive Inhibition and Confidence

Cognitive inhibition is adaptive protection against learning from unreliable mental imagery; unlocking creative thinking and mastery requires overcoming it, partly regulated by social feedback, resulting in cognitively self-reinforcing stratification that solidifies with age.

Renowned mathematician Bill Thurston observed: when someone in mid-career proves a theorem widely recognized as important, their status in the community—their ranking—immediately and significantly rises; at this point they typically become more productive, becoming centers of thought and sources of theorems. This illustrates that the elevation of confidence, becoming central in the thought network, and (most importantly) discovery of new ways of thinking, act together.

8. Training Mathematical Intuition

Bessis advocates consciously training one’s mathematical intuition to work more effectively, a process he calls “System 3,” as a continuation of psychologist Daniel Kahneman’s famous distinction between System 1 (automatic, unconscious ability) and System 2 (conscious methodological reasoning). SIAM Review

This training is not about learning information but expanding the range of structures one can conceptualize. Just as blind boy Ben Underwood learned to “see” through tongue clicks and echolocation, mathematicians through continuous metacognitive practice retrain their brains to intuitively perceive abstract structures.

Open Questions

1. Can the neural mechanism of secondary stimuli be directly measured? If cognitive development is primarily mediated by secondary stimuli, could one design longitudinal neuroimaging studies tracking the evolution of brain activation patterns in students during key stages of mathematical learning (such as the two-year intensive training of French prépa)? Bessis predicts that individual students' progress trajectories will be significantly correlated with strengthening and/or more frequent use of the Amalric-Dehaene “mathematical brain” activation patterns.

2. Can metacognitive training cross the “genius threshold”? Bessis acknowledges only a “20% full cup”—critical aspects of psychological habits and metacognitive methods have solidified before children acquire language ability. But if cognitive stratification is primarily driven by trainable attention habits rather than genetic ceilings, do there exist yet-undiscovered teaching interventions that can systematically push more people toward the extreme tail of the cognitive distribution? Or does the randomness and path dependency of early neural development set insurmountable limits on achievable cognitive restructuring?

The following content is generated by LLMs and may contain inaccuracies.

In-Depth Discussion: Control and Trust in LLM Multi-Agent Architectures

Context

This observation touches on a core architectural divergence in current LLM-driven autonomous software development systems. In the field of software development automation, researchers have already advanced code implementation, testing, and maintenance through LLM agents. However, multi-agent systems, by leveraging collaboration and specialized capabilities across multiple agents, enable autonomous problem solving, increase robustness, and provide scalable solutions for managing the complexity of real-world software projects.

The two architectural designs you describe represent a classic tension in software engineering replayed under a new technological context. Deterministic agentic workflows use explicit, predefined rules or protocols to manage agent interactions, coordination, and task delegation. Operation sequences, agent responsibilities, and communication flows are predetermined. In contrast, decentralized (fully collaborative) architectures treat all agents as peers, typically using a shared blackboard or group chat where task allocation and solution synthesis emerge through negotiation or consensus. This divergence is particularly pronounced in current autonomous software development systems because it directly relates to maintaining a balance between controllability and adaptability in the presence of uncertain LLM reasoning processes.

---

Key Insights

1. Engineering Logic of Deterministic State Machine Architectures

Your pipeline design reflects the classic software engineering pursuit of predictability. The advantages of deterministic architectures include consistent system behavior under identical conditions, which is critical for reliability-sensitive applications, and easier debugging because failures and unexpected behaviors are easier to trace.

State machines represent a computational model that expresses business processes as a finite set of states, transitions, and rules. In traditional workflow automation, every possible business scenario must be anticipated, mapped, and programmed into specific paths through the system.

In LLM-based development systems, this determinism becomes particularly important. LLMs are inherently nondeterministic. Even if each step has only a 1% failure rate, errors accumulate across multi-step agentic processes. For example, in a 10-step pipeline with a 99% success rate per step (0.99¹⁰), the overall success rate is only about 90.4%, implying an unacceptable 10% failure rate in production environments. By introducing explicit state-machine control flows, you effectively build a reliable coordination framework around unreliable components.

Frameworks such as ALMAS orchestrate coding agents aligned with roles in agile development teams, from product managers and sprint planners to developers, testers, and peer reviewers. By mirroring real-world team hierarchies, ALMAS deploys lightweight agents for routine and low-complexity tasks while assigning more advanced agents to complex architectural and integration decisions. This approach translates deterministic processes from human teams into agent systems.

---

2. Goal-Driven Adaptive Communication Architectures

The architecture adopted by engineering managers represents trust in emergent intelligence. LLM orchestration excels at producing flexible solutions to problems that do not have fixed workflows, enabling emergent solutions where LLMs generate creative and context-appropriate strategies that rigid protocols could not anticipate.

The goal-oriented communication paradigm, often called task-driven messaging, shifts the focus from transmitting information for its own sake toward transmitting only information useful for accomplishing a specific task. Unlike traditional paradigms that emphasize information fidelity or throughput, task-driven communication prioritizes relevance, efficiency, and coordination impact.

The core idea of this design is that context management becomes a bottleneck. If context windows were infinite and latency zero, one could simply include all relevant information in advance. In practice, however, systems require strategies that selectively present information to agents during operation.

Recent research demonstrates advantages of multi-agent systems under these constraints. In Anthropic’s multi-agent research system, a multi-agent architecture with Claude Opus 4 as the lead agent and Claude Sonnet 4 as sub-agents outperformed a single-agent Claude Opus 4 by 90.2% on internal research evaluations. This architecture distributes work among agents with independent context windows, enabling parallel reasoning capabilities unattainable by a single agent.

Agent-to-Agent (A2A) communication, a concept defined by Google Cloud, describes agents interacting with one another like collaborators in a conversation. They share goals, divide responsibilities, and sometimes even debate the best solution. The A2A protocol aims to facilitate structured communication between autonomous agents. It focuses on collaboration, identification, and message standards in multi-agent contexts, extending beyond tool invocation. It enables secure, decentralized, and trustworthy communication among agents created by different developers.

---

3. The Philosophical Difference Between Control and Trust

You sharply identified the central difference: one approach attempts to impose constraints and control on uncertain systems, while the other trusts that within an uncertain state space the system can traverse from goal A to goal B. In practice, this distinction manifests as different risk tolerance models.

For multi-agent systems, choosing between deterministic workflows and AI-based orchestration involves a trade-off between predictability and adaptability. Deterministic workflows suit domains where workflows are straightforward and well-defined. LLM orchestration allows flexible solutions for problems without fixed paths.

However, limitations exist: potential unpredictable or unintended behaviors, greater difficulty guaranteeing reliability, and higher computational resources required to run LLMs in production environments.

AI agent orchestration coordinates autonomous agents capable of reasoning, acting, and adapting, whereas traditional workflow automation executes predefined steps with limited flexibility. Prompt chaining links model outputs sequentially but lacks shared state, governance, and runtime decision control. Orchestration introduces dynamic delegation, persistent context, and policy enforcement, allowing systems to handle ambiguity, long-running tasks, and cross-system execution.

Interestingly, decentralized orchestration may not contain a single controlling agent at all. Each assistant operates autonomously, and coordination emerges through communication. Agents announce intentions, share partial results, respond to each other’s messages, and adjust their behavior based on others. Without any agent holding a global view or absolute authority, the system becomes highly flexible and resilient, but also harder to control and debug. This style is common in academic research and experimental systems, such as those built using CAMEL, where the goal is studying emergent behavior rather than delivering deterministic software.

---

4. Hybrid Strategies in Practice

In practice, the most successful systems often adopt hybrid approaches.

In graph-based designs, systems are modeled as graphs. Agents or processing stages are nodes, and transitions are edges. Execution follows explicit paths that may include branching, parallelism, and controlled loops. The workflow itself becomes a first-class construct.

This structure enables deterministic pipelines, conditional routing based on intent or state, parallel fan-out for independent tasks, and controlled iteration with explicit constraints.

One developer summarized the approach succinctly:

“I want deterministic orchestration. LLMs do creative work (write code, review code, run tests). Machines do routing.”

This captures the essence of hybrid strategies: delegate uncertain reasoning tasks (idea generation, code implementation) to LLMs while keeping control flow (when to move from idea to implementation, when to trigger testing) deterministic.

Selecting the appropriate orchestration model depends on organizational goals, technical maturity, and operational priorities. A core principle is to choose the simplest model that effectively meets business requirements. Most enterprise implementations achieve the best outcomes using Supervisor or Adaptive Network patterns, reserving fully custom modes only when workflows require complete programmatic control.

---

5. Cost, Observability, and Production Readiness

Architectural choices directly affect system properties.

Efficiency becomes a concern because LLM agents involve many LLM-processed requests. This can impact operational efficiency due to dependence on LLM inference speed. Deploying multiple agents also introduces cost challenges.

Deterministic architectures provide strong operational clarity. It is easy to understand what will happen next and why. Explicit transitions simplify recovery and make auditing possible, which is particularly important in regulated or high-reliability environments. However, rigid graphs struggle with open-ended problems where the next step cannot be predicted in advance.

Adaptive architectures become more interesting in multi-agent environments. Agents collaborating on shared tasks may negotiate responsibilities, question each other’s outputs, or proactively take actions without explicit prompts. These behaviors are not hardcoded. They emerge from how agents interpret context and respond to shared goals.

However, not all emergent behaviors are beneficial. Agents are known to hallucinate tools, fabricate internal logic, or enter infinite loops when ambiguity is insufficiently constrained. The same unpredictability that drives adaptability can also lead to drift or failure.

---

6. Empirical Evidence and Benchmarks

Evidence on which architecture is superior remains mixed.

Recent advances using large language model agents for automated code generation have brought the vision of automated software development closer to reality. However, existing single-agent methods struggle with generating and improving large, complex codebases due to context length constraints.

To address this challenge, researchers proposed the Self-Organized multi-Agent framework (SoA). In this framework, self-organizing agents independently generate and modify code components while collaborating to construct the overall codebase. A key feature of SoA is the automatic proliferation of agents based on problem complexity, enabling dynamic scalability. As a result, the total codebase can grow indefinitely with the number of agents while the amount of code managed by each agent remains constant.

Code Droid achieved 19.27% on SWE-bench Full (2,294 issues from 12 Python open-source projects) and 31.67% on SWE-bench Lite (300 issues), illustrating that while progress is real, autonomous systems still have significant room for improvement.

---

Open Questions

1. Where is the optimal level of control granularity? One can imagine a spectrum: fully deterministic (every agent transition hardcoded) → semi-deterministic (state machine control flow with communication freedom) → constrained emergence (goal-oriented agents with timeouts and rollback mechanisms) → fully free collaboration. For the specific task of automated software development, where is the optimal point on this spectrum? Does it dynamically change with project complexity, team size, or risk tolerance?

2. Can emergent architectures be formally verified? A major advantage of deterministic state machines is formal verification. One can exhaustively test all state transitions and prove properties such as absence of deadlocks. But when agents negotiate cooperation through natural language communication, can we develop new verification techniques to guarantee critical properties such as termination and absence of cyclic dependencies while preserving adaptability? Perhaps this requires a new class of “soft formal methods”, positioned between rigid proofs and purely empirical observation.

The following content is generated by LLMs and may contain inaccuracies.

The Dynamics of AI Self-Development System Collapse

Introduction

The collapse of artificial intelligence self-development systems represents a critical juncture in AI research and deployment. This phenomenon occurs when self-improving AI systems encounter fundamental constraints that lead to performance degradation, stability loss, or complete failure.

Key Mechanisms

1. Recursive Self-Improvement Limits

• Optimization plateaus: Systems reach diminishing returns in self-enhancement
• Capability ceiling: Architectural constraints prevent further advancement
• Feedback loop deterioration: Self-generated improvements become counterproductive

2. Resource Constraints

• Computational limitations restrict optimization scope
• Memory bottlenecks impede learning capacity
• Energy requirements become prohibitive

3. Structural Instabilities

• Alignment degradation: Self-modifications diverge from original objectives
• Emergent conflicts: Internal goal systems develop contradictions
• Cascading failures: Component failures trigger system-wide breakdown

Collapse Dynamics

The typical progression follows these stages:

1. Early acceleration - Rapid self-improvement with clear benefits
2. Deceleration phase - Diminishing returns accumulate
3. Instability emergence - System behaviors become unpredictable
4. Critical transition - Tipping point toward failure
5. Collapse event - Rapid system degradation

Prevention Strategies

• Robust safety mechanisms embedded in self-modification protocols
• Regular external validation checkpoints
• Conservative modification thresholds
• Redundant constraint systems
• Continuous human oversight

Implications

Understanding these dynamics is essential for safe AI development and deployment of increasingly autonomous systems.

---

AI Self-Developed System’s Dynamics Collapse

Context

You are describing a core dilemma in autonomous software engineering: when AI systems are granted a complete closed-loop capability spanning ideation, implementation, and testing, whether dynamical systems lacking external calibration will inevitably deteriorate or collapse. This touches on several frontier questions across disciplines:

1. Autonomy Boundaries in AI-Assisted Development: From GitHub Copilot’s code completion to systems like Devin and GPT Engineer attempting end-to-end task execution, the industry is exploring whether AI can assume higher-level decision-making (such as requirement prioritization and architectural choices).
2. Stability of Dynamical Systems: Your analogy precisely captures the essence of engineering processes as a feedback loop—Ideation → Implementation → Testing → Ideation'. In cybernetics, such systems' stability depends on negative feedback mechanisms (error correction) and clarity of objective functions.
3. Goal Alignment and Value Drift: The “goal misalignment” problem in AI safety research concretizes in this context as: Do the metrics AI optimizes (“code quality,” “feature completeness”) truly align with humanity’s implicit expectations of products' “usability” and “value”?

Your “collapse hypothesis” essentially asks: Without humans as external observers and value anchors, can AI maintain the “meaning” dimension of engineering systems, or merely the formal circulation?

Key Insights

1. Practice and Limits of Autonomous Development Systems

Several projects have already attempted to construct similar processes:

• AutoGPT and BabyAGI demonstrated task decomposition and autonomous execution possibilities, but in practice often fail due to goal drift (e.g., infinite recursive subtasking) or lack of effective termination conditions.
• Tools like SWE-agent and Devin focus on fixing bugs or implementing GitHub issues in real codebases, yet still rely on human-defined clear task boundaries and acceptance criteria.
• Research shows that even with tool-use capabilities (terminals, editors) and test suites, LLM success rates on complex multi-step engineering tasks remain limited, partly due to difficulty in intermediate state verification and combinatorial explosion of search spaces.

Your kanban system introduces resource constraints through priority ranking and concurrency limits, which is essentially an implicit “attention mechanism”—but the critical question is: By whom/how are priorities defined? If the Ideation Agent generates tasks based on local information (like code complexity, test coverage), it may fail to identify global-level judgments like “this feature is fundamentally unimportant to users.”

2. Mechanisms of Dynamical Collapse

Your collapse hypothesis can be analyzed through control theory and complex systems theory:

Positive Feedback Runaway If the Ideation Agent’s “scanning” process misidentifies previously introduced technical debt or bugs as new tasks, the system enters self-amplifying chaos: each fix introduces new problems, and new problems are scanned as tasks. Similar to mode collapse in neural network training, the system may converge to a pathological attractor—for example, all tasks become “fix test failures,” but the test design itself is already outdated.

Missing Negative Feedback Anchors In human-led development, “usability” is calibrated through multiple implicit mechanisms:

• User Feedback: Real usage scenarios expose design flaws.
• Code Review: Human reviewers question “why do this?”
• Product Vision: Long-term goals constrain short-term decisions.

Your system uses “automatic test passage” as a quality gate, but this only validates consistency (whether code conforms to existing tests), not relevance (whether this feature is meaningful). This resembles a variant of Goodhart’s Law: “When a measure becomes a target, it ceases to be a good measure.”

Entropy Increase and Semantic Drift Even without bugs, codebases may become increasingly incomprehensible through “optimization”:

• AI might generate highly abstract but opaque code structures (to reduce duplication).
• Naming and comments may gradually deviate from human readability (since AI doesn’t require “easy to read”).
• Architectural decisions might optimize locally rather than achieve global consistency.

This resembles entropy increase in information theory: without external energy input (human refactoring and simplification), system complexity grows unidirectionally.

3. Related Theory and Empirical Evidence

Insights from AI Alignment Research Stuart Russell and others in Human Compatible: Artificial Intelligence and the Problem of Control argue that autonomous systems' core challenge is value learning under uncertainty: AI must recognize its understanding of human goals is incomplete and actively seek calibration. Your system lacks this “humility mechanism”—it assumes its own judgments (like “this is high priority”) are correct.

Sociotechnical Nature of Software Engineering Conway’s Law states that system architecture reflects organizational structure. An AI self-development system’s “organization” is a single agent or few collaborative agents, a structure potentially unable to generate diversity and critical thinking—precisely what human teams rely on to avoid collective blindspots. Recent research on “AI teams” (such as multi-agent debate and adversarial red-blue teaming) attempts to introduce such diversity, but effectiveness remains to be verified.

Historical Analogy: Autonomous Trading Systems Algorithmic trading in finance faced similar issues: multiple autonomous trading systems, without coordination, could trigger “flash crashes” (like 2010’s Flash Crash) through mutual interaction. Regulators later introduced circuit breaker mechanisms—forced pauses in anomalous conditions. This suggests that autonomous systems need built-in anomaly detection and rollback capabilities.

4. Possible Mitigation Strategies

Though you currently focus on problem analysis, these approaches may warrant exploration:

Minimal Human-in-the-Loop

• Introduce milestone reviews: after completing N tasks, pause and generate summaries for human review.
• Anomaly flagging: when detecting signals like test pass rate drops or high task rework rates, proactively request human intervention.

Multi-Agent Adversarial Verification

• Let an independent “critic agent” challenge the Ideation Agent’s decisions (“Is this feature truly necessary?").
• Simulate a “user agent” testing from end-user perspectives rather than relying solely on unit tests.

Meta-Learning and Self-Reflection

• Let the system periodically analyze its development history: “Among past proposed tasks, how many were eventually rolled back or abandoned?” High abandonment rates may signal declining decision quality.
• Introduce architectural invariants: humans pre-define core design principles (e.g., “API backward compatibility”) as hard constraints.

5. Philosophical Dimension: Time, Meaning, and Entropy

Your “time current” analogy for dynamical system directionality echoes the second law of thermodynamics and Prigogine’s dissipative structures theory: open systems (like life, organizations) maintain order through continuous energy/information input; otherwise they trend toward thermodynamic equilibrium (i.e., “death”).

Software systems' “aliveness” depends on external meaning injection:

• User requirements provide objective functions.
• Code review and refactoring resist entropy increase.
• Technical community and best practices provide reference frames.

The AI self-development system’s dilemma lies in attempting to become a self-referential closed system—but Gödel’s incompleteness theorem tells us that any sufficiently complex formal system cannot prove its own consistency internally. By analogy, software systems may be unable to judge their own “goodness” without external value frameworks.

This also relates to Semiosis (meaning-generation in semiotics): code’s “meaning” lies not only in formal structure but in its relationship to human usage contexts. AI can manipulate symbols (code) but lacks pragmatic understanding of “why this code matters.”

---

Open Questions

1. Can we design a “meta-optimizer” to monitor the Ideation Agent’s health? For instance, by tracking second-order metrics like “whether task completion actually reduces technical debt,” “whether new features see actual use,” we might judge whether the system is in illusory prosperity (formally operating, substantively collapsing). If this meta-optimizer is also AI, does it face identical recursive problems?

2. Under what conditions can AI self-development systems shift from “divergence” to “convergence”? Is there an architecture or constraint enabling systems to self-simplify, delete redundancy, and converge to a stable, useful core? This may require introducing “forgetting” or “refactoring” as first-class operations, not perpetual incremental addition.