ReplicHAI › Agent & LLM run-to-run variability

Shared by Changkun Ou with everyone

Year ↓	Title added	Authors	Venue
2026	AI Coding Agents in Social Science: Methodologically Diverse, Empirically Consistent, Interpretively Vulnerable	Alizadeh et al.	arXiv
2026	Automated Reproducibility Has a Problem Statement Problem	Snelleman et al.	RAI Workshop at AAAI 2026
2026	Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents	Khanal et al.	arXiv
2026	Deployment Decision Reliability: A Generalizability-Theory Framework for Sizing Long-Horizon Agent Evaluations	Srinivasan	arXiv
2026	From paper to benchmark: agentic, framework-based reproduction of under-specified methods in machine health intelligence	Theiler et al.	arXiv
2026	Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking	Messing	arXiv
2026	How Consistent Are LLM Agents? Measuring Behavioral Reproducibility in Multi-Step Tool-Calling Pipelines	Yagubyan	arXiv
2026	It's Not Just the Prompt: Model Choice Dominates LLM Creative Output	Haase et al.	Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems
2026	Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theory	Zhou et al.	arXiv
2026	Many AI analysts, one dataset: Navigating the agentic data science multiverse	Bertran et al.	Proceedings of the National Academy of Sciences
2026	Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results	Kohler et al.	arXiv
2026	Within-Model vs Between-Prompt Variability in Large Language Models for Creative Tasks	Haase et al.	arXiv
2025	PaperBench: Evaluating AI's Ability to Replicate AI Research	Starace et al.	Proceedings of the 42nd International Conference on Machine Learning
2025	Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks	Haldar, Hockenmaier	Findings of the Association for Computational Linguistics: EMNLP 2025
2025	ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers?	Ye et al.	arXiv
2024	Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations	Miller	arXiv
2021	Consensus-based guidance for conducting and reporting multi-analyst studies	Aczel et al.	eLife
2017	The Coefficient of Determination R^2 and Intra-Class Correlation Coefficient from Generalized Linear Mixed-Effects Models Revisited and Expanded	Nakagawa et al.	Journal of the Royal Society Interface
2001	Generalizability Theory	Brennan	Springer
1996	Forming Inferences About Some Intraclass Correlation Coefficients	McGraw, Wong	Psychological Methods

/ filter · j k or ↑ ↓ select · Enter open page · c copy BibTeX