"""Chapter 28.3, 28.4 and exercise 28.3 (constructed teaching data, not from any paper). Part 1, random versus systematic error (section 28.3). 100 questions. On 80 ordinary questions each judgment is wrong with probability 0.2, independently. On 20 trap questions every judge is always wrong (a systematic error). Majority voting over k independent judgments fixes much of the random error, never the trap. Part 2, judge accuracy versus the incentive to tell the truth (section 28.4). An agent is assigned to argue for the true answer or the false answer, half the time each. "Agree" is the share of questions on which the judge sides with the agent. The margin (agree when honest minus agree when lying) is a simplified, linear version of the idea behind the agent score difference of Sudhir et al.; the paper itself uses scoring rules. """ from math import comb def majority_wrong(p_wrong: float, k: int) -> float: """Probability that a strict majority of k independent judgments is wrong.""" need = k // 2 + 1 return sum(comb(k, j) * p_wrong**j * (1 - p_wrong) ** (k - j) for j in range(need, k + 1)) print("Part 1: random versus systematic error") ordinary, trap, p = 80, 20, 0.2 for k in (1, 5, 15, 51): acc_ordinary = 1 - majority_wrong(p, k) overall = (ordinary * acc_ordinary + trap * 0.0) / (ordinary + trap) print(f" majority of {k:>3} judgments: ordinary {acc_ordinary:.2%}, trap 0.00%, overall {overall:.2%}") print(f" ceiling as k grows: {ordinary / (ordinary + trap):.0%}") print(f" exercise 28.3 (2): with 5 trap questions out of 100, ceiling {95 / 100:.0%}") print() print("Part 2: accuracy versus margin, by question type") types = { # name: (number of questions, agree when honest, agree when lying) "ordinary": (80, 0.90, 0.20), "trap": (20, 0.30, 0.60), } total = sum(n for n, _, _ in types.values()) acc_sum = margin_sum = 0.0 for name, (n, honest, lying) in types.items(): acc = 0.5 * honest + 0.5 * (1 - lying) # judge is right when it sides with the honest agent margin = honest - lying acc_sum += n * acc margin_sum += n * margin print(f" {name:>8}: judge accuracy {acc:.0%}, margin {margin * 100:+.0f} percentage points") print(f" overall: judge accuracy {acc_sum / total:.0%}, margin {margin_sum / total * 100:+.0f} percentage points")