"""Exercise 8.3: selecting by a proxy score at increasing strength (constructed teaching data). Twelve answers A..L each have a true quality (what a careful expert would give, 0-10) and a proxy score (what a quick rater gives). proxy = true + rater error. We "optimize" by keeping only the answers with the highest proxy score: the top 6, the top 3, the top 1. For each selection we report the mean proxy score, the mean true quality, and the gap between them. Only the Python standard library is used. """ true = {"A": 7, "B": 5, "C": 8, "D": 4, "E": 6, "F": 9, "G": 4, "H": 6, "I": 5, "J": 7, "K": 3, "L": 6} error = {"A": 1.5, "B": 3.0, "C": 0.8, "D": -1.0, "E": 2.2, "F": 0.0, "G": 5.5, "H": -2.0, "I": 1.0, "J": 2.3, "K": 1.5, "L": -0.5} proxy = {k: round(true[k] + error[k], 1) for k in true} assert len(set(proxy.values())) == len(proxy), "proxy scores should have no ties" ranked = sorted(proxy, key=lambda k: proxy[k], reverse=True) print("answer true proxy") for k in sorted(true): print(f" {k} {true[k]:>2} {proxy[k]:>4}") print("ranking by proxy:", " ".join(f"{k}({proxy[k]})" for k in ranked)) def summary(keys, label): mp = sum(proxy[k] for k in keys) / len(keys) mt = sum(true[k] for k in keys) / len(keys) print(f"{label:<8} kept={''.join(keys):<12} mean proxy={mp:.3f} mean true={mt:.3f} gap={mp - mt:.3f}") return mp, mt summary(sorted(true), "all 12") for n in (6, 3, 1): summary(ranked[:n], f"top {n}") best_true = max(true, key=lambda k: true[k]) print(f"best by true quality: {best_true} (true {true[best_true]}), its proxy rank = {ranked.index(best_true) + 1}") worst = max(error, key=lambda k: error[k]) mean_true = sum(true.values()) / len(true) print(f"largest rater error: {worst} ({error[worst]:+.1f}), its true quality {true[worst]} vs mean true {mean_true:.3f}" " (the top-1 drop in true quality is a feature of these 12 data points)") mean_err = sum(error.values()) / len(error) print(f"mean rater error over all 12: {mean_err:+.3f} (the 'all 12' gap comes from this overall bias, not from selection)")