"""Exercise 16.1: precision, recall, and checking cost of a model critique (constructed data). The bookkeeping program has 7 seeded bugs (known to the instructor). A critique model flags 8 locations. The reviewer checks each flag; 5 are real bugs, 3 are not. Checking takes the minutes listed. A control reviewer, given the same total minutes and no critique, finds 3 of the 7 bugs. """ flags = [ # (flag id, real bug?, minutes to check) ("F1", True, 4), ("F2", True, 6), ("F3", False, 9), ("F4", True, 3), ("F5", False, 7), ("F6", True, 5), ("F7", False, 8), ("F8", True, 6), ] seeded = 7 tp = sum(1 for _, real, _ in flags if real) fp = len(flags) - tp minutes = sum(m for *_, m in flags) fp_minutes = sum(m for _, real, m in flags if not real) print(f"flags={len(flags)} true={tp} false={fp}") print(f"precision = {tp}/{len(flags)} = {tp/len(flags):.1%}") print(f"recall = {tp}/{seeded} = {tp/seeded:.1%}") print(f"checking time = {minutes} min, of which on false flags {fp_minutes} min ({fp_minutes/minutes:.0%})") print(f"minutes per real bug found = {minutes/tp:.1f}") control_found = 3 print(f"control (same {minutes} min, no critique): recall = {control_found}/{seeded} = {control_found/seeded:.1%}")