Human Generalization Over Token Accumulation人类泛化能力 vs 代币积累

The following content is generated by LLMs and may contain inaccuracies.

Context

This idea sits at the intersection of cognitive science, machine learning theory, and the epistemology of expertise. It contrasts two models of intelligence: the human strength of generalization sample efficiency — learning powerful abstractions from very few examples — versus the increasingly dominant industry metric of token accumulation (years of practice, or literally the number of training tokens “burned”). The tension it addresses is timely: as large language models scale by consuming trillions of tokens, there’s a cultural drift toward valuing sheer accumulation as a proxy for competence and endorsement. The note argues this accumulation gives safety and a bar, but that exponential growth comes instead from good intuition and curiosity, which are largely shaped in a “pre-training” phase analogous to early-stage education and environment.

Key Insights

• Sample efficiency is humanity’s signature advantage. Humans (and children especially) generalize from a handful of examples, where deep learning systems often require orders of magnitude more data. This gap is a central theme in Lake, Ullman, Tenenbaum & Gershman, “Building Machines That Learn and Think Like People”, who argue human learning leverages compositionality, causal models, and learning-to-learn rather than brute pattern accumulation.

• The “pre-training” analogy is apt but double-edged. In ML, pre-training on broad data builds priors that make downstream few-shot learning efficient — the argument in the note that intuition/curiosity “comes from pre-training, aka early stage education and environment.” This mirrors developmental findings that early environment shapes later learning capacity (see the “learning to learn” or meta-learning framing in Thrun & Pratt, Learning to Learn). The double edge: if early priors are impoverished, the generalization advantage never fully develops — echoing environmental effects on cognitive development.

• Curiosity as an intrinsic driver of efficient learning. The claim that curiosity fuels exponential growth is supported by work on intrinsic motivation and curiosity-driven exploration, e.g. Pathak et al., “Curiosity-driven Exploration by Self-supervised Prediction”, where prediction-error-based curiosity dramatically improves learning without external reward. Curiosity effectively selects high-information samples, boosting effective sample efficiency.

• Accumulation buys a floor, not a ceiling. The note’s concession is important: years of practice / tokens burned provide “some degree of safety and a bar.” This aligns with the deliberate-practice literature (Ericsson et al.), which shows accumulation reliably produces competence — but competence and generalizing breakthroughs are not the same axis. Endorsement systems (credentials, seniority, benchmark token counts) reward the reliable floor because it is measurable, not because it captures the intuition that produces leaps.

• The scaling-law counterpoint. The industry’s bet on token accumulation is not irrational: Kaplan et al., “Scaling Laws for Neural Language Models” and the compute-optimal reweighting in Hoffmann et al., “Chinchilla” show predictable capability gains from more tokens/compute. The note’s implicit critique is that these are linear-to-power-law returns on a substrate lacking the architectural priors that make human learning exponentially efficient per example — capability rises, but per-token efficiency does not approach human levels.

Open Questions

• If human-level generalization stems from priors laid down in a “pre-training” developmental window, can we deliberately engineer curiosity and intuition priors into learning systems — or into education — rather than relying on token/practice accumulation to eventually approximate them?
• Endorsement mechanisms reward accumulation because intuition is hard to measure; what would a credible signal for sample-efficient generalization look like, and would trusting it change who and what we value in both AI and human expertise?