Changkun's Blog欧长坤的博客

Sharing and recording scattered thoughts and writings.

在这里分享并记录一些零散的想法及写作。

10 / 49 ideas

Expertise as Risk Control in Human-AI Optimization专业知识作为人工智能优化中的风险控制

A few years ago, together with Sven Mayer and Andreas Butz, we published a paper examining how human expertise impacts the human-AI optimization loop: https://arxiv.org/abs/2302.05665

At the time, the debate surrounding this work centered on determining which level of expertise is sufficient to achieve a user’s goals, whereas our paper primarily focused on evaluating judgments. In retrospect, we were fortunate the paper was published. Importantly, the core implications remain largely unchanged, even with the advent of LLMs: there is no final answer because everything involves trade-offs—or what engineers call Pareto sets. LLMs simply push the boundary further to observe. They make shallow competence appear powerful, which obscures the underlying expertise gap rather than diminishing its importance.

I believe treating expertise as a category or threshold is a mistake. Instead, we should view it as a risk control system. The underlying core capability for intelligence remains unchanged: knowing what’s unknown (calibration), spotting hallucinations (error detection), and connecting to reality (sensemaking).

The following content is generated by LLMs and may contain inaccuracies.

Context

This work sits at the intersection of human-computer interaction, AI optimization, and epistemic uncertainty — fields increasingly relevant as LLMs democratize access to AI capabilities while obscuring their limitations. The core tension: in human-AI collaborative optimization, should we treat user expertise as a binary threshold for participation, or as a continuous variable that shapes interaction quality and risk? The original IUI 2023 paper found that novices achieve expert-level output quality but terminate optimization earlier with higher satisfaction, while experts iterate longer, exploring more diverse solutions despite lower satisfaction. This challenges assumptions about “sufficient expertise” and highlights a Pareto frontier problem — there’s no universally optimal stopping point, only trade-offs between solution diversity, iteration cost, and subjective confidence.

Key Insights

The shift from expertise-as-threshold to expertise-as-risk-control aligns with emerging work on AI calibration and human trust dynamics, where the gap between perceived and actual model capability creates systematic failures. LLMs amplify this: they produce fluent, confident-sounding outputs that mask underlying brittleness, making shallow competence appear robust and reducing users' ability to detect when models hallucinate or drift from reality. This echoes research on automation bias, where over-reliance on AI tools degrades human metacognitive monitoring.

Framing expertise as calibration (knowing unknowns), error detection (spotting hallucinations), and sensemaking (grounding in reality) connects to Kahneman’s distinction between System 1 and System 2 thinking: experts don’t just produce better solutions — they maintain skeptical, iterative engagement with AI outputs, refusing premature closure. This reframes novice “satisfaction” not as success, but as potentially dangerous overconfidence in underexplored solution spaces.

Open Questions

How might we design interfaces that make expertise gaps visible rather than hidden — e.g., by exposing model uncertainty, alternative solutions, or iteration histories that prompt deeper exploration? Could we quantify the cost of premature optimization termination in domains where unexamined risks compound over time (e.g., medical diagnosis, policy design)?

几年前，我与Sven Mayer和Andreas Butz联合发表了一篇论文，研究人类专业知识如何影响人工智能优化循环：https://arxiv.org/abs/2302.05665

当时，围绕这项工作的辩论主要集中在确定哪个专业知识水平足以实现用户目标上，而我们的论文主要关注评估判断。回顾往事，我们很幸运这篇论文得以发表。重要的是，核心implications即使在大语言模型出现后仍然基本保持不变：没有最终答案，因为一切都涉及权衡——工程师称之为帕累托集合。大语言模型只是进一步推动边界来观察。它们使浅层能力看起来强大，这掩盖了潜在的专业知识差距，而非减少其重要性。

我认为将专业知识视为一个类别或阈值是一个错误。相反，我们应该将其视为一个风险控制系统。智能的核心能力保持不变：了解未知的东西（校准）、识别幻觉（错误检测）和与现实相连接（意义制造）。

以下内容由 LLM 生成，可能包含不准确之处。

背景

这项工作位于人机交互、人工智能优化和认识论不确定性的交叉点——这些领域随着大语言模型民主化了人工智能能力的使用，同时掩盖了其局限性而变得日益相关。核心矛盾在于：在人-人工智能协作优化中，我们应该将用户专业知识视为参与的二元阈值，还是作为塑造交互质量和风险的连续变量？最初的IUI 2023论文发现，新手能够达到专家级输出质量，但优化结束更早，满意度更高，而专家迭代时间更长，探索更多样化的解决方案，尽管满意度较低。这挑战了关于"充分专业知识"的假设，突出了帕累托前沿问题——没有普遍最优的停止点，只有解决方案多样性、迭代成本和主观信心之间的权衡。

关键洞见

从专业知识作为阈值向专业知识作为风险控制的转变，与人工智能校准和人类信任动态的新兴研究相一致。在这类研究中，感知模型能力与实际能力之间的差距会导致系统性失败。大语言模型放大了这一点：它们产生流畅、自信的输出，掩盖了潜在的脆弱性，使浅层能力看起来稳健，削弱了用户检测模型幻觉或偏离现实时的能力。这呼应了关于自动化偏见的研究，即过度依赖人工智能工具会降低人类元认知监测能力。

将专业知识界定为校准（认识未知）、错误检测（发现幻觉）和意义建构（植根于现实），与卡尼曼关于系统1和系统2思维的区分相联系：专家不仅产生更好的解决方案——他们对人工智能输出保持怀疑、迭代的参与，拒绝过早结束。这将新手"满意度"重新界定，不是成功，而是对欠探索解决方案空间的潜在危险过度自信。

开放性问题

我们如何设计界面，使专业知识差距可见而非隐藏——例如，通过暴露模型不确定性、替代解决方案或促进更深入探索的迭代历史？我们能否量化在风险随时间复合的领域（如医学诊断、政策设计）中过早优化结束的成本？

Language-Centric AI While Human Cognition Shifts Toward Visual-Spatial Thinking以语言为中心的人工智能，而人类认知转向视觉-空间思维

From a Sapir-Whorf perspective, one could argue that LLMs excel because they simulate the linear structure of language and, by extension, the structure of reasoning itself. This aligns nicely with a Wittgenstein-style view in which thought is fundamentally language-bound, or at least becomes intelligible only through language.

For a long time, I almost fully believed this framing.

That confidence began to erode when I started paying closer attention to Generation Z, who are growing up fully immersed in modern digital environments. Several patterns appear consistently: 1) less reliance on linear, language-centric reasoning; 2) stronger dependence on visual representations; 3) communication patterns that are compositional and spatial rather than sequential.

This feels like a fundamental shift in cognitive structure, where thinking seems less anchored in linear linguistic narratives and more scaffolded by external systems that manage sequencing, memory, and coherence on the user’s behalf. In other words, modern software increasingly carries the burden of maintaining linear structure.

It’s well known that LLMs and adjacent technologies have begun to offload key cognitive processes, and research (e.g., https://arxiv.org/abs/2506.08872) has criticized how younger generations fail to develop certain critical thinking skills that our generation believed to be crucial. I think this offloading enables rapid context switching across tasks, ideas, and modalities. The effect aligns closely with findings from our prior work on short-form video consumption (https://arxiv.org/abs/2302.03714), where fragmented attention patterns reshape how intentions are formed, sustained, and abandoned. For adults, especially those trained in long-horizon, language-heavy problem solving, this dynamic may lead to a paradoxical outcome: while LLMs dramatically increase efficiency, they also make it harder to retain stable intentions, increasing the risk of cognitive overload rather than reducing it.

What is less clear, and more interesting, is whether the same overload applies to younger generations. I tend to believe our education system needs heavy adaptation and redesign (https://doi.org/10.3389/feduc.2025.1504726) to help younger people’s cognitive systems adapt more naturally to this environment, moving away from linear, language-dominated intelligence toward forms that are more visual, spatial, and externally coordinated (https://www.emerald.com/oth/article-abstract/9/6/1/318200/Digital-Natives-Digital-Immigrants-Part-2-Do-They). If so, we may be witnessing a divergence: LLMs doubling down on linear linguistic reasoning, while human cognition gradually moves elsewhere. If that divergence holds, the long-term question is no longer whether LLMs “think like humans,” but whether humans will continue to think in the way language-centric AI systems are optimized to emulate.

The following content is generated by LLMs and may contain inaccuracies.

Context

This idea sits at the intersection of cognitive science, AI alignment, and generational media studies. It challenges the dominant framing that positions LLMs as successful because they mirror human reasoning. Instead, it raises the possibility of a cognitive bifurcation: AI systems crystallizing around mid-20th-century models of linear, language-bound thought (Wittgenstein’s Tractatus) just as younger cohorts develop intelligence shaped by visual-spatial interfaces, distributed cognition, and algorithmic curation. This tension matters now because education systems, workplace norms, and AI design philosophies still assume a stable, language-first model of competence—one that may be eroding.

Key Insights

  1. Offloading vs. Atrophy: The cognitive offloading literature distinguishes between functional offloading (tools extend capacity) and structural offloading (tools replace internal processes). Your short-form video research documents fragmented attention as a symptom of structural offloading, where algorithmic feeds manage sequencing and LLMs handle coherence. This aligns with findings that GPS reliance degrades hippocampal spatial memory (Javadi et al., Nature Comms, 2017)—not just convenience, but neuroplastic adaptation. The critical thinking concern you cite may reflect not deficiency but incommensurability: Gen Z’s compositional, multimodal problem-solving doesn’t map cleanly onto linear essay-based assessment.

  2. Divergence, Not Convergence: Prensky’s “Digital Natives” framework is dated but prescient here. Modern interfaces—TikTok, Figma, spatial canvases—privilege configurational over sequential reasoning. If cognition co-evolves with its media (McLuhan, Understanding Media), then LLMs optimizing for linguistic coherence may be solving yesterday’s problem. This echoes concerns in HCI about mode confusion when tools embody outdated mental models.

Open Questions

  • If younger users develop visual-spatial reasoning that LLMs cannot replicate, will human-AI collaboration require new interface paradigms—perhaps spatial or diagrammatic—that translate between modalities rather than defaulting to text?
  • Could educational systems paradoxically widen the cognitive gap by forcing Gen Z into language-centric evaluation schemes, making them less competitive in contexts where LLMs excel, while also failing to validate their native strengths?

从Sapir-Whorf假说的角度来看，可以论证LLM之所以表现出色，是因为它们模拟了语言的线性结构，进而模拟了推理本身的结构。这与维特根斯坦式的观点相吻合，即思想从根本上受语言束缚，或者至少只有通过语言才能被理解。

在很长一段时间里，我几乎完全相信了这个框架。

当我开始更仔细地观察完全沉浸在现代数字环境中成长的Z代时，这种信心开始动摇。几个模式一致地出现：1) 对线性、以语言为中心的推理的依赖减少；2) 对视觉表现形式的依赖增强；3) 交流模式更具组合性和空间性，而非顺序性。

这感觉像是认知结构的根本转变，思维似乎不再那么受线性语言叙述的束缚，而是更多地由外部系统支撑，这些系统代表用户管理顺序、记忆和连贯性。换句话说，现代软件越来越多地承担了维持线性结构的负担。

众所周知，LLM及相关技术已经开始卸载关键的认知过程，研究（例如https://arxiv.org/abs/2506.08872）批评年轻一代未能发展出我们这代人认为至关重要的某些批判性思维技能。我认为这种卸载使得在任务、观点和模式之间进行快速上下文切换成为可能。这个效果与我们之前关于短视频消费的研究（https://arxiv.org/abs/2302.03714）的发现紧密一致，其中碎片化的注意力模式重塑了意图如何形成、维持和放弃。对于成年人，尤其是那些受过长期、语言密集型问题解决训练的人来说，这种动态可能导致一个看似矛盾的结果：虽然LLM大幅提高了效率，但也使保持稳定意图变得更加困难，增加了认知过载的风险，而不是减少了它。

不太清楚但更有趣的是，相同的过载是否也适用于年轻一代。我倾向于认为我们的教育系统需要大幅调整和重新设计（https://doi.org/10.3389/feduc.2025.1504726），以帮助年轻人的认知系统更自然地适应这种环境，从线性、语言主导的智能向更具视觉性、空间性和外部协调性的形式转变（https://www.emerald.com/oth/article-abstract/9/6/1/318200/Digital-Natives-Digital-Immigrants-Part-2-Do-They）。如果是这样，我们可能正在见证一种分化：LLM加倍投入于线性语言推理，而人类认知则逐渐转向他处。如果这种分化成立，那么长期问题就不再是LLM是否"像人类一样思考"，而是人类是否会继续以语言中心的AI系统被优化为仿效的方式思考。

以下内容由 LLM 生成，可能包含不准确之处。

背景

这一观点处于认知科学、AI对齐和代际媒体研究的交叉点。它挑战了将大语言模型定位为成功因为它们镜像人类推理的主流框架。相反，它提出了一种认知分化的可能性：AI系统围绕20世纪中期的线性、以语言为中心的思维模型（维特根斯坦的《逻辑哲学论》）结晶化，而年轻群体则发展出由视觉-空间界面、分布式认知和算法策展塑造的智能。这种紧张关系现在很重要，因为教育系统、工作规范和AI设计哲学仍然假设一种稳定的、以语言为先的能力模型——而这种模型可能正在被侵蚀。

关键洞见

  1. 卸载vs.衰退：认知卸载文献区分了功能性卸载（工具扩展能力）和结构性卸载（工具替代内部流程）。你的短视频研究将碎片化注意力记录为结构性卸载的症状，其中算法信流管理序列，而大语言模型处理连贯性。这与GPS依赖降低海马体空间记忆的发现一致（Javadi等，Nature Communications, 2017）——不仅是便利，还有神经可塑性适应。你引用的批判性思维关切可能反映的不是缺陷，而是不可通约性：Gen Z的组合型、多模态问题解决不能清晰地对应到线性论文型评估。

  2. 分化，而非趋同：Prensky的"数字原住民“框架已过时但具有预见性。现代界面——TikTok、Figma、空间画布——优先考虑配置型而非序列型推理。如果认知与其媒介共同进化（麦克卢汉，《理解媒介》），那么为语言连贯性优化的大语言模型可能在解决昨天的问题。这呼应了人机交互中的模式混淆关切——当工具体现过时的心智模型时。

开放问题

  • 如果年轻用户发展出大语言模型无法复制的视觉-空间推理，人机协作是否需要新的界面范式——也许是空间型或图解型——来翻译不同的模式，而不是默认使用文本？
  • 教育系统是否会矛盾地加剧认知差距，通过强制Gen Z进入以语言为中心的评估体系，使他们在大语言模型擅长的环境中竞争力下降，同时也无法验证他们的本土优势？

Testing the ideas API测试想法 API

Testing the ideas API

The following content is generated by LLMs and may contain inaccuracies.

Testing the Ideas API

Context

This appears to be a technical exploration of an API designed for idea management or knowledge work systems. Such APIs typically sit at the intersection of personal knowledge management (PKM), developer tools, and AI-augmented workflows. The testing phase is critical—it reveals not just whether the API works, but whether its abstractions match how users actually think about and manipulate ideas. This matters now because we’re seeing a proliferation of “tools for thought” that promise seamless capture and retrieval, yet often fail at the crucial middleware layer where ideas need to be programmatically accessible, transformable, and connectable.

Key Insights

API Design as Epistemology: The structure of an ideas API encodes assumptions about what ideas are—atomic units? Networks of relations? Versioned entities? Research on API usability (Myers & Stylos, 2016, “Improving API Usability”) shows that developer mental models often clash with API design. For idea systems specifically, the challenge is even steeper: the API must support both hierarchical organization (folders, tags) and emergent networked thinking (bidirectional links, semantic clustering). Notion’s API launch revealed this tension—its block-based model works for structured data but struggles with fluid ideation.

Testing Beyond Functionality: Traditional API testing focuses on endpoints, response codes, and data validation. But idea APIs require evaluating cognitive friction. Does retrieving related ideas require too many calls? Can you express complex queries (e.g., “ideas tagged ‘AI ethics’ connected to papers from 2023”) elegantly? Tiago Forte’s work on “Building a Second Brain” suggests users need both frictionless capture and flexible retrieval—your tests should measure both.

Observability for Thought: Unlike transactional APIs, idea systems benefit from rich metadata about usage patterns. When testing, consider: Are you capturing creation timestamps, modification history, connection strength? These enable future features like “ideas you abandoned” or “concepts gaining momentum.” The difference between a CRUD API and a thoughtful ideas API lies here.

Open Questions

  • How does the API handle the “idea lifecycle”—from fleeting note to fully developed concept? Can it support progressive elaboration without forcing premature structure?

  • What does versioning mean for ideas? Should the API treat idea evolution as git-style commits, or as continuous transformation where history fades?

测试想法 API

以下内容由 LLM 生成，可能包含不准确之处。

测试想法 API

背景

这是一次对面向想法管理或知识工作系统的 API 的技术探索。此类 API 通常位于个人知识管理（PKM）、开发者工具和 AI 增强工作流的交汇点。测试阶段至关重要——它不仅揭示 API 是否可用，还揭示其抽象是否与用户实际思考和操作想法的方式相匹配。这在当下尤为重要，因为我们正在看到大量"思维工具"的涌现，它们承诺无缝的捕获和检索，却常常在关键的中间件层——想法需要以编程方式被访问、转换和连接的地方——出现问题。

关键洞察

API 设计即认识论：想法 API 的结构编码了关于想法是什么的假设——原子单元？关系网络？版本化实体？API 可用性研究（Myers & Stylos, 2016, “Improving API Usability”）表明，开发者的心智模型常常与 API 设计相冲突。对于想法系统来说，挑战更为严峻：API 必须同时支持层级组织（文件夹、标签）和涌现的网络化思维（双向链接、语义聚类）。Notion 的 API 发布揭示了这一矛盾——其基于块的模型适用于结构化数据，但在流畅的构思方面力不从心。

超越功能性的测试：传统 API 测试关注端点、响应码和数据验证。但想法 API 需要评估认知摩擦。检索相关想法是否需要太多调用？是否能优雅地表达复杂查询（例如，“标记为’AI 伦理’并与 2023 年论文相关联的想法”）？Tiago Forte 关于"打造第二大脑"的工作表明，用户需要无摩擦的捕获和灵活的检索——你的测试应该衡量这两者。

思维的可观测性：与事务性 API 不同，想法系统受益于关于使用模式的丰富元数据。在测试时，考虑：是否在捕获创建时间戳、修改历史、连接强度？这些为未来的功能提供支持，如"你放弃的想法"或"势头渐起的概念"。CRUD API 和一个深思熟虑的想法 API 的区别就在于此。

开放问题

  • API 如何处理"想法生命周期"——从转瞬即逝的笔记到完全发展的概念？是否能支持渐进式细化而不强制提前结构化？

  • 版本控制对想法意味着什么？API 应该将想法的演变视为 git 式的提交，还是视为历史逐渐淡出的持续转变？

PBO Preferential Bayesian Optimization MethodsPBO 偏好贝叶斯优化方法

Looking into different ideas about PBO preferential bayesian optimization

The following content is generated by LLMs and may contain inaccuracies.

Deep Dive: Preferential Bayesian Optimization (PBO)

Context

Preferential Bayesian Optimization operates at the intersection of human-in-the-loop machine learning and decision theory, addressing a fundamental challenge: optimizing objectives that are difficult to quantify numerically but easy to compare relatively. Traditional BO assumes access to explicit function evaluations f(x), but many real-world scenarios—from interface design to material aesthetics to policy selection—only provide comparative feedback (“A is better than B”). PBO matters now because alignment problems in AI increasingly require capturing nuanced human preferences that resist scalar quantification, while foundation models create new opportunities for encoding these preferences at scale.

Key Insights

Dueling bandits meets Gaussian processes: PBO extends the dueling bandits framework by modeling latent utility functions with GPs, enabling efficient exploration in continuous spaces. The seminal work by Chu & Ghahramani (2005) and later González et al. (2017) showed that pairwise comparisons, when modeled through probit or logistic likelihoods, can recover underlying preference landscapes with sample efficiency approaching standard BO. The key technical challenge lies in inference scalability—computing acquisition functions over comparison spaces grows quadratically.

Acquisition function adaptations: While standard BO uses Expected Improvement or UCB, PBO requires specialized criteria. Expected Information Gain (EIG) about the optimum location, introduced by Sadigh et al. (2017) for active preference learning, proves particularly effective. Recent work on Preferential Thompson Sampling (Lin et al., 2022) demonstrates that posterior sampling can match or exceed EIG while remaining computationally tractable through Laplace approximations.

Connection to RLHF: Modern RLHF pipelines (Christiano et al., 2017; Ouyang et al., 2022) are essentially high-dimensional PBO problems where LLM outputs are optimized via human preference comparisons. The Bradley-Terry reward model used in RLHF is a direct descendant of PBO’s pairwise comparison models, though RLHF typically operates in representation spaces rather than direct input spaces.

Open Questions

  • How can we efficiently handle intransitive or inconsistent preferences that violate the utility function assumption, particularly when human feedback reflects contextual or time-varying values?

  • Can meta-learning over preference functions accelerate PBO in new domains by transferring knowledge about how humans structure their comparative judgments across related tasks?

研究关于 PBO 偏好贝叶斯优化的不同想法

以下内容由 LLM 生成，可能包含不准确之处。

深入探讨：偏好贝叶斯优化（PBO）

背景

偏好贝叶斯优化运作在人机协同机器学习和决策理论的交汇点，解决一个基本挑战：优化那些难以用数值量化但易于进行相对比较的目标。传统的 BO 假设可以获得显式的函数评估 f(x)，但许多现实场景——从界面设计到材料美学到策略选择——只提供比较性反馈（“A 比 B 好”）。PBO 在当下尤为重要，因为 AI 中的对齐问题越来越需要捕获难以标量化的细微人类偏好，而基础模型为大规模编码这些偏好创造了新的机会。

关键洞察

对决赌博机遇上高斯过程：PBO 通过使用 GP 对潜在效用函数建模，扩展了对决赌博机框架，实现了在连续空间中的高效探索。Chu & Ghahramani（2005）以及后来 González 等人（2017）的开创性工作表明，通过 probit 或 logistic 似然建模的成对比较，可以以接近标准 BO 的样本效率恢复底层偏好景观。关键技术挑战在于推断可扩展性——在比较空间上计算采集函数的增长是二次方的。

采集函数的适配：标准 BO 使用期望改进或 UCB，而 PBO 需要专门的准则。Sadigh 等人（2017）为主动偏好学习引入的关于最优位置的期望信息增益（EIG）被证明特别有效。Lin 等人（2022）关于偏好 Thompson 采样的最新工作表明，后验采样可以通过 Laplace 近似在保持计算可行性的同时匹配或超越 EIG。

与 RLHF 的联系：现代 RLHF 流水线（Christiano et al., 2017; Ouyang et al., 2022）本质上是高维 PBO 问题，其中 LLM 输出通过人类偏好比较进行优化。RLHF 中使用的 Bradley-Terry 奖励模型是 PBO 成对比较模型的直接后代，尽管 RLHF 通常在表示空间而非直接输入空间中运作。

开放问题

  • 如何高效处理违反效用函数假设的不可传递或不一致的偏好，特别是当人类反馈反映的是上下文相关或随时间变化的价值观时？

  • 对偏好函数的元学习能否通过迁移人类在相关任务中构建比较判断的知识来加速新领域中的 PBO？

Big Changes in Go 1.17Go 1.17 的重大变化

Runtime changes:

Toolchain changes:

运行时变化：

工具链变化：

A New GC Pacer全新的 GC 调步器

Today, the Go team released a brand new GC pacer design. Let’s briefly discuss what problems existed in the previous design and what the new design aims to solve.

The current Go runtime GC is a concurrent mark-sweep collector, which involves two core problems that need to be solved: 1) when to start GC and how many workers to launch for collection, to prevent the collector from using too many computing resources and affecting efficient execution of user code; 2) how to prevent the garbage collection speed from being slower than the memory allocation speed.

To address these problems, as early as Go 1.5, the Go team treated this as an optimization problem of minimizing heap growth rate and CPU usage, which led to two key components: 1) the pacer: predicting GC trigger timing based on heap growth speed; 2) mark assist: pausing user code that allocates too fast, redirecting goroutines that are allocating memory to perform garbage marking work, in order to smoothly complete the current GC cycle.

However, when making pacing decisions, this GC contains a hidden assumption: the allocation rate is always a constant (1+GOGC/100). Unfortunately, due to the existence of mark assist and discrepancies between implementation and the theoretical model, this assumption is actually incorrect. This leads to several hard-to-solve problems: 1) when the allocation rate violates the constant assumption, the predicted start time is too late, requiring excessive CPU consumption — while GOGC can be dynamically adjusted, it remains a hyperparameter requiring extensive domain experience to tune manually; 2) since the optimization targets heap growth without heap memory size limits, either setting GOGC too large or encountering peak allocations causes rapid heap growth leading to OOM; 3) newly allocated memory within the current GC cycle is left to the next GC cycle for collection, and mark assist’s allocation throttling causes latency pauses (STW); 4) …

So what has the new pacer redesigned to solve these problems?

As mentioned above, the main source of various problems is the incorrect assumption that the allocation rate is a constant. So naturally, it’s easy to think of using the mark assist component to dynamically calculate the allocation rate during modeling, thereby achieving the goal of dynamically adjusting the heap target. Unfortunately, the original design only tracked allocations on the heap, without considering the stack or global variables. To make the problem more comprehensive, the new design introduces an “assist ratio” — the ratio of allocations produced but not collected in the current GC cycle (A) to the amount of scanning completed in the current GC cycle (B), i.e., A/B. This metric more intuitively reflects the actual difficulty of GC work: if the user allocation rate is too high, A increases, the assist ratio rises, and more assistance is needed from the mark assist; if the allocation rate is moderate, the assist ratio decreases. With the introduction of the assist ratio, the pacer can dynamically adjust the assist work, thereby resolving the pauses caused by the assist.

Let’s look at a practical scenario: when a sudden burst of peak requests arrives, the number of goroutines increases dramatically, generating a large number of stacks and allocation tasks. Here are the simulation results: Figure 1 shows the pacer before adjustment, Figure 2 shows the pacer after adjustment. As shown in the lower-left of Figure 1, the heap target workload is consistently underestimated, causing the heap to always overshoot; while the new pacer can quickly converge to zero and complete the heap target prediction. The upper-right of Figure 1 shows that the actual GC CPU usage is always lower than the target usage, failing to meet the expected metrics; while the newly designed pacer can quickly converge to the target CPU usage.

Of course, due to space constraints, the above is only a very brief introduction to the new pacer design. If you are interested in this topic, you can refer to the following links. There will be opportunities to share more detailed analysis in the future.

  1. Existing problems with the GC pacer: https://golang.org/issue/42430
  2. Design document for the new pacer: https://go.googlesource.com/proposal/+/a216b56e743c5b6b300b3ef1673ee62684b5b63b/design/44167-gc-pacer-redesign.md
  3. Related proposal: https://golang.org/issue/44167
  4. Simulator for the new GC pacer model: https://github.com/mknyszek/pacer-model

今天，Go 团队发布了一个全新的 GC 的调步器（Pacer）设计。这次就来简单聊一聊这个以前的设计有什么问题，新的设计又旨在解决什么问题。

目前 Go 运行时的 GC 是一个并发标记清理的回收器，这涉及两个需要解决的核心问题：1）何时启动 GC 并启动多少数量的 worker 进行搜集从而防止回收器使用过多的计算资源影响用户代码的高效执行；2）如何防止收集垃圾的速度慢于内存分配的速度。

为解决这些问题，早在 1.5，Go 团队将这个问题视作一个最小化堆的增长速率和 CPU 的使用率的优化问题，从而促成了两个关键组件：1）调步器：根据堆的增长速度来预测 GC 的触发时机；2）标记助理 （Mark Assist）：暂停分配速度过快的用户代码，将正在分配内存的用户代码转去执行垃圾标记的工作，以便顺利完成当前的 GC 周期。

然而这样的 GC 在实施调步决策时，包含一个隐藏的假设：分配速率总是一个常数（1+GOGC/100），可惜由于标记助理的存在、实现与理论模型的差异，导致这个假设其实并不正确。进而带来的很难解决的问题：1）当分配速率违反常数假设时，预测的启动时间太晚反而需要消耗过多的 CPU，虽然可以动态的调整 GOGC，但这仍然是一个超参数，人工优化需要大量的领域经验，很难直观的使用这个变量对 GC 进行优化；2）由于优化问题是以堆的增长为目标，由于没有堆内存大小的使用限制，无论是设置过大的 GOGC 或者出现峰值分配时都会导致堆的迅速增长从而 OOM；3）在当前 GC 周期内新分配的内存将留到下一个 GC 周期进行回收，标记助理暂缓分配带来的延迟停顿 STW；4）… 那么新的调步器为解决这些问题做了什么重新设计呢？

正如前面所说，产生各类问题的主要来源是对分配速率为常数这一错误的假设，那么自然也就很容易想到在建模的过程：利用标记助理这一组件来动态的计算分配的速率，从而达到动态调整堆目标的目的。可惜的是原来的设计中标记助理仅统计了堆上的分配情况，而对栈或全局变量没有加以考虑。为了让问题考虑得更加全面，新设计中引入了一个「辅助率」，表示当前 GC 周期新产生但没有回收的分配量（A）与当前 GC 周期完成的扫描量（B）之比，A/B。这一指标更加直观的反应了 GC 的实际工作难度：如果用户分配速率过高，那么 A 将增大，进而辅助率增高，需要助理提供更多的辅助；如果分配速率适中，辅助率下降。根据辅助率的引入，调步器便可动态的的调整助理的辅助工作，进而解决辅助时带来的停顿。

我们来看一个实际的场景：当突然出现大量峰值请求时，goroutine数量大量增加，从而产生大量栈和分配任务，极其模拟的结果：图 1 是调整前的调步器，图 2 是调整后的调步器。可见图1左下角显示，总是错误的低估了堆目标工作量，导致堆总是在过冲；而新的调步器能很快的收敛到零，完成堆目标的预测；图 1 右上角则表明实际的 GC CPU 使用率总是比目标使用率低，从而为能完成预期指标；而新设计的调步器则能很快收敛到目标的 CPU 使用率。

当然，限于篇幅上面只是对新的调步器设计做了一个非常简略的介绍。如果对这个内容感兴趣，可以查阅后面的这些链接，之后有机会再对此设计做进一步详细的分享。

  1. GC 调步器现存的问题：https://golang.org/issue/42430
  2. 新调步器的设计文档：https://go.googlesource.com/proposal/+/a216b56e743c5b6b300b3ef1673ee62684b5b63b/design/44167-gc-pacer-redesign.md
  3. 相关的提案：https://golang.org/issue/44167
  4. GC 新调步器模型的模拟器：https://github.com/mknyszek/pacer-model

Go 1.16 Big ChangesGo 1.16 的重大变化

Go 1.16 has released many very interesting changes. Here is a brief summary:

russ cos: deprecated.

  1. Support for darwin/arm64
    1. Issues encountered supporting darwin/arm64
      • Apple’s bug: related to signal preemption
    2. Apple Silicon M1 performance
    3. Compiler bootstrapping process

  • Installing Go: https://gist.github.com/Dids/dbe6356377e2a0b0dc8eacb0101dc3a7

  • https://github.com/golang/go/issues/42684

    • Kernel Panic Episode 62: Your Computer Isn’t Yours, Code Signing, OCSP Server
    • Ken Thompson Turing Award lecture: Reflections on Trusting Trust
      • TODO
    • Apple’s long-standing code signing issues; I encountered similar problems doing Electron in the early days, and these issues still exist today

  • Asynchronous preemption random crashes, a Rosetta bug: https://github.com/golang/go/issues/42700

  • Bootstrapping, installation confusion: https://github.com/golang/go/issues/38485#issuecomment-735360572

    • Go’s bootstrapping consists of three steps
        1. 1.4 C version TODO
        1. tool chain 1
        1. tool chain 2
        1. tool chain 3

  • Run x86 programs under Rosetta: arch --x86_64

  • M1 compatibility status in dotfiles: https://github.com/changkun/dotfiles/issues/2

  • Got it in early December, have been using it for almost two months now — very smooth, battery life is incredible

  • My essential third-party software list:

    • homebrew (compatibility is not great, but fortunately most dependent software is written in Go, and Go’s support is very complete)
      • Breaks compatibility casually, removes software distribution — there was a tool called rmtrash that I had been using since around 2014, but it was removed from distribution last year, so I wrote a fully compatible tool changkun.de/s/rmtrash, but it wasn’t merged; they said it needed to be approved by the original author to bypass popularity restrictions, but the original author is unreachable
    • vscode (have been using Insider long-term)
    • macvim
    • tmux
    • oh-my-zsh
    • Blender (Cycles ray tracing doesn’t support GPU, but editing meshes with less than a million vertices is fine)
    • iTerm: supports M1
    • Chrome: supports M1
    • MacTex: supports M1
    • Docker: released support a week before Christmas, works perfectly, no issues so far

  1. Go Modules changes

    1. Collecting feedback
    2. Complex dependency management — what’s the most complex project dependency you’ve managed in practice, how many modules, and what do you write for each dependency upgrade? What did you use before Go modules?
      1. My experience: Go vendor, 1.10 dep, 1.11 go modules
      2. GOPATH project management — although GOPATH has been removed, I still follow the GOPATH convention
    3. Minimum version selection
      1. Semantic Versioning: major.minor.patch
      2. The classic diamond dependency problem: A depends on B and C, B and C each depend on different versions of D that are incompatible, so no specific version of D can be selected — semantic import versioning eliminates this by adding the major version number requirement at the end of the import path /v2
      3. dep doesn’t allow diamond dependencies, upgrades are very difficult
      4. Build reproducibility — without a lock file, >= dependencies change over time
      5. Select the minimum usable version — builds don’t change over time
      6. https://www.youtube.com/watch?v=F8nrpe0XWRg&ab_channel=SingaporeGophers
      7. Misunderstood working methods
      8. GOPATH
      9. vendor
      10. Three key points
      11. Compatibility
      12. Reproducibility
      13. Cooperation (often overlooked by many)
    4. Go Modules enabled by default, go build must include go.mod file, otherwise compilation fails
    5. build/test will not upgrade modules
    6. Default -mod=vendor

  2. File system interface

    1. Why is the fs.FS abstraction important

      1. Unix file system abstract always disk blocks
      2. Network file systems (Upspin) abstract away machines
      3. REST abstracts nearly anything
      4. cp doesn’t care whether it’s moving file blocks, or even where the file is — it could be different disks or different machines
      5. Defines the “generics” for any file type tools

    2. What major changes it caused

      1. io/ioutil
        1. Russ Cox’s explanation of deprecated in Go (https://twitter.com/_rsc/status/1351676094664110082)
        2. https://www.srcbeat.com/2021/01/golang-ioutil-deprecated/
      2. Other fs abstractions
      3. Rob Pike’s 2016/2017 Gopherfest, Upspin, Changkun’s Midgard
        1. https://www.youtube.com/watch?v=ENLWEfi0Tkg&ab_channel=TheGoProgrammingLanguage
        2. FUSE: filesystem in userspace
        3. https://changkun.de/s/midgard
        4. Every user has a private root, no global root, r@golang.org/some/stuff, user names look like email addresses
        5. Access control defined by plain text files read: r@golang.org, ann@example.com
      4. Currently a very simple implementation, just a read-only file system
      5. ReadDir and DirEntry
        1. https://benhoyt.com/writings/go-readdir/
      6. Extensible directions: memoryFS, support for writing back to disk, hashFS for CDN support
      7. Remaining issues… e.g. 44166
      1
2
3
4
5
6
7

      import _ "embed"
//go:embed a.txt
var s string

import "embed"
type embed.String string
var s embed.String

      1

  3. File embedding //go:embed

    1. Basic functionality of the new feature
    2. Some possible applications
    3. Some features discussed during the feature freeze cycle
    4. https://blog.carlmjohnson.net/post/2021/how-to-use-go-embed/

  4. Runtime memory management

    1. Return to MADV_DONTNEED
    1. New monitoring infrastructure runtime/metrics
    • Previous monitoring functions: runtime.ReadMemStats, debug.GCStats
    • runtime/metrics:
      • metrics.All()
      • Issue 37112
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

package main

import (
	"fmt"
	"runtime/metrics"
)

func main() {
	// Get descriptions for all supported metrics.
	descs := metrics.All()

	// Create a sample for each metric.
	samples := make([]metrics.Sample, len(descs))
	for i := range samples {
		samples[i].Name = descs[i].Name
	}

	// Sample the metrics. Re-use the samples slice if you can!
	metrics.Read(samples)

	// Iterate over all results.
	for _, sample := range samples {
		// Pull out the name and value.
		name, value := sample.Name, sample.Value

		// Handle each sample.
		switch value.Kind() {
		case metrics.KindUint64:
			fmt.Printf("%s: %d\n", name, value.Uint64())
		case metrics.KindFloat64:
			fmt.Printf("%s: %f\n", name, value.Float64())
		case metrics.KindFloat64Histogram:
			// The histogram may be quite large, so let's just pull out
			// a crude estimate for the median for the sake of this example.
			fmt.Printf("%s: %f\n", name, medianBucket(value.Float64Histogram()))
		case metrics.KindBad:
			// This should never happen because all metrics are supported
			// by construction.
			panic("bug in runtime/metrics package!")
		default:
			// This may happen as new metrics get added.
			//
			// The safest thing to do here is to simply log it somewhere
			// as something to look into, but ignore it for now.
			// In the worst case, you might temporarily miss out on a new metric.
			fmt.Printf("%s: unexpected metric Kind: %v\n", name, value.Kind())
		}
	}
}

func medianBucket(h *metrics.Float64Histogram) float64 {
	total := uint64(0)
	for _, count := range h.Counts {
		total += count
	}
	thresh := total / 2
	total = 0
	for i, count := range h.Counts {
		total += count
		if total > thresh {
			return h.Buckets[i]
		}
	}
	panic("should not happen")
}
  1. Other noteworthy features
    1. os/signal.NotifyContext
    2. Memory model fixes
    3. Linker optimizations

Go 1.16 发布了非常多非常有趣的变，尝试做一个简单的总结：

russ cos: deprecated.

  1. 支持 darwin/arm64
    1. 支持 darwin/arm64 上遇到的问题
      • 苹果的bug: 与信号抢占有关
    2. Apple Silicon M1 性能
      • 但是在加密上性能很差
      • 发版周期：https://github.com/golang/go/wiki/Go-Release-Cycle
    3. 编译器自举过程

  • 安装 Go：https://gist.github.com/Dids/dbe6356377e2a0b0dc8eacb0101dc3a7

  • https://github.com/golang/go/issues/42684

    • 内核恐慌的第 62 期：你的电脑不是你的，代码签名，OCSP Server
    • ken thompson 图灵奖演讲：reflections on trusting trust
      • TODO
    • 苹果代码签名的老问题，早年做 electron 也是这类问题，现在这样的问题还是存在

  • 异步抢占随机崩溃，是 Rosetta 的 Bug：https://github.com/golang/go/issues/42700

  • 自居，安装的困惑：https://github.com/golang/go/issues/38485#issuecomment-735360572

    • Go 语言的自举分为三个步骤
        1. 1.4 C version TODO
        1. tool chain 1
        1. tool chain 2
        1. tool chain 3

  • 在 Rosetta 下运行 x86 程序：arch --x86_64

  • dotfiles 中关于 M1 的兼容性情况：https://github.com/changkun/dotfiles/issues/2

  • 十二月初入手 如今已经使用快两个月了 非常流畅 续航逆天

  • 我的必备第三方软件列表：

    • homebrew (支持性不好，好在现在大部分依赖的软件是用 Go 写的，而且 Go 的支持非常完善)
      • 不考虑兼容性 随意破坏兼容性移除软件分发，有一个 rmtrash 的工具，我从2014年左右就开始使用，但是去年被从软件分发中移除了，所以自己写了一个全兼容的工具changkun.de/s/rmtrash，但没有被合并，他们说了要被原软件作者任何才能不受受欢迎程度的限制，但实际上软件作者已经联系不到了
    • vscode（已在长期使用 Insider）
    • macvim
    • tmux
    • oh-my-zsh
    • Blender（Cycles 光追渲染不支持 GPU，但编辑顶点小于百万级别的网格是没有问题的）
    • iTerm：支持 M1
    • Chrome：支持 M1
    • MacTex：支持 M1
    • Docker：圣诞节前一周发布支持，很完美，至今没有遇到问题

  1. Go Modules 的变更

    1. 收集反馈
    2. 复杂依赖管理，你实践中管理过最复杂的项目依赖多少模块，每次依赖升级都有写什么？在没有 Go modules 之前你用的是什么？
      1. 我的经历：Go vendor, 1.10 dep, 1.11 go modules,
      2. GOPATH 的项目管理，现在虽然移除了 gopath，但我还是沿用了 gopath 的习惯
    3. 最小版本选择
      1. Semantic Versioning: major.minor.patch
      2. 经典的钻石依赖问题：A依赖B和C，BC分别依赖 D 的不同版本，而这两个版本的 D 不兼容，所以无法在依赖中选取一个特定的D版本，semantic import versioning 消除了这种依赖，在import path的最后添加了主版本号的要求/v2
      3. dep 不允许钻石依赖，升级非常难
      4. 构建的可重复性，没有lock文件，>=的依赖会随着时间的变化而变化
      5. 选择最小的可以依赖的版本，构建不会随时间的变化而变化
      6. https://www.youtube.com/watch?v=F8nrpe0XWRg&ab_channel=SingaporeGophers
      7. 不被理解的工作方式
      8. GOPATH
      9. vendor
      10. 三大要点
      11. 兼容性
      12. 可重复性
      13. 合作(通常被很多人忽略)
    4. 默认启用 Go Moduels, go build 必须包含 go.mod 文件，否则编译失败
    5. build/test 不会升级 modules
    6. 默认 -mod=vendor

  2. 文件系统接口

    1. fs.FS 抽象的重要性在哪里

      1. unix file system abstract always disk blocks
      2. network file systems (upspin) abstract away machines
      3. rest abstract nearly anything
      4. cp 不关心是否移动文件的区块，甚至不关心文件在哪个位置，可能是不同的磁盘也可能是不同的机器
      5. 定义任何文件类型工具的「泛型」

    2. 导致了哪些主要变化

      1. io/ioutil
        1. Russ cox 对 deprecated 在 go 中的解释（https://twitter.com/_rsc/status/1351676094664110082）
        2. https://www.srcbeat.com/2021/01/golang-ioutil-deprecated/
      2. 其他 fs 的抽象
      3. Rob Pike 的 2016/2017 Gopherfest， Upspin、Changkun 的 Midgard
        1. https://www.youtube.com/watch?v=ENLWEfi0Tkg&ab_channel=TheGoProgrammingLanguage
        2. FUSE: filesystem in userspace
        3. https://changkun.de/s/midgard
        4. every user has a private root, no global root, r@golang.org/some/stuff, user names look like email address
        5. access control defined by plain text files read: r@golang.org, ann@example.com
      4. 目前的非常简单的实现，只是一个只读文件系统
      5. ReadDir and DirEntry
        1. https://benhoyt.com/writings/go-readdir/
      6. 可扩展的方向：memoryFS，支持回写到磁盘、hashFS 为 CDN 提供支持
      7. 还存在的问题。。例如 44166
      1
2
3
4
5
6
7

      import _ "embed"
//go:embed a.txt
var s string

import "embed"
type embed.String string
var s embed.String

      1

  3. 文件嵌入 //go:embed

    1. 新特性的基本功能
    2. 一些可能的应用
    3. 一些在feature freeze cycle 中才讨论出来的feature
    4. https://blog.carlmjohnson.net/post/2021/how-to-use-go-embed/

  4. 运行时内存管理

    1. 回归 MADV_DONTNEED
    1. 新的监控基础设施 runtime/metrics
    • 以前的监控函数：runtime.ReadMemStats, debug.GCStats,
    • runtime/metrics:
      • metrics.All()
      • Issue 37112
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

package main

import (
	"fmt"
	"runtime/metrics"
)

func main() {
	// Get descriptions for all supported metrics.
	descs := metrics.All()

	// Create a sample for each metric.
	samples := make([]metrics.Sample, len(descs))
	for i := range samples {
		samples[i].Name = descs[i].Name
	}

	// Sample the metrics. Re-use the samples slice if you can!
	metrics.Read(samples)

	// Iterate over all results.
	for _, sample := range samples {
		// Pull out the name and value.
		name, value := sample.Name, sample.Value

		// Handle each sample.
		switch value.Kind() {
		case metrics.KindUint64:
			fmt.Printf("%s: %d\n", name, value.Uint64())
		case metrics.KindFloat64:
			fmt.Printf("%s: %f\n", name, value.Float64())
		case metrics.KindFloat64Histogram:
			// The histogram may be quite large, so let's just pull out
			// a crude estimate for the median for the sake of this example.
			fmt.Printf("%s: %f\n", name, medianBucket(value.Float64Histogram()))
		case metrics.KindBad:
			// This should never happen because all metrics are supported
			// by construction.
			panic("bug in runtime/metrics package!")
		default:
			// This may happen as new metrics get added.
			//
			// The safest thing to do here is to simply log it somewhere
			// as something to look into, but ignore it for now.
			// In the worst case, you might temporarily miss out on a new metric.
			fmt.Printf("%s: unexpected metric Kind: %v\n", name, value.Kind())
		}
	}
}

func medianBucket(h *metrics.Float64Histogram) float64 {
	total := uint64(0)
	for _, count := range h.Counts {
		total += count
	}
	thresh := total / 2
	total = 0
	for i, count := range h.Counts {
		total += count
		if total > thresh {
			return h.Buckets[i]
		}
	}
	panic("should not happen")
}
  1. 其他值得一提的特性
    1. os/signal.NotifyContext
    2. 内存模型修复
    3. 链接器优化

Daily Reading每日阅读

Read these articles:

读了这些文章：

  • 监控现代 Go 应用内存使用时的注意事项。https://www.bwplotka.dev/2019/golang-memory-monitoring/
  • 分布式系统。https://www.youtube.com/watch?v=UEAMfLPZZhE
  • Golang 新闻。https://www.golangnews.com/
  • SIGCHI 交互式计算系统工程研讨会。http://eics.acm.org/
  • runtime：在 Linux 上使用 MADV_FREE（如果可用）。https://go-review.googlesource.com/c/go/+/135395/
  • runtime：使页分配器可扩展。https://github.com/golang/go/issues/35112
  • runtime：添加 per-p mspan 缓存。https://go-review.googlesource.com/c/go/+/196642
  • 一种用于四边形和六面体网格的新平滑算法。https://link.springer.com/content/pdf/10.1007%2F11758525_32.pdf
  • OpenGL 文档。https://docs.gl
  • 论下棋。https://blog.gardeviance.org/2018/03/on-playing-chess.html
  • 内存模型：重新思考并行语言和硬件。https://cacm.acm.org/magazines/2010/8/96610-memory-models-a-case-for-rethinking-parallel-languages-and-hardware/fulltext
  • 工程师等级与能力框架。https://github.com/spring2go/engineer_competency_framework
  • 一个并发窗口系统。http://doc.cat-v.org/bell_labs/concurrent_window_system/concurrent_window_system.pdf

Creating A Window创建一个窗口

How to create a window using Go? macOS has Cocoa, Linux has X11, but accessing these APIs seems to require Cgo. Is it possible to avoid Cgo? Some existing GUI libraries and graphics engines:

GUI Toolkits:

2D/3D Graphics:

Here is a partial list:

https://github.com/avelino/awesome-go#gui

Most people use glfw and OpenGL. Here are some required libraries (Cgo bindings):

Some more low-level ones, e.g. X-related:

For example, if you need Metal on macOS:

Like the GUI tool ebiten mentioned above, on Windows it no longer needs Cgo. The approach seems to be packaging the window management DLLs directly into the binary and then using DLL dynamic linking calls.

Besides GLFW, there is the heavier SDL:

Relationships between some basic terms:

1
2
3
4
5
6
7

Unix:      The ancestor
BSD:       Unix-like, Berkeley distribution, one of the two traditional flavors of Unix
System V:  Developed by AT&T, one of the two traditional flavors of Unix
Linux:     A fresh flavor of Unix-like OS
POSIX:     A standard attempting to reduce implementation differences
Darwin:    Apple's open-source Unix-like kernel XNU
Mach-O:    Microkernel hybridized in Darwin, developed by CMU

By Eraserhead1, Infinity0, Sav_vas - Levenez Unix History Diagram, Information on the history of IBM's AIX on ibm.com, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=1801948

Relationships between some Wayland-related tools:

1
2
3
4
5
6

X Window System == X11: Display standard developed by MIT, the foundation for GNOME and KDE
Xorg:    The official implementation of X11
Wayland: An alternative communication protocol between display server and client, distinct from X11
EGL:     Wayland clients use EGL to directly manipulate the framebuffer
libDRM:  Kernel API exposed to userspace, underlying dependency of EGL/X11
DRM:     Kernel-level component for manipulating the framebuffer

By Shmuel Csaba Otto Traian, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=28029855

By Shmuel Csaba Otto Traian, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=31768083

By Shmuel Csaba Otto Traian, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=27858390

By Shmuel Csaba Otto Traian，CC BY-SA 3.0，https://commons.wikimedia.org/w/index.php?curid=27799196

So through DRM you can directly operate on the Frame Buffer under Linux, i.e., Direct Rendering Manager. Related libraries:

With this, you can do pure Go rendering directly on Linux, eliminating the dependency on C legacy.

如何使用 Go 创建一个窗口？macOS 有 Cocoa、Linux 有 X11，但访问这些 API 似乎都需要 引入 Cgo，可不可以不实用 Cgo？一些现有的 GUI 库或这图形引擎：

GUI 工具包:

2D/3D 图形相关:

这里有一小部分：

https://github.com/avelino/awesome-go#gui

大部分人的做法是使用 glfw 和 OpenGL，这是一些需要使用到的库（Cgo 绑定）：

这里面有一些相对底层一些的，比如 X 相关:

  • X 绑定：https://github.com/BurntSushi/xgb
  • X 窗口管理：https://github.com/BurntSushi/wingo

比如 macOS 上如果需要用到 Metal：

像是前面的 GUI 工具中的 ebiten，在 windows 上已经不需要 Cgo 了，做法似乎是将窗口管理相关的 DLL 直接打包进二进制，然后走 DLL 动态链接调用。

除了 GLFW 之外，还有相对重一些的 SDL：

一些基本名词之间的关系：

1
2
3
4
5
6
7

Unix:      上帝
BSD:       类 Unix，伯克利分发，两种传统风味的 Unix 之一
System V:  AT&T 开发，两种传统风味的 Unix 之一
Linux:     新鲜风味的类 Unix
POSIX:     尝试减少实现差异的标准
Darwin:    苹果的开源类 Unix 内核 XNU
Mach-O:    Darwin 混合的微内核，CMU 开发

By Eraserhead1, Infinity0, Sav_vas - Levenez Unix History Diagram, Information on the history of IBM's AIX on ibm.com，CC BY-SA 3.0，https://commons.wikimedia.org/w/index.php?curid=1801948

关于 Wayland 的一些工具之间的关系：

1
2
3
4
5
6

X Window System == X11: MIT 开发的显示标准，GNOME、KDE 依赖的基础
Xorg:    X11 的官方实现
Wayland: 另一种显示服务器和客户端之间的通信协议，区别于 X11
EGL:     Wayland 客户端使用 EGL 来直接操作 framebuffer
libDRM:  EGL/X11 底层依赖的内核对 userspace 开放的 API
DRM:     内核级操作 framebuffer 的组件

By Shmuel Csaba Otto Traian, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=28029855

By Shmuel Csaba Otto Traian, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=31768083

By Shmuel Csaba Otto Traian, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=27858390

By Shmuel Csaba Otto Traian，CC BY-SA 3.0，https://commons.wikimedia.org/w/index.php?curid=27799196

所以通过 DRM 可以在 Linux 下直接操作 Frame Buffer 上，也就是 Direct Rendering Manager。相关的库有：

有了这个就可以在 Linux 上直接做纯 Go 的绘制了，从而消除对 C 遗产的依赖。

New Idea新想法
© 2008 - 2026 Changkun Ou. All rights reserved.保留所有权利。 | PV/UV: /
0%