Read this article:

• What are the most important statistical ideas of the past 50 years? https://arxiv.org/pdf/2012.00174.pdf

Runtime changes:

• New GC Pacer: https://golang.org/issue/44167
• Scheduler performance: https://golang.org/issue/43997

Toolchain changes:

• Register-based calling convention: https://golang.org/issue/40724
• Fuzzing: golang.org/s/draft-fuzzing-design

Today, the Go team released a brand new GC pacer design. Let’s briefly discuss what problems existed in the previous design and what the new design aims to solve.

The current Go runtime GC is a concurrent mark-sweep collector, which involves two core problems that need to be solved: 1) when to start GC and how many workers to launch for collection, to prevent the collector from using too many computing resources and affecting efficient execution of user code; 2) how to prevent the garbage collection speed from being slower than the memory allocation speed.

To address these problems, as early as Go 1.5, the Go team treated this as an optimization problem of minimizing heap growth rate and CPU usage, which led to two key components: 1) the pacer: predicting GC trigger timing based on heap growth speed; 2) mark assist: pausing user code that allocates too fast, redirecting goroutines that are allocating memory to perform garbage marking work, in order to smoothly complete the current GC cycle.

However, when making pacing decisions, this GC contains a hidden assumption: the allocation rate is always a constant (1+GOGC/100). Unfortunately, due to the existence of mark assist and discrepancies between implementation and the theoretical model, this assumption is actually incorrect. This leads to several hard-to-solve problems: 1) when the allocation rate violates the constant assumption, the predicted start time is too late, requiring excessive CPU consumption — while GOGC can be dynamically adjusted, it remains a hyperparameter requiring extensive domain experience to tune manually; 2) since the optimization targets heap growth without heap memory size limits, either setting GOGC too large or encountering peak allocations causes rapid heap growth leading to OOM; 3) newly allocated memory within the current GC cycle is left to the next GC cycle for collection, and mark assist’s allocation throttling causes latency pauses (STW); 4) …

So what has the new pacer redesigned to solve these problems?

As mentioned above, the main source of various problems is the incorrect assumption that the allocation rate is a constant. So naturally, it’s easy to think of using the mark assist component to dynamically calculate the allocation rate during modeling, thereby achieving the goal of dynamically adjusting the heap target. Unfortunately, the original design only tracked allocations on the heap, without considering the stack or global variables. To make the problem more comprehensive, the new design introduces an “assist ratio” — the ratio of allocations produced but not collected in the current GC cycle (A) to the amount of scanning completed in the current GC cycle (B), i.e., A/B. This metric more intuitively reflects the actual difficulty of GC work: if the user allocation rate is too high, A increases, the assist ratio rises, and more assistance is needed from the mark assist; if the allocation rate is moderate, the assist ratio decreases. With the introduction of the assist ratio, the pacer can dynamically adjust the assist work, thereby resolving the pauses caused by the assist.

Let’s look at a practical scenario: when a sudden burst of peak requests arrives, the number of goroutines increases dramatically, generating a large number of stacks and allocation tasks. Here are the simulation results: Figure 1 shows the pacer before adjustment, Figure 2 shows the pacer after adjustment. As shown in the lower-left of Figure 1, the heap target workload is consistently underestimated, causing the heap to always overshoot; while the new pacer can quickly converge to zero and complete the heap target prediction. The upper-right of Figure 1 shows that the actual GC CPU usage is always lower than the target usage, failing to meet the expected metrics; while the newly designed pacer can quickly converge to the target CPU usage.

Of course, due to space constraints, the above is only a very brief introduction to the new pacer design. If you are interested in this topic, you can refer to the following links. There will be opportunities to share more detailed analysis in the future.

1. Existing problems with the GC pacer: https://golang.org/issue/42430
2. Design document for the new pacer: https://go.googlesource.com/proposal/+/a216b56e743c5b6b300b3ef1673ee62684b5b63b/design/44167-gc-pacer-redesign.md
3. Related proposal: https://golang.org/issue/44167
4. Simulator for the new GC pacer model: https://github.com/mknyszek/pacer-model

Go 1.16 has released many very interesting changes. Here is a brief summary:

russ cos: deprecated.

• https://twitter.com/_rsc/status/1351676094664110082
• https://go-review.googlesource.com/c/go/+/285378
• https://github.com/golang/go/issues/43724

1. Support for darwin/arm64
   1. Issues encountered supporting darwin/arm64
      • Apple’s bug: related to signal preemption
   2. Apple Silicon M1 performance
      • But crypto performance is poor
      • Release cycle: https://github.com/golang/go/wiki/Go-Release-Cycle
   3. Compiler bootstrapping process

• Installing Go: https://gist.github.com/Dids/dbe6356377e2a0b0dc8eacb0101dc3a7

• https://github.com/golang/go/issues/42684

  • Kernel Panic Episode 62: Your Computer Isn’t Yours, Code Signing, OCSP Server
  • Ken Thompson Turing Award lecture: Reflections on Trusting Trust
    • TODO
  • Apple’s long-standing code signing issues; I encountered similar problems doing Electron in the early days, and these issues still exist today

• Asynchronous preemption random crashes, a Rosetta bug: https://github.com/golang/go/issues/42700

• Bootstrapping, installation confusion: https://github.com/golang/go/issues/38485#issuecomment-735360572

  • Go’s bootstrapping consists of three steps
    • 0. 1.4 C version TODO
    • 1. tool chain 1
    • 2. tool chain 2
    • 3. tool chain 3

• Run x86 programs under Rosetta: arch --x86_64

• M1 compatibility status in dotfiles: https://github.com/changkun/dotfiles/issues/2

  • https://doesitarm.com/
  • https://isapplesiliconready.com/

• Got it in early December, have been using it for almost two months now — very smooth, battery life is incredible

• My essential third-party software list:

  • homebrew (compatibility is not great, but fortunately most dependent software is written in Go, and Go’s support is very complete)
    • Breaks compatibility casually, removes software distribution — there was a tool called rmtrash that I had been using since around 2014, but it was removed from distribution last year, so I wrote a fully compatible tool changkun.de/s/rmtrash, but it wasn’t merged; they said it needed to be approved by the original author to bypass popularity restrictions, but the original author is unreachable
  • vscode (have been using Insider long-term)
  • macvim
  • tmux
  • oh-my-zsh
  • Blender (Cycles ray tracing doesn’t support GPU, but editing meshes with less than a million vertices is fine)
  • iTerm: supports M1
  • Chrome: supports M1
  • MacTex: supports M1
  • Docker: released support a week before Christmas, works perfectly, no issues so far

1. Go Modules changes

   1. Collecting feedback
   2. Complex dependency management — what’s the most complex project dependency you’ve managed in practice, how many modules, and what do you write for each dependency upgrade? What did you use before Go modules?
      1. My experience: Go vendor, 1.10 dep, 1.11 go modules
      2. GOPATH project management — although GOPATH has been removed, I still follow the GOPATH convention
   3. Minimum version selection
      1. Semantic Versioning: major.minor.patch
      2. The classic diamond dependency problem: A depends on B and C, B and C each depend on different versions of D that are incompatible, so no specific version of D can be selected — semantic import versioning eliminates this by adding the major version number requirement at the end of the import path /v2
      3. dep doesn’t allow diamond dependencies, upgrades are very difficult
      4. Build reproducibility — without a lock file, >= dependencies change over time
      5. Select the minimum usable version — builds don’t change over time
      6. https://www.youtube.com/watch?v=F8nrpe0XWRg&ab_channel=SingaporeGophers
      7. Misunderstood working methods
      8. GOPATH
      9. vendor
      10. Three key points
      11. Compatibility
      12. Reproducibility
      13. Cooperation (often overlooked by many)
   4. Go Modules enabled by default, go build must include go.mod file, otherwise compilation fails
   5. build/test will not upgrade modules
   6. Default -mod=vendor

2. File system interface

   1. Why is the fs.FS abstraction important

      1. Unix file system abstract always disk blocks
      2. Network file systems (Upspin) abstract away machines
      3. REST abstracts nearly anything
      4. cp doesn’t care whether it’s moving file blocks, or even where the file is — it could be different disks or different machines
      5. Defines the “generics” for any file type tools

   2. What major changes it caused

      1. io/ioutil
         1. Russ Cox’s explanation of deprecated in Go (https://twitter.com/_rsc/status/1351676094664110082)
         2. https://www.srcbeat.com/2021/01/golang-ioutil-deprecated/
      2. Other fs abstractions
      3. Rob Pike’s 2016/2017 Gopherfest, Upspin, Changkun’s Midgard
         1. https://www.youtube.com/watch?v=ENLWEfi0Tkg&ab_channel=TheGoProgrammingLanguage
         2. FUSE: filesystem in userspace
         3. https://changkun.de/s/midgard
         4. Every user has a private root, no global root, r@golang.org/some/stuff, user names look like email addresses
         5. Access control defined by plain text files read: r@golang.org, ann@example.com
      4. Currently a very simple implementation, just a read-only file system
      5. ReadDir and DirEntry
         1. https://benhoyt.com/writings/go-readdir/
      6. Extensible directions: memoryFS, support for writing back to disk, hashFS for CDN support
      7. Remaining issues… e.g. 44166

      1 2 3 4 5 6 7

      import _ "embed" //go:embed a.txt var s string import "embed" type embed.String string var s embed.String

      1

3. File embedding //go:embed

   1. Basic functionality of the new feature
   2. Some possible applications
   3. Some features discussed during the feature freeze cycle
   4. https://blog.carlmjohnson.net/post/2021/how-to-use-go-embed/

4. Runtime memory management

   1. Return to MADV_DONTNEED
   • https://blog.changkun.de/posts/pss-uss-rss/
   2. New monitoring infrastructure runtime/metrics
   • Previous monitoring functions: runtime.ReadMemStats, debug.GCStats
   • runtime/metrics:
     • metrics.All()
     • Issue 37112

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

package main

import (
	"fmt"
	"runtime/metrics"
)

func main() {
	// Get descriptions for all supported metrics.
	descs := metrics.All()

	// Create a sample for each metric.
	samples := make([]metrics.Sample, len(descs))
	for i := range samples {
		samples[i].Name = descs[i].Name
	}

	// Sample the metrics. Re-use the samples slice if you can!
	metrics.Read(samples)

	// Iterate over all results.
	for _, sample := range samples {
		// Pull out the name and value.
		name, value := sample.Name, sample.Value

		// Handle each sample.
		switch value.Kind() {
		case metrics.KindUint64:
			fmt.Printf("%s: %d\n", name, value.Uint64())
		case metrics.KindFloat64:
			fmt.Printf("%s: %f\n", name, value.Float64())
		case metrics.KindFloat64Histogram:
			// The histogram may be quite large, so let's just pull out
			// a crude estimate for the median for the sake of this example.
			fmt.Printf("%s: %f\n", name, medianBucket(value.Float64Histogram()))
		case metrics.KindBad:
			// This should never happen because all metrics are supported
			// by construction.
			panic("bug in runtime/metrics package!")
		default:
			// This may happen as new metrics get added.
			//
			// The safest thing to do here is to simply log it somewhere
			// as something to look into, but ignore it for now.
			// In the worst case, you might temporarily miss out on a new metric.
			fmt.Printf("%s: unexpected metric Kind: %v\n", name, value.Kind())
		}
	}
}

func medianBucket(h *metrics.Float64Histogram) float64 {
	total := uint64(0)
	for _, count := range h.Counts {
		total += count
	}
	thresh := total / 2
	total = 0
	for i, count := range h.Counts {
		total += count
		if total > thresh {
			return h.Buckets[i]
		}
	}
	panic("should not happen")
}

1. Other noteworthy features
   1. os/signal.NotifyContext
   2. Memory model fixes
   3. Linker optimizations

Read these articles:

• What to expect when monitoring memory usage for modern Go applications. https://www.bwplotka.dev/2019/golang-memory-monitoring/
• Distributed Systems. https://www.youtube.com/watch?v=UEAMfLPZZhE
• Golang News. https://www.golangnews.com/
• SIGCHI Symposium on Engineering Interactive Computing Systems. http://eics.acm.org/
• runtime: use MADV_FREE on Linux if available. https://go-review.googlesource.com/c/go/+/135395/
• runtime: make the page allocator scale. https://github.com/golang/go/issues/35112
• runtime: add per-p mspan cache. https://go-review.googlesource.com/c/go/+/196642
• A New Smoothing Algorithm for Quadrilateral and Hexahedral Meshes. https://link.springer.com/content/pdf/10.1007%2F11758525_32.pdf
• OpenGL Docs. https://docs.gl
• On Playing Chess. https://blog.gardeviance.org/2018/03/on-playing-chess.html
• Memory Models: A Case For Rethinking Parallel Languages and Hardware. https://cacm.acm.org/magazines/2010/8/96610-memory-models-a-case-for-rethinking-parallel-languages-and-hardware/fulltext
• Engineer level & competency framework. https://github.com/spring2go/engineer_competency_framework
• A Concurrent Window System. http://doc.cat-v.org/bell_labs/concurrent_window_system/concurrent_window_system.pdf

How to create a window using Go? macOS has Cocoa, Linux has X11, but accessing these APIs seems to require Cgo. Is it possible to avoid Cgo? Some existing GUI libraries and graphics engines:

GUI Toolkits:

• https://github.com/hajimehoshi/ebiten
• https://github.com/gioui/gio
• https://github.com/fyne-io/fyne
• https://github.com/g3n/engine
• https://github.com/goki/gi
• https://github.com/peterhellberg/gfx
• https://golang.org/x/exp/shiny

2D/3D Graphics:

• https://github.com/llgcode/draw2d
• https://github.com/fogleman/gg
• https://github.com/ajstarks/svgo
• https://github.com/BurntSushi/graphics-go
• https://github.com/azul3d/engine
• https://github.com/KorokEngine/Korok
• https://github.com/EngoEngine/engo/
• http://mumax.github.io/

Here is a partial list:

https://github.com/avelino/awesome-go#gui

Most people use glfw and OpenGL. Here are some required libraries (Cgo bindings):

• https://github.com/go-gl/gl
• https://github.com/go-gl/glfw
• https://github.com/remogatto/egl

Some more low-level ones, e.g. X-related:

• X bindings: https://github.com/BurntSushi/xgb
• X window management: https://github.com/BurntSushi/wingo

For example, if you need Metal on macOS:

• https://dmitri.shuralyov.com/gpu/mtl

Like the GUI tool ebiten mentioned above, on Windows it no longer needs Cgo. The approach seems to be packaging the window management DLLs directly into the binary and then using DLL dynamic linking calls.

Besides GLFW, there is the heavier SDL:

• https://github.com/veandco/go-sdl2

Relationships between some basic terms:

1
2
3
4
5
6
7

Unix:      The ancestor
BSD:       Unix-like, Berkeley distribution, one of the two traditional flavors of Unix
System V:  Developed by AT&T, one of the two traditional flavors of Unix
Linux:     A fresh flavor of Unix-like OS
POSIX:     A standard attempting to reduce implementation differences
Darwin:    Apple's open-source Unix-like kernel XNU
Mach-O:    Microkernel hybridized in Darwin, developed by CMU

By Eraserhead1, Infinity0, Sav_vas - Levenez Unix History Diagram, Information on the history of IBM's AIX on ibm.com, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=1801948

Relationships between some Wayland-related tools:

1
2
3
4
5
6

X Window System == X11: Display standard developed by MIT, the foundation for GNOME and KDE
Xorg:    The official implementation of X11
Wayland: An alternative communication protocol between display server and client, distinct from X11
EGL:     Wayland clients use EGL to directly manipulate the framebuffer
libDRM:  Kernel API exposed to userspace, underlying dependency of EGL/X11
DRM:     Kernel-level component for manipulating the framebuffer

By Shmuel Csaba Otto Traian, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=28029855

By Shmuel Csaba Otto Traian, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=31768083

By Shmuel Csaba Otto Traian, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=27858390

By Shmuel Csaba Otto Traian，CC BY-SA 3.0，https://commons.wikimedia.org/w/index.php?curid=27799196

So through DRM you can directly operate on the Frame Buffer under Linux, i.e., Direct Rendering Manager. Related libraries:

• https://github.com/NeowayLabs/drm

With this, you can do pure Go rendering directly on Linux, eliminating the dependency on C legacy.

Read these articles:

• The Right to Read. https://www.gnu.org/philosophy/right-to-read.en.html
• Your Computer Isn’t Yours. https://sneak.berlin/20201112/your-computer-isnt-yours/
• Pirate Cinema. https://craphound.com/pc/download/

Noticed these two clipboard-related projects:

• https://github.com/binwiederhier/pcopy
• https://github.com/nakabonne/pbgopy

It seems their development timelines are very close to when I developed midgard, but we all took very different paths:

• pcopy focuses on the clipboard itself, with features like password protection, multiple clipboards, WebUI, etc.
• pbgopy emphasizes security for cross-device syncing, with various key configurations, but very simple functionality — only supports text

The midgard I developed focuses on these features:

• Automatic cross-device syncing (clipboard write-back), no commands needed
• Ability to query devices currently online
• Supports not only text but also image syncing
• Supports creating public URLs for clipboard content
• Supports converting code in clipboard to Carbon images
• Supports keyboard shortcuts
• …