The All in Go Stack全面转向 Go 技术栈

I stumbled upon an excerpt from an article called “The Rise of Worse is Better.” The author, Richard, reflects on why C and Unix succeeded. The article discusses the four major goals of software design: simplicity, correctness, consistency, and completeness. Two highly representative schools of thought have developed around these four goals: the MIT school and the New Jersey school (where Bell Labs is located). The MIT school believes that software must be absolutely correct and consistent first, then complete, and finally simple. It also “satirizes” the New Jersey school for doing the opposite – they set simplicity as the highest priority, even willing to sacrifice correctness for the sake of simplicity. In other words, software quality (popularity) does not increase with more features; from the perspective of practicality and ease of use, software with fewer features is actually more favored by users and the market.

So now you can see why some people always complain that Go can’t do this and can’t do that, is missing this and missing that. It’s because Rob Pike from Bell Labs is a through-and-through New Jersey school person. So to sum up, Go’s characteristics are:

1. Simple
2. Very simple
3. Nothing but simple

There are several follow-up articles on “Worse is Better”:

• Original: Richard P. Gabriel. The Rise of Worse is Better. 1989. https://www.dreamsongs.com/RiseOfWorseIsBetter.html
• Follow-up 1: Nickieben Bourbaki. Worse is Better is Worse. 1991. https://dreamsongs.com/Files/worse-is-worse.pdf
• Follow-up 2: Richard P. Gabriel. Is Worse Really Better? 1992. https://dreamsongs.com/Files/IsWorseReallyBetter.pdf
• Follow-up 3: Richard P. Gabriel. Worse is Better. 2000. https://www.dreamsongs.com/WorseIsBetter.html
• Follow-up 4: Richard P. Gabriel. Back to the Future: Worse (Still) is Better! Dec 04, 2000. https://www.dreamsongs.com/Files/ProWorseIsBetterPosition.pdf
• Follow-up 5: Richard P. Gabriel. Back to the Future: Is Worse (Still) Better? Aug 2, 2002. https://www.dreamsongs.com/Files/WorseIsBetterPositionPaper.pdf

So which school do you lean towards?

Today I read an extra paper. Although it’s not directly related to Go, I think it offers some insightful perspective on the current state of the Go language, so I’d like to share it. The paper is called “On Proebsting’s Law.”

We all know Moore’s Law says the number of transistors on integrated circuits doubles every 18 months, but this paper studies and validates the so-called Proebsting’s Law: the performance improvement brought by compiler optimization techniques doubles every 18 years. Proebsting’s Law was proposed in 1998, and its author Todd Proebsting was probably half-joking, because he suggested that the compiler and programming language research community should reduce their focus on performance optimization and instead pay more attention to improving programmer productivity.

Now, looking back at this suggestion with hindsight, we can see it’s not without merit: although Go’s compiler has gone through several major optimization versions, the techniques it uses aren’t particularly fancy – rather, they are quite traditional and conventional optimization techniques. However, this hasn’t hindered Go’s success, because what it tries to address is exactly programmer productivity:

1. By avoiding circular dependencies, it greatly reduces the time programmers spend waiting for compilation
2. Its very concise language design and feature set greatly reduces the time programmers spend thinking about how to use the language
3. Forward compatibility guarantees almost entirely eliminate the migration and maintenance time caused by version upgrades

• Paper: https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.29.434&rep=rep1&type=pdf
• Proebsting’s Law: http://proebsting.cs.arizona.edu/law.html

Because the COVID situation in Europe is still terrible, even shopping at an Apple Store requires an appointment in advance. Since I urgently needed to visit an Apple Store recently but couldn’t find any available appointment slots, I quickly hacked together a tool to check availability and send a reminder message via Telegram when an appointment becomes available. Tool link: https://changkun.de/s/apreserve

Interacting with Telegram using Go is straightforward:

1. Create a bot from BotFather
2. Obtain the bot’s token and the chat ID for your conversation with it
3. Then you can handle messages

• BotFather: https://t.me/botfather
• Tg bot API Go bindings: https://github.com/go-telegram-bot-api/telegram-bot-api

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

package main

import (
	tg "github.com/go-telegram-bot-api/telegram-bot-api"
)

func main() {
	token := …
	chatid := …
	bot, err := tg.NewBotAPI(token)
	if err != nil { … }
	bot.Send(tg.NewMessage(chatid, "text"))
}

How is Go’s compilation performance on darwin/arm64? I did a rough and non-rigorous comparison of Go compilation performance between an Intel Mac and an M1 Mac. This compilation report was generated with the following commands:

$ go build -gcflags='-bench=bench.out' -a $ cat bench.out

where -a disables the compilation cache.

MacBook Air (M1, 2020), Apple M1, 16 GB:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

commit: devel +7716a2fbb7 Sun Nov 22 11:44:49 2020 -0500
goos: darwin
goarch: arm64
BenchmarkCompile:main:fe:init              1     305167 ns/op     17.86 %
BenchmarkCompile:main:fe:loadsys           1     164916 ns/op      9.65 %
BenchmarkCompile:main:fe:parse             1     199209 ns/op     11.66 %    23 lines    115457 lines/s
BenchmarkCompile:main:fe:typecheck:top1    1       7625 ns/op      0.45 %
BenchmarkCompile:main:fe:typecheck:top2    1       2375 ns/op      0.14 %
BenchmarkCompile:main:fe:typecheck:func    1      34000 ns/op      1.99 %     1 funcs     29412 funcs/s
BenchmarkCompile:main:fe:capturevars       1        250 ns/op      0.01 %
BenchmarkCompile:main:fe:inlining          1       9500 ns/op      0.56 %
BenchmarkCompile:main:fe:escapes           1       5834 ns/op      0.34 %
BenchmarkCompile:main:fe:xclosures         1      49583 ns/op      2.90 %
BenchmarkCompile:main:fe:subtotal          1     778459 ns/op     45.56 %
BenchmarkCompile:main:be:compilefuncs      1     279125 ns/op     16.34 %     1 funcs      3583 funcs/s
BenchmarkCompile:main:be:externaldcls      1        500 ns/op      0.03 %
BenchmarkCompile:main:be:dumpobj           1     639667 ns/op     37.44 %
BenchmarkCompile:main:be:subtotal          1     919292 ns/op     53.81 %
BenchmarkCompile:main:unaccounted          1      10791 ns/op      0.63 %
BenchmarkCompile:main:total                1    1708542 ns/op    100.00 %

Mac mini (2018), 3 GHz 6-Core Intel Core i5, 8 GB 2667 MHz DDR4:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

commit: go1.15.6
goos: darwin
goarch: amd64
BenchmarkCompile:main:fe:init              1     333752 ns/op     15.31 %
BenchmarkCompile:main:fe:loadsys           1     246343 ns/op     11.30 %
BenchmarkCompile:main:fe:parse             1     372343 ns/op     17.08 %    23 lines    61771 lines/s
BenchmarkCompile:main:fe:typecheck:top1    1      19620 ns/op      0.90 %
BenchmarkCompile:main:fe:typecheck:top2    1       7347 ns/op      0.34 %
BenchmarkCompile:main:fe:typecheck:func    1      23073 ns/op      1.06 %     1 funcs    43341 funcs/s
BenchmarkCompile:main:fe:capturevars       1        238 ns/op      0.01 %
BenchmarkCompile:main:fe:inlining          1      13277 ns/op      0.61 %
BenchmarkCompile:main:fe:escapes           1      19283 ns/op      0.88 %
BenchmarkCompile:main:fe:xclosures         1     102213 ns/op      4.69 %
BenchmarkCompile:main:fe:subtotal          1    1137489 ns/op     52.19 %
BenchmarkCompile:main:be:compilefuncs      1     598194 ns/op     27.45 %     1 funcs     1672 funcs/s
BenchmarkCompile:main:be:externaldcls      1        766 ns/op      0.04 %
BenchmarkCompile:main:be:dumpobj           1     415450 ns/op     19.06 %
BenchmarkCompile:main:be:subtotal          1    1014410 ns/op     46.54 %
BenchmarkCompile:main:unaccounted          1      27696 ns/op      1.27 %
BenchmarkCompile:main:total                1    2179595 ns/op    100.00

ioutil will be fully deprecated in Go 1.16. Although these APIs will continue to exist due to the compatibility guarantee, they are no longer recommended for use. So the question is: what should we use instead? Here are all the APIs in the ioutil package:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

package ioutil

var Discard io.Writer
func NopCloser(r io.Reader) io.ReadCloser
func ReadAll(r io.Reader) ([]byte, error)

func ReadDir(dirname string) ([]os.FileInfo, error)
func ReadFile(filename string) ([]byte, error)
func WriteFile(filename string, data []byte, perm os.FileMode) error

func TempDir(dir, pattern string) (name string, err error)
func TempFile(dir, pattern string) (f *os.File, err error)

The corresponding replacement APIs in 1.16:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

package io

var Discard Writer
func NopCloser(r Reader) ReadCloser
func ReadAll(r Reader) ([]byte, error)


package os

func ReadDir(name string) ([]DirEntry, error)
func ReadFile(name string) ([]byte, error)
func WriteFile(name string, data []byte, perm fs.FileMode) error

func MkdirTemp(dir, pattern string) (string, error)
func CreateTemp(dir, pattern string) (f *File, err error)

In summary, three key changes:

1. Discard, NopCloser, ReadAll have been moved to the io package
2. ReadDir, ReadFile, WriteFile have been moved to the os package
3. TempDir, TempFile have been renamed to MkdirTemp, CreateTemp and moved to the os package

io/fs is getting closer and closer. The functionality is great, but how do we test it? There is a function in testing/fstest that can do just that.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

package fstest

// TestFS tests a file system implementation.
// It walks the entire tree of files in fsys,
// opening and checking that each file behaves correctly.
// It also checks that the file system contains at least the expected files.
// As a special case, if no expected files are listed, fsys must be empty.
// Otherwise, fsys must only contain at least the listed files: it can also contain others.
//
// If TestFS finds any misbehaviors, it returns an error reporting all of them.
// The error text spans multiple lines, one per detected misbehavior.
//
// Typical usage inside a test is:
//
//	if err := fstest.TestFS(myFS, "file/that/should/be/present"); err != nil {
//		t.Fatal(err)
//	}
//
func TestFS(fsys fs.FS, expected ...string) error

Are you sure you understand asynchronous preemption? Today, while discussing with Cao Da (@Xargin) about how the interrupted G in the asynchronous preemption flow restores its previous execution context, I realized my understanding of asynchronous preemption was not comprehensive enough. In “Go Under The Hood,” asynchronous preemption is described as follows: let’s name the two running threads M1 and M2. The overall logic of a preemption call can be summarized as:

1. M1 sends an interrupt signal (signalM(mp, sigPreempt))
2. M2 receives the signal; the OS interrupts its executing code and switches to the signal handler (sighandler(signum, info, ctxt, gp))
3. M2 modifies the execution context and resumes at the modified location (asyncPreempt)
4. Re-enters the scheduling loop to schedule other Goroutines (preemptPark and gopreempt_m)

This summary is not entirely correct, because it does not clearly explain the difference between preemptPark and gopreempt_m. This week, let’s briefly supplement the overall behavior of asynchronous preemption:

Assuming the system monitor acts as M1, after the system monitor sends the interrupt signal, execution arrives at asyncPreempt2:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

//go:nosplit
func asyncPreempt2() {
	gp := getg()
	gp.asyncSafePoint = true
	if gp.preemptStop {
		mcall(preemptPark)
	} else {
		mcall(gopreempt_m)
	}
	gp.asyncSafePoint = false
}

But will it ultimately choose preemptPark or gopreempt_m? The asynchronous preemption issued by sysmon calling preemptone does not set the preemptStop flag on G, so it enters the gopreempt_m flow. gopreempt_m ultimately calls goschedImpl, which places the preempted G into the global queue to be scheduled later.

So what about the other half (preemptPark)? When we carefully examine the implementation of preemptPark, we find that the preempted G is not added to the scheduling queue at all — instead, it directly calls schedule:

1
2
3
4
5
6
7
8

//go:systemstack
func preemptPark(gp *g) {
	...
	casGToPreemptScan(gp, _Grunning, _Gscan|_Gpreempted)
	dropg()
	casfrom_Gscanstatus(gp, _Gscan|_Gpreempted, _Gpreempted)
	schedule()
}

So how does the preempted G get back into the scheduling loop? It turns out that the branch where gp.preemptStop is true occurs when the GC needs it (markroot): it uses suspendG to mark the running G (gp.preemptStop = true), sends the preemption signal (preemptM), and returns the state of the interrupted G. When the GC’s marking work is complete and preemption ends, it passes this state and calls resumeG, which ultimately calls ready to resume the interrupted G:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

//go:nowritebarrier
func markroot(gcw *gcWork, i uint32) {
	...
	var gp *g
	systemstack(func() {
		...
		stopped := suspendG(gp)
		...
		scanstack(gp, gcw) // GC stack scanning
		gp.gcscandone = true
		resumeG(stopped)
		...
	})
}
//go:systemstack
func suspendG(gp *g) suspendGState {
	...
	stopped := false
	for i := 0; ; i++ {
		switch s := readgstatus(gp); s {
		...
		case _Gpreempted:
			...
			stopped = true
			gp.preemptStop = false
			gp.preempt = false
			gp.stackguard0 = gp.stack.lo + _StackGuard
			return suspendGState{g: gp, stopped: stopped}

		case _Grunning:
			...
			gp.preemptStop = true
			gp.preempt = true
			gp.stackguard0 = stackPreempt
			casfrom_Gscanstatus(gp, _Gscanrunning, _Grunning)
			preemptM(gp.m)
		}
		...
	}
}
func resumeG(state suspendGState) {
	...
	gp := state.g
	ready(gp, 0, true)
}

To see how tedious error handling can be, just look at this very thorough summary:

https://seankhliao.com/blog/12020-11-23-go-error-handling-proposals/

Does this code have a data race?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

// from https://golang.org/issue/42598
type module struct {
	v int
}

func foo() {
	mods := []*module{
		&module{v: 0},
		&module{v: 1},
		&module{v: 2},
	}
	type token struct{}
	sem := make(chan token, runtime.GOMAXPROCS(0))
	for _, m := range mods {
		add := func(m *module) {
			sem <- token{}
			go func() {
				*m = module{v: 42} // write
				<-sem
			}()
		}
		add(m) // read
	}
	// Fill semaphore channel to wait for all tasks to finish.
	for n := cap(sem); n > 0; n-- {
		sem <- token{}
	}
}

An issue submitted yesterday appears to point out a bug in the current Go memory model. Further reading:

• https://golang.org/issue/42598
• https://golang.org/issue/37355
• https://go-review.googlesource.com/c/go/+/220419/
• https://reviews.llvm.org/D76322