Revisiting Asynchronous Preemption

Are you sure you understand asynchronous preemption? Today, while discussing with Cao Da (@Xargin) about how the interrupted G in the asynchronous preemption flow restores its previous execution context, I realized my understanding of asynchronous preemption was not comprehensive enough. In “Go Under The Hood,” asynchronous preemption is described as follows: let’s name the two running threads M1 and M2. The overall logic of a preemption call can be summarized as:

M1 sends an interrupt signal (signalM(mp, sigPreempt))
M2 receives the signal; the OS interrupts its executing code and switches to the signal handler (sighandler(signum, info, ctxt, gp))
M2 modifies the execution context and resumes at the modified location (asyncPreempt)
Re-enters the scheduling loop to schedule other Goroutines (preemptPark and gopreempt_m)

This summary is not entirely correct, because it does not clearly explain the difference between preemptPark and gopreempt_m. This week, let’s briefly supplement the overall behavior of asynchronous preemption:

Assuming the system monitor acts as M1, after the system monitor sends the interrupt signal, execution arrives at asyncPreempt2:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


//go:nosplit
func asyncPreempt2() {
	gp := getg()
	gp.asyncSafePoint = true
	if gp.preemptStop {
		mcall(preemptPark)
	} else {
		mcall(gopreempt_m)
	}
	gp.asyncSafePoint = false
}

But will it ultimately choose preemptPark or gopreempt_m? The asynchronous preemption issued by sysmon calling preemptone does not set the preemptStop flag on G, so it enters the gopreempt_m flow. gopreempt_m ultimately calls goschedImpl, which places the preempted G into the global queue to be scheduled later.

So what about the other half (preemptPark)? When we carefully examine the implementation of preemptPark, we find that the preempted G is not added to the scheduling queue at all — instead, it directly calls schedule:

1
2
3
4
5
6
7
8


//go:systemstack
func preemptPark(gp *g) {
	...
	casGToPreemptScan(gp, _Grunning, _Gscan|_Gpreempted)
	dropg()
	casfrom_Gscanstatus(gp, _Gscan|_Gpreempted, _Gpreempted)
	schedule()
}

So how does the preempted G get back into the scheduling loop? It turns out that the branch where gp.preemptStop is true occurs when the GC needs it (markroot): it uses suspendG to mark the running G (gp.preemptStop = true), sends the preemption signal (preemptM), and returns the state of the interrupted G. When the GC’s marking work is complete and preemption ends, it passes this state and calls resumeG, which ultimately calls ready to resume the interrupted G:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


//go:nowritebarrier
func markroot(gcw *gcWork, i uint32) {
	...
	var gp *g
	systemstack(func() {
		...
		stopped := suspendG(gp)
		...
		scanstack(gp, gcw) // GC stack scanning
		gp.gcscandone = true
		resumeG(stopped)
		...
	})
}
//go:systemstack
func suspendG(gp *g) suspendGState {
	...
	stopped := false
	for i := 0; ; i++ {
		switch s := readgstatus(gp); s {
		...
		case _Gpreempted:
			...
			stopped = true
			gp.preemptStop = false
			gp.preempt = false
			gp.stackguard0 = gp.stack.lo + _StackGuard
			return suspendGState{g: gp, stopped: stopped}

		case _Grunning:
			...
			gp.preemptStop = true
			gp.preempt = true
			gp.stackguard0 = stackPreempt
			casfrom_Gscanstatus(gp, _Gscanrunning, _Grunning)
			preemptM(gp.m)
		}
		...
	}
}
func resumeG(state suspendGState) {
	...
	gp := state.g
	ready(gp, 0, true)
}

你确定你看懂异步抢占了吗？今天跟曹大@Xargin 交流起异步抢占的流程里被中断的 G 是如何恢复到之前的执行现场时才发现对异步抢占的理解还不够全面。在《Go 语言原本》中是这样描述异步抢占的: 不妨给正在运行的两个线程命名为 M1 和 M2，抢占调用的整体逻辑可以被总结为:

M1 发送中断信号（signalM(mp, sigPreempt)）
M2 收到信号，操作系统中断其执行代码，并切换到信号处理函数（sighandler(signum, info, ctxt, gp)）
M2 修改执行的上下文，并恢复到修改后的位置（asyncPreempt）
重新进入调度循环进而调度其他 Goroutine（preemptPark 和 gopreempt_m）

这个总结并不完全正确，因为它并没有总结清楚 preemptPark 和 gopreempt_m 这两者之间的区别。这周我们来简单补充一下异步抢占的整体行为:

假设系统监控充当 M1，当系统监控发送中断信号后，会来到 asyncPreempt2:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


//go:nosplit
func asyncPreempt2() {
	gp := getg()
	gp.asyncSafePoint = true
	if gp.preemptStop {
		mcall(preemptPark)
	} else {
		mcall(gopreempt_m)
	}
	gp.asyncSafePoint = false
}

但最终会选择 preemptPark 还是 gopreempt_m 呢？sysmon 调用 preemptone 的代码中发出的异步抢占并不会为 G 设置 preemptStop 标记，从而会进入 gopreempt_m 的流程，而 gopreempt_m 最终会调用 goschedImpl 将被抢占的 G 放入全局队列，等待日后被调度。

那么另一半（preemptPark）呢？当我们仔细查看 preemptPark 的实现则会发现，被抢占的 G 其实并没有被加入到调度队列中，而是直接就调用了 schedule:

1
2
3
4
5
6
7
8


//go:systemstack
func preemptPark(gp *g) {
	...
	casGToPreemptScan(gp, _Grunning, _Gscan|_Gpreempted)
	dropg()
	casfrom_Gscanstatus(gp, _Gscan|_Gpreempted, _Gpreempted)
	schedule()
}

那这时被抢占的 G 怎样才会恢复到调度循环呢？原来 gp.preemptStop 为 true 的分支发生在 GC 需要时（markroot）通过 suspendG 来标记正在运行的 G（gp.preemptStop = true），再发送抢占信号（preemptM），返回被中断 G 的状态。当 GC 的标记工作完成，抢占结束后，在将这个状态传递并调用 resumeG，最终 ready 并恢复这个被中断的 G:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


//go:nowritebarrier
func markroot(gcw *gcWork, i uint32) {
	...
	var gp *g
	systemstack(func() {
		...
		stopped := suspendG(gp)
		...
		scanstack(gp, gcw) // GC 栈扫描
		gp.gcscandone = true
		resumeG(stopped)
		...
	})
}
//go:systemstack
func suspendG(gp *g) suspendGState {
	...
	stopped := false
	for i := 0; ; i++ {
		switch s := readgstatus(gp); s {
		...
		case _Gpreempted:
			...
			stopped = true
			gp.preemptStop = false
			gp.preempt = false
			gp.stackguard0 = gp.stack.lo + _StackGuard
			return suspendGState{g: gp, stopped: stopped}

		case _Grunning:
			...
			gp.preemptStop = true
			gp.preempt = true
			gp.stackguard0 = stackPreempt
			casfrom_Gscanstatus(gp, _Gscanrunning, _Grunning)
			preemptM(gp.m)
		}
		...
	}
}
func resumeG(state suspendGState) {
	...
	gp := state.g
	ready(gp, 0, true)
}