The All in Go Stack全面转向 Go 技术栈

Since graduating from university I have had very little time to tinker with my own website. Most of the time I would ssh into the server, throw together some service, and then leave it to fend for itself — even if a service went down I did not pay much attention. After all, it was something I ran for my own use, and as far as visitors were concerned: ABSOLUTELY NO WARRANTY. But as the site’s user numbers have recently grown at an almost unbelievable rate (despite doing essentially nothing), I naturally find myself wanting to make it more “reliable”.

As some of the last items on my 2020 TODO list, I finally spent the first three days of my Christmas holiday completing a full “architecture upgrade” of changkun.de. The previous setup was an organisationally chaotic, dependency-heavy mess of native Nginx + Docker + tmux-served binaries + Jupyter notebooks + hypervisors + crontabs + … relying on C/C++/Python/Node.js/Go/… and countless third-party dependencies. All of that has now been replaced by a pure Go backend guided by the principle: “build in-house wherever possible, avoid dependencies wherever you can, and when you must depend on something, depend on something written in Go”.

This post describes changkun.de as a personal website — from its early days of fewer than a hundred active users per year to today’s tens of thousands of active users per month — covering what services (public but not especially visible) have accumulated along the way, and the migration story behind them.

Migration History

Every “major” migration of the blog changkun.de/blog has been documented in a post, and they can all be found here.

Looking back, changkun.de has quite a long migration history. From the earliest days of content scattered across various BBS platforms (Tieba, Sina Blog, QQ Zone), to WordPress during my undergraduate years (2012–2015), then to the first and second iterations of Hexo (2015–2017), the third Hexo iteration that lasted until recently (2017–2020), and now the freshly launched pure Go application (2021–present).

The Wild Age (2008–2013): BBS

Before 2013, I was a pure mathematics enthusiast (in high school) who knew absolutely nothing about computing and had once looked down on it. I spent my time on early BBS communities like the geometry and mathematics boards on Tieba, Mop, and Tianya, chatting about maths with strangers — naturally I had no idea what “building a website” meant. Much of my early content is scattered across those platforms; a lot of what appears in the blog as early writing was later collected from those sources. The specific history is almost impossible to trace, and at that point I had not yet registered changkun.de.

The Ancient Age (2013–2015): WordPress (LAMP Stack)

Figure 1: The LAMP Stack

I had expected to study mathematics at university, but ended up in computer science almost by accident. That was when I truly started learning the subject. A year in, while still sneaking into mathematics department lectures, I spent a small amount of time picking up various C tricks (reading books like C and Pointers and Expert C Programming) and formed a basic understanding of what computer science was about — and it was then that I built my first website. The domain I registered back then is not the one I use now (I no longer own it, so I will not mention it here). In 2013 there was no convenient tool like Docker; building an independent site meant going through roughly these steps:

1. Purchase a domain and a server.
2. Manually compile and install MySQL on the server.
3. Manually install WordPress on the server.
4. No HTTPS.

The First Age (2015–2016): Hexo + GitHub Pages

Figure 2: Hexo + GitHub Pages

Three years of undergraduate study came and went quickly. I had grown from someone who knew nothing about computing into someone who could write a parser in C by hand — but before I could enter the workforce, in the second half of 2015 I found myself in Germany for a semester of study abroad. At the time I had very little knowledge of web technology, until I sat in on a course called Online Multimedia. That opened up a whole new world: Polymer, Angular, MongoDB. Curious about various web technologies, I discovered the concept of static blogs and came across the relatively mature Hexo.

Given that the blog was a personal space with fewer than a hundred active users per year, continuing with Linux+Apache+MySQL+PHP was clearly overkill and unwieldy. On top of that, my server was on Alibaba Cloud in China while I was in Germany, and a direct SSH connection would typically drop after just a few commands — making it impossible to get anything done. So I bit the bullet and migrated the site to GitHub Pages. That migration involved:

1. Exporting posts from WordPress/MySQL and converting them to Markdown.
2. Compiling the static blog locally and pushing to GitHub by hand.
3. Still no HTTPS.

The Second Age (2016–2017): Hexo + Git Server + HTTPS

The First Age brought liberation from the heavy LAMP stack, cutting the whole thing down to zero-cost “serverless” computing.

But this was not without drawbacks. The internet environment back then was not as good as it is today. My early WordPress site had been well indexed by search engines and received a fair amount of organic traffic. After deploying to GitHub Pages, however, the blog was never indexed again — the site had effectively vanished from the internet. Another drawback was that GitHub Pages at the time did not support custom HTTPS, and although it is arguably unnecessary for a static blog, being able to enable HTTPS was already a mark of prestige.

Figure 3: HTTPS is now required

A key historical turning point that pushed the entire internet toward HTTPS was Apple’s decision to require iOS app APIs to use HTTPS. At the time, the mobile internet boom was in full swing and I was very keen to build an app for my blog. If any early readers of the blog are still around, they may remember that this blog once had both a desktop client (based on Electron) and a mobile client (based on React Native). To support API access, I even wrote a Hexo RESTful API Generator.

So this migration involved:

1. Configuring Nginx as a gateway on the server.
2. Setting up a Git server and configuring a post-hook script.
3. Compiling the static blog locally, pushing to the Git server, and triggering the post-hook to update the static files under /www/blog.
4. Enabling HTTPS.

Worth noting: in 2016, Let’s Encrypt did not yet exist, and self-signed certificates were not trusted by mainstream browsers, so finding a reliable and free SSL certificate took some time.

The Third Age (2017–2020): Hexo + GitHub + Travis CI + VPS

Figure 3: HTTPS is now required

The approach of the Second Age was not particularly secure, and it was heavily dependent on the safety of the local source files. If those files were lost, recovering Markdown from compiled HTML would have been considerable work (handling images, equations, and so on). Furthermore, pushing the compiled static files to the server each time was severely limited by the transfer speed between the local machine and the server. This was manageable for plain text, but pushing images or other binary files was a disaster. I still remember opening the laptop, running push, and then waiting a full hour for a single blog post update — uploads from Germany to an Alibaba Cloud server in southern China were around 10 KB/s at the time.

To address the slow connection, the blog was given a CI/CD pipeline. The core purpose was simple: once I pushed the source, the CI system would handle the compilation and upload automatically, reducing each blog update to a single commit. While CI did not completely solve the upload speed problem to Alibaba Cloud, the upload was now asynchronous, which was acceptable on the whole.

The dominant CI/CD system at the time was Travis CI, so that is what went in. The migration itself was not large, but a significant amount of time was spent tuning the pipeline — for example using gulp (a bundler that predates Webpack) to minify assets and compress images. These things seem trivial today, but were not so straightforward back then.

There was also one notably unusual aspect: route design. The site’s URLs were structured as:

• Posts: /archives/{{year}}/{{month}}/{{id}}
• Images: /images/posts/{{id}}/{{image}}.{{format}}

The ID here was a global post ID. Since Hexo does not assign global IDs to posts the way a database would, I wrote a dedicated script for this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76

import yaml
import glob
import time

POST_DIR = 'source/_posts/'


def loader():
    posts = glob.glob(POST_DIR + '*.md')
    settings = []

    for post in posts:
        raw = open(post).read()
        head = raw.split('---')[1]
        try:
            setting = yaml.load(head)
            if set(['title', 'date', 'tags', 'id']).issubset(setting):
                setting['path'] = post
                settings.append(setting)
            else:
                print('YAML parsing error: ', post)
        except Exception as e:
            print('YAML head not avaliable: ', post)

    return len(settings) == len(posts), settings


def id_repair():
    all_correct, settings = loader()
    if all_correct:
        print('ALL SUCCESS')
        fixed = sorted(settings, key=lambda head: head['date'])
        for index, fix in enumerate(fixed):
            fix['id'] = index + 1
        return fixed, settings
    else:
        print('PLEASE FIX YAML HEAD FORMAT FIRST')


def id_writer():
    # goal:
    # repair id: ../images/posts/old/* => /images/posts/fixed/*
    fixed, old = id_repair()
    for i, fix in enumerate(fixed):
        try:
            with open(fix['path'], 'r', encoding='utf-8') as f:
                raw = f.read()
                try:
                    content = raw.split('---', 1)[1].split('---', 1)[1]
                    content = content.replace('](../images/posts/', '](/images/posts/')
                    content = content.replace(
                        '/images/posts/' + str(old[i]['id']), '/images/posts/' + str(fix['id']))
                except Exception as e:
                    print('ERROR: ', fix['path'])
                    print(e)

            with open(fix['path'], 'w', encoding='utf-8') as f:
                f.write('---\n')
                stream = yaml.dump(
                    fix, default_flow_style=False, allow_unicode=True)
                f.write(stream.replace('\n- ', '\n  - '))
                f.write('---')
                f.write(content)
                f.close()
            print('SUCCESS: ', fix['path'])
        except Exception as e:
            print('FAILED: ', fix['path'])
            print('REASON: ', e)


def main():
    id_writer()


if __name__ == '__main__':
    main()

The script iterates over all posts, sorts them by publication date, and assigns IDs in chronological order. The reasons for this approach were:

1. A new post might need to be inserted at an old date (for example, recovering something written on an early BBS that was worth archiving).
2. During writing, using relative paths for images makes previewing convenient; after generation, the image URLs are rewritten to ensure correct routing.

Looking back, the /year/month/id routing was frankly user-hostile, because it conveys no useful information whatsoever. At the time it was perhaps motivated by a desire to make scraping harder — but that now seems pointless, and it has only complicated the current migration.

Motivation for Migrating

The first major reason for this migration was data-driven. As mentioned above, this blog once had fewer than a hundred active users per year, but has now grown to tens of thousands of active users per month — something that genuinely astonished me.

This is partly thanks to some anonymous promotions this year that I had nothing to do with — for example, the clumsy C++ tutorial I wrote as an undergraduate was promoted by the well-known Chinese publication Jiqizhixin without ever contacting me. That vaulted me into the ranks of people with a GitHub project boasting 11k+ stars. If any reader has spotted the many typos in that tutorial, I sincerely apologise — but seeing one’s own work recognised by others is genuinely pleasing, whatever the circumstances.

The second reason for migrating is that it had become increasingly difficult to compile the entire Hexo-based blog on a fresh machine. My personal habit is to wrap every build step in a single make command. But over time I found the Hexo ecosystem and everything around it making ever more breaking changes:

1. The gulp build script no longer ran correctly — fixing the breaking changes required re-learning how gulp works and rewriting the build script from scratch.
2. It was impossible to upgrade the blog cleanly to Hexo 5.0 or later.
3. The upstream theme, hexo-theme-next, had long since stopped being maintained.
4. My own fork, changkun/hexo-theme-next, contained a great deal of bespoke customisation.
5. …and so on.

All of this made the entire Node.js ecosystem feel profoundly unreliable. Combined with my increasingly heavy use of Go over recent years, and a growing attraction to the New Jersey Style philosophy that Go embodies, I found I trusted Go’s authors far more than I trusted the wildly inconsistent Node.js and npm ecosystem. That set a clear goal for this migration:

Could these personal services — a mixture of C/C++/Python/JS/Go — be restructured and simplified into a website driven purely by Go?

As the existence of this post demonstrates, the answer is yes.

The Cost of Migration

Any migration naturally prompts the question: it’s 2020 — why aren’t you using Kubernetes yet? Admittedly, a personal website is a technology playground, and experimenting has always been a major motivation for maintaining one. So it is not that I never considered it. Between 2018 and 2020 I made three attempts to move toward Kubernetes and the broader cloud-native ecosystem for this modest static blog. But my answer today remains: do not multiply entities beyond necessity (Occam’s Razor). Kubernetes is excellent and solves real problems in its domain; a personal website has different needs.

Before the migration, all my services ran on a $5/month Digital Ocean Droplet. Let us work out what Kubernetes would actually cost:

1. Kubernetes Cluster: $30/month (1 Worker Node = $10/month, ×3)
2. Container Registry: $5/month (Basic Plan, 5 repos, 5 GB storage)
3. Spaces: $5/month (250 GB)

I did look at other clouds, but most are more expensive than DO (GCP, for instance). It is not that I could not afford it — the undergraduate version of me, obsessed with adopting the latest technology, would certainly have switched. But with experience comes conservatism.

The first reason is having witnessed enough cycles: I was once a heavy Evernote user, but in my view the wildly popular Notion is just another Evernote riding a wave of venture capital, and I no longer use Evernote either — migrating all those notes out is still a headache I have not fully dealt with. There are many other examples. The second reason is my conviction that the essence of technology does not really change; what is called “new” is usually old wine in a new bottle. Rather than chasing every wave, I would rather sit down, read some papers, and ask myself how many mathematically grounded algorithms I can actually derive and implement.

So back to the original question: what Go stack actually makes sense for a personal website in 2020?

The New Age (2021–)

Migrating from Hexo to Hugo

The first major step toward a Go stack was escaping from Hexo. This was not a trivial requirement. Although I could discard the old Next theme entirely, I had used it for so many years that I hoped to find an equivalent for Hugo — and sure enough, someone had already done it: xtfly/hugo-theme-next.

Unfortunately the author built it for their own use, with limited migration support, and the initial experience revealed several problems:

1. Animation flickering on load: this seems to be a longstanding Next theme bug — the author had apparently ported the JavaScript directly.
2. Superfluous features: geolocation tracking in the sidebar, post categories (I prefer tags exclusively), and so on.
3. Required customisations: changing the “Powered by XYZ” footer to “Copyright Changkun Ou”, and similar tweaks.

The route design from the Third Age had to be dealt with in this migration too. Post URLs were unified to the /posts/:slug format. Hugo helpfully provides an alias feature: you can list alternative URLs in a post’s YAML front matter, and visitors to those URLs will be redirected to the canonical slug:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

---
-date: 2020-03-08 21:56:31
+date: "2020-03-08 21:56:31"
+toc: true
id: 264
-path: source/_posts/2020-03-08-我为什么不再写博客了？.md
+slug: /posts/why-i-stopped-blogging
+aliases:
+    - /archives/2020/03/264/
tags:
-  - 随笔
+    - 随笔
+    - 博客
title: 我为什么不再写博客了？
---

This was not done by hand, of course. I wrote a small Go script for it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63

type oldh struct {
	Date  string   `yaml:"date"`
	ID    int      `yaml:"id"`
	Path  string   `yaml:"path"`
	Tags  []string `yaml:"tags"`
	Title string   `yaml:"title"`
}
type newh struct {
	Date    string   `yaml:"date"`
	TOC     bool     `yaml:"toc"`
	ID      int      `yaml:"id"`
	Slug    string   `yaml:"slug"`
	Aliases []string `yaml:""`
	Tags    []string `yaml:"tags"`
	Title   string   `yaml:"title"`
}
func main() {
	filepath.Walk("../../content/posts", func(path string, info fs.FileInfo, err error) error {
		if info.IsDir() { return nil }

		b, err := os.ReadFile(path)
		if err != nil { panic(err) }

		ss := strings.Split(string(b), "---")
		header := []byte(ss[1])

		var old oldh
		err = yaml.Unmarshal(header, &old)
		if err != nil {
			log.Fatalf("cannot parse configuration, err: %v\n", err)
		}

		dd, err := time.Parse("2006-01-02 15:04:05", old.Date)
		if err != nil {
			panic(err)
		}

		newHeader := newh{
			Date: old.Date,
			TOC:  true,
			ID:   old.ID,
			Slug: fmt.Sprintf("/posts/%s", strings.Replace(strings.ToLower(old.Title), " ", "-", -1)),
			Aliases: []string{
				fmt.Sprintf("/archives/%d/%02d/%d/", dd.Year(), dd.Month(), old.ID),
			},
			Tags:  old.Tags,
			Title: old.Title,
		}

		b, err = yaml.Marshal(newHeader)
		if err != nil {
			panic(err)
		}

		ss[1] = string(b)

		err = os.WriteFile(path, []byte(strings.Join(ss, "---\n")), os.ModePerm)
		if err != nil {
			panic(err)
		}
		return nil
	})
}

When migrating the routes, the Disqus comments also needed migrating. Disqus provides a migration tool that accepts a URL mapping table, for example:

1
2
3
4
5
6

https://changkun.de/blog/archives/2020/01/262/, https://changkun.de/blog/posts/2018-2019-reading/
https://changkun.de/blog/archives/2020/02/263/, https://changkun.de/blog/posts/2019-summary/
https://changkun.de/blog/archives/2020/03/264/, https://changkun.de/blog/posts/why-i-stopped-blogging/
https://changkun.de/blog/archives/2020/03/265/, https://changkun.de/blog/posts/setup-wordpress-in-10-minutes/
https://changkun.de/blog/archives/2020/09/266/, https://changkun.de/blog/posts/eliminating-a-source-of-measurement-errors-in-benchmarks/
https://changkun.de/blog/archives/2020/11/267/, https://changkun.de/blog/posts/pointers-might-not-be-ideal-for-parameters/

In summary, the final Hugo blog uses a theme I forked and maintain myself. Interested readers can find my customised version in the blog’s source repository. The fork focuses on fixing the issues mentioned above and adds:

1. Dark mode
2. Disqus support; removal of some redundant sharing buttons (Baidu, WeChat, Duoshuo — the last of which has long since shut down)
3. Blogroll (友链)
4. Tag cloud scaled by frequency
5. …and more

Building with the Go 1.16 Embed Feature

Go 1.16 introduced a new feature for embedding files directly into a compiled binary.

The following code shows how to use this Go 1.16 feature to bundle all the generated static files under public/* into a single binary:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

package main

import (
	"embed"
	"io/fs"
	"net/http"
	"os"
	"path"
)

//go:embed public/*
var public embed.FS

// blogFS implements fs.FS
type blogFS struct {
	content embed.FS
}

// Open opens a given filename from the public folder.
func (b blogFS) Open(name string) (fs.File, error) {
	return b.content.Open(path.Join("public", name))
}

func main() {
	r := http.NewServeMux()
	r.Handle("/", http.FileServer(http.FS(blogFS{public})))
	const addr = "0.0.0.0:9129"
	s := &http.Server{Addr: addr, Handler: r}
	if err := s.ListenAndServe(); err != nil && err != http.ErrServerClosed {
		panic(fmt.Sprintf("cannot listen on %s, err: %v\n", addr, err))
	}
}

Lightweight Container Orchestration

Having decided to move the entire site to a pure Go stack, we needed to think through the changes required. Given that Kubernetes was ruled out, what about Docker Swarm as a lighter alternative? Equally unnecessary. But containerisation itself was a given: the packaging and distribution convenience it brings, along with runtime isolation and the ability to run multiple replicas, far outweighs its downsides for long-term management. And since Docker is also a Go project (politically correctly called Moby), Docker Compose was the natural companion.

One of the things I love about Go is its near-perfect compilation characteristics. A simple go build resolves almost all dependencies — no static linking to worry about, no runtime dynamic linking to manage. Those steps happen, but they are not my problem as the developer. This makes the entire build pipeline remarkably simple:

1
2
3
4
5

NAME=main
VERSION = $(shell git describe --always --tags)
build:
	CGO_ENABLED=0 GOOS=linux go build # cross-compile
	docker build -t $(NAME):$(VERSION) -t $(NAME):latest -f Dockerfile .

The Dockerfile simply drops the compiled binary into an alpine image:

1
2
3
4

# just copy the binary
FROM alpine
COPY main /app/main
CMD ["/app/main"]

And the Docker Compose file is equally straightforward:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

version: "3"
services:
  main:
    build:
      context: .
      dockerfile: Dockerfile
    restart: always
    image: main:latest
    deploy:
      replicas: 3

Monitoring, Alerting, and Other Services

Beyond the main website and blog, several other services are running on the server. To maintain stability, these need to be monitored and any problems flagged. While a personal website does not come with an SLA, having services down for days at a time is a poor experience for visitors. Personal monitoring can be done very crudely — a crontab that keeps curling an endpoint and fires off an email script when the endpoint is unavailable — or very elaborately, with Prometheus, Grafana, CPU and memory usage dashboards, and so on. Both are viable, but from long experience, only things you write yourself remain maintainable and compatible over the long term.

So as part of this upgrade I wrote a monitoring service for changkun.de called upbot, integrated with the Mailgun email service. It pings each service at a fixed interval to check availability, and sends me an email alert when something goes down.

Alongside the monitoring service, changkun.de also runs several services I use personally:

• midgard: a cross-platform clipboard service that automatically syncs the clipboard between Linux, Mac, and iOS, and can quickly create a public link for clipboard contents or local files (built out of personal need, though setting it up is somewhat involved — I should record a usage tutorial some day, Orz)
• occamy: a universal remote desktop service supporting VNC/RDP/SSH protocol translation
• redir: a URL shortener with PV/UV analytics

Space does not permit going into detail here; we will cover them another time. Interested readers can find the links at the end of this post.

There are also a few services running on the server that I use privately without making public — for instance, a Jupyter server. In the old days these were either spun up with python3 -m http.server directly inside a tmux session, or kept alive with a hypervisor (deployment method depending on mood at the time). This migration has cleaned all of that up and containerised everything properly.

Finally, with this many services running on one server, a reverse proxy is essential. I could write one myself — Go’s standard library does provide httputil.ReverseProxy — but building a reasonably reliable, feature-complete reverse proxy is considerably more complex than writing a simple monitoring service (at minimum it would need request cancellation, retries, caching, and logging). From the ancient Apache, to the third-age Nginx, and now aiming for at least a Go-written solution, Traefik — which grew out of the cloud-native ecosystem — is the obvious choice. It is extremely easy to use; reading the documentation is enough to get started.

Retrospective

This post has traced the migration history of changkun.de as a personal website, the problems and decisions encountered at each step, and the most recent “architecture upgrade” those decisions motivated. It also introduced the services that were built in-house to support the migration and are now running in production. The highlight of this migration is that the entire website now runs entirely on Go, achieving microservices in a meaningful sense. As a bonus, the post also covered how to use Go 1.16’s embedded files feature to bundle a site’s static files directly into the distributable binary.

The overall structure of changkun.de as it stands today is shown in the diagram below:

All of these services are written in Go and open source. Interested readers can find them at the following links:

• https://changkun.de/s/main
• https://changkun.de/s/blog
• https://changkun.de/s/midgard
• https://changkun.de/s/redir
• https://changkun.de/s/upbot
• https://changkun.de/s/occamy
• https://changkun.de/s/modern-cpp-tutorial (this one is an exception :)

In addition, there are services running on golang.design:

• https://changkun.de/s/go-under-the-hood
• https://changkun.de/s/go-history
• https://changkun.de/s/gossa
• https://changkun.de/s/code2img

They were once hosted on changkun.de and can be thought of as projects that incubated there. The Third Age design lasted around three years; I hope this new “architecture” — backed by Go’s stability and compatibility — will serve reliably for more than three years to come.

Changkun Ou

24 December 2020, Munich