Changkun's Blog欧长坤的博客

Science and art, life in between.科学与艺术,生活在其间。

  • Home首页
  • Ideas想法
  • Posts文章
  • Tags标签
  • Bio关于
Changkun Ou

Changkun Ou

Human-AI interaction researcher, engineer, and writer.人机交互研究者、工程师、写作者。

Bridging HCI, AI, and systems programming. Building intelligent human-in-the-loop optimization systems. Informed by psychology, sociology, cognitive science, and philosophy.连接人机交互、AI 与系统编程。构建智能的人在环优化系统。融合心理学、社会学、认知科学与哲学。

Science and art, life in between.科学与艺术,生活在其间。

281 Blogs博客
168 Tags标签
Changkun's Blog欧长坤的博客

Are PSS/USS and RSS Actually the Same Thing?PSS/USS 和 RSS 其实是一回事,吗?

Published at发布于:: 2021-01-23   |   Reading阅读:: 15 min

Since Go 1.12, there have been a steady stream of false-alarm monitoring incidents. The root cause is that starting with Go 1.12, Go changed the memory reclamation strategy used in madvise system calls from MADV_DONTNEED to MADV_FREE. Based on available documentation, RSS — the most commonly used …

从 Go 1.12 开始就不断有人踩到监控误报的坑,原因是 Go 从 1.12 开始将 madvise 系统调用 使用的内存回收策略从 MADV_DONTNEED 改为了 MADV_FREE。 从可查的一些文档来看,RSS 作为最常用的内存监控指标,不会反映进程中未被操作系统回收的那部分内存。 自然就会有一些说法建议将 RSS 更换为可能更妥当的指标,比如 PSS 甚至 USS。 这就导致了一些比 …

Read More阅读更多 »

Performance Differences from Page Faults vs. Prefetching缺页与预取带来的性能差异

Published at发布于:: 2021-01-18   |   Reading阅读:: 11 min

Just how large can the performance difference caused by page faults be? Let’s write a benchmark to find out. Simulating Page Fault Behavior To build this benchmark, we need to understand two low-level Linux syscalls for memory management: mmap and madvise. For memory allocation, mmap can be …

缺页错误产生的性能差异究竟能够有多大?不妨做一个基准测试。 模拟缺页行为 想要实现这样的基准测试,需要了解 Linux 下对内存管理的两个底层的系统调用:mmap 和 madvise。 对于内存分配场景,mmap 可以使用匿名、私有映射两个参数 MAP_ANON 和 MAP_PRIVATE, 这时候创建的内存实际上属于缺页状态,任何对其申请到内存区域的访问行为都将导致缺页,利用这一原理, 便可以 …

Read More阅读更多 »
idea想法 2021-01-18 00:00:00

Daily Reading每日阅读

Read these articles:

  • What to expect when monitoring memory usage for modern Go applications. https://www.bwplotka.dev/2019/golang-memory-monitoring/
  • Distributed Systems. https://www.youtube.com/watch?v=UEAMfLPZZhE
  • Golang News. https://www.golangnews.com/
  • SIGCHI Symposium on Engineering Interactive Computing Systems. http://eics.acm.org/
  • runtime: use MADV_FREE on Linux if available. https://go-review.googlesource.com/c/go/+/135395/
  • runtime: make the page allocator scale. https://github.com/golang/go/issues/35112
  • runtime: add per-p mspan cache. https://go-review.googlesource.com/c/go/+/196642
  • A New Smoothing Algorithm for Quadrilateral and Hexahedral Meshes. https://link.springer.com/content/pdf/10.1007%2F11758525_32.pdf
  • OpenGL Docs. https://docs.gl
  • On Playing Chess. https://blog.gardeviance.org/2018/03/on-playing-chess.html
  • Memory Models: A Case For Rethinking Parallel Languages and Hardware. https://cacm.acm.org/magazines/2010/8/96610-memory-models-a-case-for-rethinking-parallel-languages-and-hardware/fulltext
  • Engineer level & competency framework. https://github.com/spring2go/engineer_competency_framework
  • A Concurrent Window System. http://doc.cat-v.org/bell_labs/concurrent_window_system/concurrent_window_system.pdf

读了这些文章:

  • 监控现代 Go 应用内存使用时的注意事项。https://www.bwplotka.dev/2019/golang-memory-monitoring/
  • 分布式系统。https://www.youtube.com/watch?v=UEAMfLPZZhE
  • Golang 新闻。https://www.golangnews.com/
  • SIGCHI 交互式计算系统工程研讨会。http://eics.acm.org/
  • runtime:在 Linux 上使用 MADV_FREE(如果可用)。https://go-review.googlesource.com/c/go/+/135395/
  • runtime:使页分配器可扩展。https://github.com/golang/go/issues/35112
  • runtime:添加 per-p mspan 缓存。https://go-review.googlesource.com/c/go/+/196642
  • 一种用于四边形和六面体网格的新平滑算法。https://link.springer.com/content/pdf/10.1007%2F11758525_32.pdf
  • OpenGL 文档。https://docs.gl
  • 论下棋。https://blog.gardeviance.org/2018/03/on-playing-chess.html
  • 内存模型:重新思考并行语言和硬件。https://cacm.acm.org/magazines/2010/8/96610-memory-models-a-case-for-rethinking-parallel-languages-and-hardware/fulltext
  • 工程师等级与能力框架。https://github.com/spring2go/engineer_competency_framework
  • 一个并发窗口系统。http://doc.cat-v.org/bell_labs/concurrent_window_system/concurrent_window_system.pdf
idea想法 2021-01-06 00:00:00

Creating A Window创建一个窗口

How to create a window using Go? macOS has Cocoa, Linux has X11, but accessing these APIs seems to require Cgo. Is it possible to avoid Cgo? Some existing GUI libraries and graphics engines:

GUI Toolkits:

  • https://github.com/hajimehoshi/ebiten
  • https://github.com/gioui/gio
  • https://github.com/fyne-io/fyne
  • https://github.com/g3n/engine
  • https://github.com/goki/gi
  • https://github.com/peterhellberg/gfx
  • https://golang.org/x/exp/shiny

2D/3D Graphics:

  • https://github.com/llgcode/draw2d
  • https://github.com/fogleman/gg
  • https://github.com/ajstarks/svgo
  • https://github.com/BurntSushi/graphics-go
  • https://github.com/azul3d/engine
  • https://github.com/KorokEngine/Korok
  • https://github.com/EngoEngine/engo/
  • http://mumax.github.io/

Here is a partial list:

https://github.com/avelino/awesome-go#gui

Most people use glfw and OpenGL. Here are some required libraries (Cgo bindings):

  • https://github.com/go-gl/gl
  • https://github.com/go-gl/glfw
  • https://github.com/remogatto/egl

Some more low-level ones, e.g. X-related:

  • X bindings: https://github.com/BurntSushi/xgb
  • X window management: https://github.com/BurntSushi/wingo

For example, if you need Metal on macOS:

  • https://dmitri.shuralyov.com/gpu/mtl

Like the GUI tool ebiten mentioned above, on Windows it no longer needs Cgo. The approach seems to be packaging the window management DLLs directly into the binary and then using DLL dynamic linking calls.

Besides GLFW, there is the heavier SDL:

  • https://github.com/veandco/go-sdl2

Relationships between some basic terms:

1
2
3
4
5
6
7
Unix:      The ancestor
BSD:       Unix-like, Berkeley distribution, one of the two traditional flavors of Unix
System V:  Developed by AT&T, one of the two traditional flavors of Unix
Linux:     A fresh flavor of Unix-like OS
POSIX:     A standard attempting to reduce implementation differences
Darwin:    Apple's open-source Unix-like kernel XNU
Mach-O:    Microkernel hybridized in Darwin, developed by CMU

By Eraserhead1, Infinity0, Sav_vas - Levenez Unix History Diagram, Information on the history of IBM's AIX on ibm.com, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=1801948

Relationships between some Wayland-related tools:

1
2
3
4
5
6
X Window System == X11: Display standard developed by MIT, the foundation for GNOME and KDE
Xorg:    The official implementation of X11
Wayland: An alternative communication protocol between display server and client, distinct from X11
EGL:     Wayland clients use EGL to directly manipulate the framebuffer
libDRM:  Kernel API exposed to userspace, underlying dependency of EGL/X11
DRM:     Kernel-level component for manipulating the framebuffer

By Shmuel Csaba Otto Traian, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=28029855

By Shmuel Csaba Otto Traian, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=31768083

By Shmuel Csaba Otto Traian, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=27858390

By Shmuel Csaba Otto Traian,CC BY-SA 3.0,https://commons.wikimedia.org/w/index.php?curid=27799196

So through DRM you can directly operate on the Frame Buffer under Linux, i.e., Direct Rendering Manager. Related libraries:

  • https://github.com/NeowayLabs/drm

With this, you can do pure Go rendering directly on Linux, eliminating the dependency on C legacy.

如何使用 Go 创建一个窗口?macOS 有 Cocoa、Linux 有 X11,但访问这些 API 似乎都需要 引入 Cgo,可不可以不实用 Cgo?一些现有的 GUI 库或这图形引擎:

GUI 工具包:

  • https://github.com/hajimehoshi/ebiten
  • https://github.com/gioui/gio
  • https://github.com/fyne-io/fyne
  • https://github.com/g3n/engine
  • https://github.com/goki/gi
  • https://github.com/peterhellberg/gfx
  • https://golang.org/x/exp/shiny

2D/3D 图形相关:

  • https://github.com/llgcode/draw2d
  • https://github.com/fogleman/gg
  • https://github.com/ajstarks/svgo
  • https://github.com/BurntSushi/graphics-go
  • https://github.com/azul3d/engine
  • https://github.com/KorokEngine/Korok
  • https://github.com/EngoEngine/engo/
  • http://mumax.github.io/

这里有一小部分:

https://github.com/avelino/awesome-go#gui

大部分人的做法是使用 glfw 和 OpenGL,这是一些需要使用到的库(Cgo 绑定):

  • https://github.com/go-gl/gl
  • https://github.com/go-gl/glfw
  • https://github.com/remogatto/egl

这里面有一些相对底层一些的,比如 X 相关:

  • X 绑定:https://github.com/BurntSushi/xgb
  • X 窗口管理:https://github.com/BurntSushi/wingo

比如 macOS 上如果需要用到 Metal:

  • https://dmitri.shuralyov.com/gpu/mtl

像是前面的 GUI 工具中的 ebiten,在 windows 上已经不需要 Cgo 了,做法似乎是将窗口管理相关的 DLL 直接打包进二进制,然后走 DLL 动态链接调用。

除了 GLFW 之外,还有相对重一些的 SDL:

  • https://github.com/veandco/go-sdl2

一些基本名词之间的关系:

1
2
3
4
5
6
7
Unix:      上帝
BSD:       类 Unix,伯克利分发,两种传统风味的 Unix 之一
System V:  AT&T 开发,两种传统风味的 Unix 之一
Linux:     新鲜风味的类 Unix
POSIX:     尝试减少实现差异的标准
Darwin:    苹果的开源类 Unix 内核 XNU
Mach-O:    Darwin 混合的微内核,CMU 开发

By Eraserhead1, Infinity0, Sav_vas - Levenez Unix History Diagram, Information on the history of IBM's AIX on ibm.com,CC BY-SA 3.0,https://commons.wikimedia.org/w/index.php?curid=1801948

关于 Wayland 的一些工具之间的关系:

1
2
3
4
5
6
X Window System == X11: MIT 开发的显示标准,GNOME、KDE 依赖的基础
Xorg:    X11 的官方实现
Wayland: 另一种显示服务器和客户端之间的通信协议,区别于 X11
EGL:     Wayland 客户端使用 EGL 来直接操作 framebuffer
libDRM:  EGL/X11 底层依赖的内核对 userspace 开放的 API
DRM:     内核级操作 framebuffer 的组件

By Shmuel Csaba Otto Traian, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=28029855

By Shmuel Csaba Otto Traian, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=31768083

By Shmuel Csaba Otto Traian, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=27858390

By Shmuel Csaba Otto Traian,CC BY-SA 3.0,https://commons.wikimedia.org/w/index.php?curid=27799196

所以通过 DRM 可以在 Linux 下直接操作 Frame Buffer 上,也就是 Direct Rendering Manager。相关的库有:

  • https://github.com/NeowayLabs/drm

有了这个就可以在 Linux 上直接做纯 Go 的绘制了,从而消除对 C 遗产的依赖。

idea想法 2021-01-05 00:00:00

Daily Reading每日阅读

Read these articles:

  • The Right to Read. https://www.gnu.org/philosophy/right-to-read.en.html
  • Your Computer Isn’t Yours. https://sneak.berlin/20201112/your-computer-isnt-yours/
  • Pirate Cinema. https://craphound.com/pc/download/

Noticed these two clipboard-related projects:

  • https://github.com/binwiederhier/pcopy
  • https://github.com/nakabonne/pbgopy

It seems their development timelines are very close to when I developed midgard, but we all took very different paths:

  • pcopy focuses on the clipboard itself, with features like password protection, multiple clipboards, WebUI, etc.
  • pbgopy emphasizes security for cross-device syncing, with various key configurations, but very simple functionality — only supports text

The midgard I developed focuses on these features:

  • Automatic cross-device syncing (clipboard write-back), no commands needed
  • Ability to query devices currently online
  • Supports not only text but also image syncing
  • Supports creating public URLs for clipboard content
  • Supports converting code in clipboard to Carbon images
  • Supports keyboard shortcuts
  • …

读了这几篇文章:

  • 阅读的权利。https://www.gnu.org/philosophy/right-to-read.en.html
  • 你的电脑不属于你。https://sneak.berlin/20201112/your-computer-isnt-yours/
  • 盗版电影院。https://craphound.com/pc/download/

注意到了这两个跟剪贴板相关的项目:

  • https://github.com/binwiederhier/pcopy
  • https://github.com/nakabonne/pbgopy

看起来他们开发这两个项目的时间跟我开发 midgard 的时间非常接近,不过大家都走了很不同的路线:

  • pcopy 着重剪贴板本身,比如有剪贴板的密码保护,多种剪贴板、WebUI 等特性
  • pggopy 注重设备间同步的安全性,有各种密钥配置,但功能非常简单,只支持文本

我开发的 midgard 的则主打这些特性:

  • 多设备间自动同步(剪贴板回写),不需要敲命令
  • 能够查询同时在线的设备
  • 不仅支持文本,还支持图片的同步
  • 支持对剪贴板中的内容创建公开的 URL
  • 支持剪贴板中的代码转 Carbon 图片
  • 支持键盘快捷键
  • …

2020 Year-End Review2020 年终总结

Published at发布于:: 2021-01-03   |   Reading阅读:: 22 min

2020 is finally, completely over. As I have done every year since undergraduate, I open my blog and sit down to write this annual reflection. So much seemed to happen this year — I was still in the office working overtime on the very last day of it. If I had to name the single greatest takeaway from …

2020 年算是彻底结束了,像往常一样,我又打开了自己的博客,开始写下这篇我从本科开始就 坚持年更类型的文章。这一年里看似发生了许多,以至于在一年的最后一天我还在办公室里加班。 要说这一年里最大的收获是什么,可能会总结为两个字:转变。 到了博士阶段,也就不能像是本科、研究生那样只扩宽自己技能的广度,需要更多的专注在如何将自身 的能力做深度的培养(尽管学习的时候是系统性的学习,但如果没有长期实践这 …

Read More阅读更多 »
idea想法 2020-12-31 00:00:00

Daily Reading每日阅读

Read these articles:

  • A History of the GUI. http://www.cdpa.co.uk/UoP/Found/Downloads/reading6.pdf, https://www.readit-dtp.de/PDF/gui_history.pdf
  • History of the graphical user interface. https://en.wikipedia.org/wiki/History_of_the_graphical_user_interface
  • Algebraic Effects for the Rest of Us. https://overreacted.io/zh-hans/algebraic-effects-for-the-rest-of-us/
  • What does algebraic effects mean in FP? https://stackoverflow.com/questions/49626714/what-does-algebraic-effects-mean-in-fp
  • A Distributed Systems Reading List. https://dancres.github.io/Pages/
  • Tutte Embedding for Parameterization. https://isaacguan.github.io/2018/03/19/Tutte-Embedding-for-Parameterization/
  • Commentary on the Sixth Edition UNIX Operating System. http://www.lemis.com/grog/Documentation/Lions/index.php
  • Personal knowledge management beyond versioning. https://dl.acm.org/doi/10.1145/2362456.2362492
  • The Plain Text Life: Note Taking, Writing and Life Organization Using Plain Text Files. http://www.markwk.com/plain-text-life.html
  • Post-Evernote: How to Migrate Your Evernote Notes, Images and Tags Into Plain Text Markdown. http://www.markwk.com/migrate-evernote-plaintext.html
  • Five Levels of Error Handling in Both Python and JavaScript. https://dev.to/jesterxl/five-levels-of-error-handling-in-both-python-and-javascript-13ok

读了这些文章:

  • GUI 的历史。http://www.cdpa.co.uk/UoP/Found/Downloads/reading6.pdf, https://www.readit-dtp.de/PDF/gui_history.pdf
  • 图形用户界面的历史。https://en.wikipedia.org/wiki/History_of_the_graphical_user_interface
  • 写给普通人的代数效应。https://overreacted.io/zh-hans/algebraic-effects-for-the-rest-of-us/
  • 代数效应在函数式编程中意味着什么?https://stackoverflow.com/questions/49626714/what-does-algebraic-effects-mean-in-fp
  • 分布式系统阅读清单。https://dancres.github.io/Pages/
  • 用于参数化的 Tutte 嵌入。https://isaacguan.github.io/2018/03/19/Tutte-Embedding-for-Parameterization/
  • 第六版 UNIX 操作系统注释。http://www.lemis.com/grog/Documentation/Lions/index.php
  • 超越版本控制的个人知识管理。https://dl.acm.org/doi/10.1145/2362456.2362492
  • 纯文本生活:使用纯文本文件进行笔记、写作和生活管理。http://www.markwk.com/plain-text-life.html
  • 后 Evernote 时代:如何将 Evernote 笔记、图片和标签迁移到纯文本 Markdown。http://www.markwk.com/migrate-evernote-plaintext.html
  • Python 和 JavaScript 中错误处理的五个层次。https://dev.to/jesterxl/five-levels-of-error-handling-in-both-python-and-javascript-13ok
idea想法 2020-12-30 00:00:00

Daily Reading每日阅读

Read these articles:

  • The UNIXHATERS Handbook. http://web.mit.edu/~simsong/www/ugh.pdf
  • Why are video games graphics (still) a challenge? Productionizing rendering algorithms. https://bartwronski.com/2020/12/27/why-are-video-games-graphics-still-a-challenge-productionizing-rendering-algorithms/
  • BPF and Go: Modern forms of introspection in Linux. https://medium.com/bumble-tech/bpf-and-go-modern-forms-of-introspection-in-linux-6b9802682223
  • Systems design explains the world: volume 1. https://apenwarr.ca/log/20201227
  • Error handling guidelines for Go. https://jayconrod.com/posts/116/error-handling-guidelines-for-go
  • The Missing Semester of Your CS Education. https://missing.csail.mit.edu/
  • Build your own React. https://pomb.us/build-your-own-react/

读了这些文章:

  • UNIX 憎恨者手册。http://web.mit.edu/~simsong/www/ugh.pdf
  • 为什么电子游戏图形(仍然)是一个挑战?渲染算法的产品化。https://bartwronski.com/2020/12/27/why-are-video-games-graphics-still-a-challenge-productionizing-rendering-algorithms/
  • BPF 与 Go:Linux 中的现代内省方法。https://medium.com/bumble-tech/bpf-and-go-modern-forms-of-introspection-in-linux-6b9802682223
  • 系统设计解释世界:第一卷。https://apenwarr.ca/log/20201227
  • Go 错误处理指南。https://jayconrod.com/posts/116/error-handling-guidelines-for-go
  • 计算机科学教育中缺失的一课。https://missing.csail.mit.edu/
  • 构建你自己的 React。https://pomb.us/build-your-own-react/

Migration with Zero Downtime零宕机迁移

Published at发布于:: 2020-12-28   |   Reading阅读:: 12 min

In this post I gave an overview of the restructured architecture of changkun.de, but I didn’t go into detail about how the migration itself was carried out — whether there was any downtime, and so on. This time, let’s take a closer look at the migration process. Migration Checklist …

在这篇文章 中我介绍了 changkun.de 重新调整后的一个整体的架构, 但并没有仔细的介绍进行架构升级的过程是怎样的、升级过程中是否有进行停机等等。 这次我们就来简单聊一聊这个迁移过程。 迁移清单 首先,changkun.de 服务器上运行了诸多服务,比如使用频繁的有 redir、midgard 等等,如图所示: 这些服务包括: https://changkun.de/s/main …

Read More阅读更多 »
idea想法 2020-12-27 00:00:00

Concurrency Patterns并发模式

Fan-In multiplexes multiple input channels onto one output channel.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
func Funnel(sources ...<-chan int) <-chan int {
    dest := make(chan int)                  // The shared output channel

    var wg sync.WaitGroup                   // Used to automatically close dest
                                            // when all sources are closed

    wg.Add(len(sources))                    // Increment the sync.WaitGroup

    for _, ch := range sources {            // Start a goroutine for each source
        go func(c <-chan int) {
            defer wg.Done()                 // Notify WaitGroup when c closes

            for n := range c {
                dest <- n
            }
        }(ch)
    }

    go func() {                             // Start a goroutine to close dest
        wg.Wait()                           // after all sources close
        close(dest)
    }()

    return dest
}

Fan-Out evenly distributes messages from an input channel to multiple output channels.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
func Split(source <-chan int, n int) []<-chan int {
    dests := make([]<-chan int, 0)          // Create the dests slice

    for i := 0; i < n; i++ {                // Create n destination channels
        ch := make(chan int)
        dests = append(dests, ch)

        go func() {                         // Each channel gets a dedicated
            defer close(ch)                 // goroutine that competes for reads

            for val := range source {
                ch <- val
            }
        }()
    }

    return dests
}

Future provides a placeholder for a value that’s not yet known.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
type Future interface {
    Result() (string, error)
}

type InnerFuture struct {
    once sync.Once
    wg   sync.WaitGroup

    res   string
    err   error
    resCh <-chan string
    errCh <-chan error
}

func (f *InnerFuture) Result() (string, error) {
    f.once.Do(func() {
        f.wg.Add(1)
        defer f.wg.Done()
        f.res = <-f.resCh
        f.err = <-f.errCh
    })

    f.wg.Wait()

    return f.res, f.err
}

func SlowFunction(ctx context.Context) Future {
    resCh := make(chan string)
    errCh := make(chan error)

    go func() {
        select {
        case <-time.After(time.Second * 2):
            resCh <- "I slept for 2 seconds"
            errCh <- nil
        case <-ctx.Done():
            resCh <- ""
            errCh <- ctx.Err()
        }
    }()

    return &InnerFuture{resCh: resCh, errCh: errCh}
}

Sharding splits a large data structure into multiple partitions to localize the effects of read/write locks.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
type Shard struct {
    sync.RWMutex                            // Compose from sync.RWMutex
    m map[string]interface{}                // m contains the shard's data
}

type ShardedMap []*Shard                    // ShardedMap is a *Shards slice

func NewShardedMap(nshards int) ShardedMap {
    shards := make([]*Shard, nshards)       // Initialize a *Shards slice

    for i := 0; i < nshards; i++ {
        shard := make(map[string]interface{})
        shards[i] = &Shard{m: shard}
    }

    return shards                           // A ShardedMap IS a *Shards slice!
}

func (m ShardedMap) getShardIndex(key string) int {
    checksum := sha1.Sum([]byte(key))       // Use Sum from "crypto/sha1"
    hash := int(checksum[17])               // Pick a random byte as our hash
    index := hash % len(shards)             // Mod by len(shards) to get index
}

func (m ShardedMap) getShard(key string) *Shard {
    index := m.getShardIndex(key)
    return m[index]
}

func (m ShardedMap) Get(key string) interface{} {
    shard := m.getShard(key)
    shard.RLock()
    defer shard.RUnlock()

    return shard.m[key]
}

func (m ShardedMap) Set(key string, value interface{}) {
    shard := m.getShard(key)
    shard.Lock()
    defer shard.Unlock()

    shard.m[key] = value
}

func (m ShardedMap) Keys() []string {
    keys := make([]string, 0)               // Create an empty keys slice
    wg := sync.WaitGroup{}                  // Create a wait group and add a
    wg.Add(len(m))                          // wait value for each slice
    for _, shard := range m {               // Run a goroutine for each slice
        go func(s *Shard) {
            s.RLock()                       // Establish a read lock on s
            for key, _ := range s.m {       // Get the slice's keys
                keys = append(keys, key)
            }
            s.RUnlock()                     // Release the read lock
            wg.Done()                       // Tell the WaitGroup it's done
        }(shard)
    }
    wg.Wait()                               // Block until all reads are done
    return keys                             // Return combined keys slice
}

扇入(Fan-In) 将多个输入通道多路复用到一个输出通道上。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
func Funnel(sources ...<-chan int) <-chan int {
    dest := make(chan int)                  // The shared output channel

    var wg sync.WaitGroup                   // Used to automatically close dest
                                            // when all sources are closed

    wg.Add(len(sources))                    // Increment the sync.WaitGroup

    for _, ch := range sources {            // Start a goroutine for each source
        go func(c <-chan int) {
            defer wg.Done()                 // Notify WaitGroup when c closes

            for n := range c {
                dest <- n
            }
        }(ch)
    }

    go func() {                             // Start a goroutine to close dest
        wg.Wait()                           // after all sources close
        close(dest)
    }()

    return dest
}

扇出(Fan-Out) 将输入通道的消息均匀分发到多个输出通道。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
func Split(source <-chan int, n int) []<-chan int {
    dests := make([]<-chan int, 0)          // Create the dests slice

    for i := 0; i < n; i++ {                // Create n destination channels
        ch := make(chan int)
        dests = append(dests, ch)

        go func() {                         // Each channel gets a dedicated
            defer close(ch)                 // goroutine that competes for reads

            for val := range source {
                ch <- val
            }
        }()
    }

    return dests
}

Future 为尚未确定的值提供一个占位符。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
type Future interface {
    Result() (string, error)
}

type InnerFuture struct {
    once sync.Once
    wg   sync.WaitGroup

    res   string
    err   error
    resCh <-chan string
    errCh <-chan error
}

func (f *InnerFuture) Result() (string, error) {
    f.once.Do(func() {
        f.wg.Add(1)
        defer f.wg.Done()
        f.res = <-f.resCh
        f.err = <-f.errCh
    })

    f.wg.Wait()

    return f.res, f.err
}

func SlowFunction(ctx context.Context) Future {
    resCh := make(chan string)
    errCh := make(chan error)

    go func() {
        select {
        case <-time.After(time.Second * 2):
            resCh <- "I slept for 2 seconds"
            errCh <- nil
        case <-ctx.Done():
            resCh <- ""
            errCh <- ctx.Err()
        }
    }()

    return &InnerFuture{resCh: resCh, errCh: errCh}
}

分片(Sharding) 将大型数据结构拆分为多个分区,以局部化读写锁的影响。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
type Shard struct {
    sync.RWMutex                            // Compose from sync.RWMutex
    m map[string]interface{}                // m contains the shard's data
}

type ShardedMap []*Shard                    // ShardedMap is a *Shards slice

func NewShardedMap(nshards int) ShardedMap {
    shards := make([]*Shard, nshards)       // Initialize a *Shards slice

    for i := 0; i < nshards; i++ {
        shard := make(map[string]interface{})
        shards[i] = &Shard{m: shard}
    }

    return shards                           // A ShardedMap IS a *Shards slice!
}

func (m ShardedMap) getShardIndex(key string) int {
    checksum := sha1.Sum([]byte(key))       // Use Sum from "crypto/sha1"
    hash := int(checksum[17])               // Pick a random byte as our hash
    index := hash % len(shards)             // Mod by len(shards) to get index
}

func (m ShardedMap) getShard(key string) *Shard {
    index := m.getShardIndex(key)
    return m[index]
}

func (m ShardedMap) Get(key string) interface{} {
    shard := m.getShard(key)
    shard.RLock()
    defer shard.RUnlock()

    return shard.m[key]
}

func (m ShardedMap) Set(key string, value interface{}) {
    shard := m.getShard(key)
    shard.Lock()
    defer shard.Unlock()

    shard.m[key] = value
}

func (m ShardedMap) Keys() []string {
    keys := make([]string, 0)               // Create an empty keys slice
    wg := sync.WaitGroup{}                  // Create a wait group and add a
    wg.Add(len(m))                          // wait value for each slice
    for _, shard := range m {               // Run a goroutine for each slice
        go func(s *Shard) {
            s.RLock()                       // Establish a read lock on s
            for key, _ := range s.m {       // Get the slice's keys
                keys = append(keys, key)
            }
            s.RUnlock()                     // Release the read lock
            wg.Done()                       // Tell the WaitGroup it's done
        }(shard)
    }
    wg.Wait()                               // Block until all reads are done
    return keys                             // Return combined keys slice
}
2 3 4 5 6 7 8 9 10
© 2008 - 2026 Changkun Ou. All rights reserved.保留所有权利。 | PV/UV: /
0%