最近收到 Hyperledger 社区的一些朋友反馈在阿里云环境上部署开源区块链项目 Hyperledger Fabric的过程中遇到了和 SIGSEV 相关的 fatal error，正好笔者此前也遇到并解决过类似的问题，因此这里分享一下当时问题的分析过程和解决的经验，希望能为大家带来一点启发和帮助。
问题描述
在部署Hyperledger Fabric 过程中，peer、orderer 服务启动失败，同时 cli 容器上执行 cli-test.sh 测试时也报错。错误类型均是 signal SIGSEGV: segmentation violation。错误日志示例如下：
2017-11-01 02:44:04.247 UTC [peer] updateTrustedRoots -> DEBU 2a0 Updating trusted root authorities for channel mychannel
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x63 pc=0x7f9d15ded259]
runtime stack:
runtime.throw(0xdc37a7, 0x2a)
/opt/go/src/runtime/panic.go:566 +0x95
runtime.sigpanic()
/opt/go/src/runtime/sigpanic_unix.go:12 +0x2cc
goroutine 64 [syscall, locked to thread]:
runtime.cgocall(0xb08d50, 0xc4203bcdf8, 0xc400000000)
/opt/go/src/runtime/cgocall.go:131 +0x110 fp=0xc4203bcdb0 sp=0xc4203bcd70
net._C2func_getaddrinfo(0x7f9d000008c0, 0x0, 0xc420323110, 0xc4201a01e8, 0x0, 0x0, 0x0)分析过程
我们进行了深入分析和试验，在 Hyperledger Fabric这个 bug https://jira.hyperledger.org/browse/FAB-5822的启发下，采用了如下 workaround 可以解决这个问题：
- 在 docker compose yaml 里对 peer、orderer、cli 的环境变量加入 GODEBUG=netdns=go
这个设置的作用是不采用 cgo resolver （从错误日志里可看到是 cgo resolver 抛出的错误）而采用 pure go resolver。
进一步分析 golang 在什么情况下会在 cgo resolver 和 pure go resolver 之间切换：
- golang 的官方文档说明：https://golang.org/pkg/net/
Name Resolution
The method for resolving domain names, whether indirectly with functions like Dial or directly with functions like LookupHost and LookupAddr, varies by operating system.
On Unix systems, the resolver has two options for resolving names. It can use a pure Go resolver that sends DNS requests directly to the servers listed in /etc/resolv.conf, or it can use a cgo-based resolver that calls C library routines such as getaddrinfo and getnameinfo.
By default the pure Go resolver is used, because a blocked DNS request consumes only a goroutine, while a blocked C call consumes an operating system thread. When cgo is available, the cgo-based resolver is used instead under a variety of conditions: on systems that do not let programs make direct DNS requests (OS X), when the LOCALDOMAIN environment variable is present (even if empty), when the RES_OPTIONS or HOSTALIASES environment variable is non-empty, when the ASR_CONFIG environment variable is non-empty (OpenBSD only), when /etc/resolv.conf or /etc/nsswitch.conf specify the use of features that the Go resolver does not implement, and when the name being looked up ends in .local or is an mDNS name.
The resolver decision can be overridden by setting the netdns value of the GODEBUG environment variable (see package runtime) to go or cgo, as in:
export GODEBUG=netdns=go # force pure Go resolver
export GODEBUG=netdns=cgo # force cgo resolver*
根据这一线索，我们对比了此前部署成功环境和最近部署失败环境各自的底层配置文件，最终找到了不同之处：
- 在老环境（区块链部署成功)上的容器里，查看
# cat /etc/resolv.conf nameserver 127.0.0.11 options ndots:0 - 在新环境（区块链部署失败）上的容器里，查看
# cat /etc/resolv.conf nameserver 127.0.0.11 options timeout:2 attempts:3 rotate single-request-reopen ndots:0
这个差异导致了老的成功环境是采用 pure Go resolver 的，而在新的失败环境被切换到 cgo resolver， 这是因为含有 pure Go resolver 不支持的 options single-request-reopen。
注：Pure Go resolver 目前仅支持 ndots, timeout, attempts, rotate
https://github.com/golang/go/blob/964639cc338db650ccadeafb7424bc8ebb2c0f6c/src/net/dnsconfig_unix.go
case "options": // magic options
for _, s := range f[1:] {
switch {
case hasPrefix(s, "ndots:"):
n, _, _ := dtoi(s[6:])
if n < 0 {
n = 0
} else if n > 15 {
n = 15
}
conf.ndots = n
case hasPrefix(s, "timeout:"):
n, _, _ := dtoi(s[8:])
if n < 1 {
n = 1
}
conf.timeout = time.Duration(n) * time.Second
case hasPrefix(s, "attempts:"):
n, _, _ := dtoi(s[9:])
if n < 1 {
n = 1
}
conf.attempts = n
case s == "rotate":
conf.rotate = true
default:
conf.unknownOpt = true
}
}进一步的，我们尝试分析是什么原因导致了新老容器内的 resolv.conf 的内容变化，发现了原来是最近宿主机 ECS 的配置文件发生了变化：
- 失败的环境 – 新创建的 ECS:
# cat /etc/resolv.conf # Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8) # DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN nameserver 100.100.2.138 nameserver 100.100.2.136 options timeout:2 attempts:3 rotate single-request-reopen - 成功的环境 – 原来的 ECS:
# cat /etc/resolv.conf # Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8) # DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN nameserver 100.100.2.136 nameserver 100.100.2.138
另一方面，我们也尝试分析为什么切换到 cgo resolver 之后会产生SIGSEGV的错误，以下这篇文章解释了 static link cgo 会导致SIGSEGV的错误：
http://tbg.github.io/golang-static-linking-bug
而这个 Hyperledger Fabric的 bug 则指出了 Hyperledger Fabric的 build（尤其是和 getaddrinfo 相关方法）正是 static link 的：
https://jira.hyperledger.org/browse/FAB-6403
至此，我们找到了问题的根源和复盘了整个问题发生的逻辑：
- 近期新创建的 ECS 主机中的 resolv.conf 内容发生了变化 -> 导致 Hyperledger Fabric的容器内域名解析从 pure Go resolver 切换至 cgo resolver -> 触发了一个已知的由静态链接 cgo 导致的SIGSEGV错误 -> 导致 Hyperledger Fabric 部署失败。
解决方法建议
更新 Hyperledger Fabric 的 docker compose yaml 模板，为所有 Hyperledger Fabric 的节点（如 orderer, peer, ca, cli 等）添加环境变量 GODEBUG=netdns=go 以强制使用 pure Go resolver。
