golang

Limited security support

The Go ecosystem heavily relies on static linking. To fix a CVE in a library, we need to:

  • Fix the library package,

  • AND identify reverse dependencies that actually use the affected code (not all rdeps),

  • AND rebuild the affected reverse dependencies,

which is currently impractical.

See wiki:StaticLinking for a general overview and other impacted ecosystems (Rust, Haskell, etc.).

See also this debian-devel 2014 thread where many common questions were already asked.

  • Debian Security position

The Debian infrastructure currently doesn’t properly enable rebuilding packages that statically link parts of other packages on a large scale. Until buster that hasn’t been a problem in practice, but with the growth of the Go ecosystem it means that Go based packages won’t be covered by regular security support until the infrastructure is improved to deal with them maintainably.

If updates are warranted, they can only come via regular point releases, which may be slow in arriving.

Libraries written in Go are packaged for Debian with the only purpose of building other Go programs for Debian. They are specifically not available for users in their regular development workflow. For that, users should use go get.

See also: stable update.

Current mitigation

Sponsors/customers are aware that these packages have limited support. They know that, depending on how difficult it is to rebuild dependencies, fixes may not be provided.

This is currently (2025-02) sustainable because there are not that many critical CVEs in golang packages, and not that many actual reverse dependencies to rebuild, so we can manually rebuild as needed. However this probably won’t hold forever, and doesn’t help supporting golang itself, see the next section for long-term plans.

Elements of decision/triage:

  • golang package: we may or may not fix CVEs depending on how many reverse dependencies we need to rebuild. See below to identify them.

  • golang itself: we cannot rebuild the whole golang ecosystem when fixing bugs in golang (too many packages to rebuild, no tooling to discriminate packages depending on standard library modules usage).

  • Backporting: if the Go packagers themselves fixed a CVE through a DSA or PU, it makes sense to follow suit, even if they didn’t rebuild reverse-dependencies

  • Up-porting: if an LTS/ELTS suite fixed a CVE, it makes sense to fix it in more recent suites, to avoid a confusing regression on upgrade.

General note: if the reverse-dependencies cannot be rebuilt, it makes sense not to fix a CVE:

  • the vulnerability status would be wrong (marked as fixed, but affected in practice)

  • this would be a time bomb, as regressions may surface later e.g. if a reverse-dependency is rebuilt one year later for a different reason

  • considering the packagers view above, and the relatively fast evolution of the language, the Go toolchain in Debian is aimed at building Debian dependencies more than compiling user/third-party programs; LTS users may still use LTS golang to rebuild old custom programs though

Releasing the update: reference rebuilt packages in the main DLA. This caused confusion for a user though.

TODO:

  • rebuild source dependencies in LTS and ELTS: which is better, source uploads or binNMUs? If binNMUs, how to request or implement them?

Future plans

  • Help improve the current Debian infrastructure to fix this issue by enabling mass/auto-rebuilding:

  • There was research into using dynamic linking in Go, but apparently that was limited to amd64, and the ABI broke too often, see https://wiki.debian.org/StaticLinking#Go. Note that ABI change is a problem for unstable, but not for a frozen dist, so this could be a solution for Stable and LTS/ELTS.

  • debusine could be improved to handle this when/if adopted to replace the current buildd infrastructure.

Identify reverse build dependencies

With dose-extra

Method: recursively analyzes the reverse build dependencies declared by the packages.

Limitation: Built-Depends and go itself may have different opinions on the dependencies list (field not up-to-date, testsuite-only dependency, dependency optimized out?…).

Limitation: it includes the golang-* dev packages (source-only/arch-all), which often only build binary -dev packages containing the sources, hence do not ship actual binaries and shouldn’t need a rebuild.

Set golang_binary_package to the package you want to find reverse build-dependencies for, e.g. golang-1.xx-go or golang-github-prometheus-client-golang-dev:

apt install dose-extra dctrl-tools

dist=bullseye
golang_binary_package=golang-github-hashicorp-go-retryablehttp-dev

dose-ceve --deb-native-arch=amd64 -r ${golang_binary_package} -T debsrc \
   debsrc://$(ls /var/lib/apt/lists/*_dists_${dist}_main_source_Sources) \
      deb://$(ls /var/lib/apt/lists/*_dists_${dist}_main_binary-amd64_Packages) \
 | grep-dctrl -n -s Package '' | sort -u

See https://manpages.debian.org/stable/dose-extra/dose-ceve.1.en.html#EXAMPLES

(For jessie, dose-ceve doesn’t seem to work at all; just import a jessie apt/lists/ in a more recent distro and run from there.)

ratt currently (202502) uses this method.

To better understand why dose-ceve returns a particular package, you can run the reverse operation (here with consul):

apt-rdepends -b consul > consul-bdeps.txt
# be patient, takes a couple minutes :)

With Static-Built-Using/Built-Using

Method: checks the Static-Built-Using (Built-Using prior 2022) field in the Packages file, which is generated by dh-golang, relying on go list, which should be recursive.

Limitation: misses Go packages with no/incomplete dh-golang support (e.g. versions of aptly < buster).

Limitation: not well supported/present in older LTS/ELTS releases.

Maintainer snippet:

apt-cache dumpavail | \
    grep-dctrl \
        -F Built-Using 'golang-1.7' -a \
        '(' --not -F Architecture all ')' \
        -s Source,Package,Version

Current status

Overall, it seems dose-ceve may suggest too many packages, and Built-Using too few.

Pending further insight, it is suggested to run both methods. Start with the Built-Using method, then check the dose-ceve result for packages that should have been included, especially for older (ELTS) dists.

Limitations / TODO:

  • How to find packages affected by a golang standard library, e.g. packages that use net/http or archive/zip and could be rebuilt following e.g. Debian 11.3/golang-1.15.

Examples

Example non-obvious affected packages (no Go dependencies in binary packages): heartbleader, toxiproxy

Example reverse-dependencies rebuilds:

  • DLA-4056-1 for golang-glog/bullseye, causing 4 rebuilds (source upload)

  • DLA-3455-1 for golang-go.crypto/buster, causing 18 rebuilds (source uploads)

  • Debian 10.3 rebuilds debos/1.0.0+git20190123.d6e16be-1+b1 for non-security issue (binNMU #946467)

  • DLA-2402-1 for golang-go.crypto/stretch, resulting in multiple subsequent DLAs: DLA-2442-1 (obfs4proxy), DLA-2453-1 (restic), DLA-2454-1 (rclone), DLA-2455-1 (packer), DLA-2527-1 (snapd)

  • Debian 9.13 includes numerous go-based packages rebuilds through binNMUs (not tracked in source package), e.g. heartbleeder/0.1.1-5+b3 and mongo-tools/3.2.11-1+b3

  • DLA-1664-1 for golang, causing heartbleeder/0.1.1-2+deb8u1 and aptly/0.8-3+deb8u1 (source uploads)

Building golang

Part of the golang-1.11 test suite requires the USER variable, which is removed by debuild as part of environment sanitization.

If you use debuild, make sure you work-around using:

debuild -e USER ...

There’s a work-around in 1.11.6-1+deb10u5, and upstream fixed at 3a18f0ecb5 (go1.12), but this may still be needed when recompiling old versions.

ARM64, ARMHF: building golang-1.11 with schroot on the porter boxes consistently triggers many errors in the test suites that never happen in the buildds (notably os/signal and TestGdb* from runtime).

ARMHFonARM64: some armhf (32-bit) buildds are running on actual 64-bit host architecture, which may trigger additional build failures, especially when the test is built-and-run with an empty Go cache. Beware when attempting to reproduce issues. See DLA 3395-2.

Run test suite

Run/re-run full test suite:

debian/rules override_dh_auto_test-arch
debian/rules override_dh_auto_test RUN_TEST=true  # jessie

Run a specific test:

debuild

# Simple case
cd src/pkg/net/url/
go test -v  # default to '.'
go test -v -run '^TestParse$' .

# Clean-up
# - Test results:
go clean -testcache
# - Force rebuild:
go clean -cache
# - In doubt:
rm -rf ~/.cache/go-build/
# - Run without caching results (cf. 'go help test')
go test -count=1 ...

# More complex case
cd /.../debian-source-packages/golang-1.x/src/  # src/pkg/ for jessie
rm -rf ../pkg/linux_*/  # Go reuses the .a files there
GOROOT=/.../debian-source-packages/golang-1.x/ PATH=../bin:$PATH go test -v ./net/http/     # not 'net/http/', this would check the system install
GOROOT=/.../debian-source-packages/golang-1.x/ PATH=../bin:$PATH go test -v ./net/http/...  # '...' means 'with subdirs'
GOROOT=/.../debian-source-packages/golang-1.x/ PATH=../bin:$PATH go test -v ./net/http/httputil/reverseproxy*.go
GOROOT=/.../debian-source-packages/golang-1.x/ PATH=../bin:$PATH go test -v -list '.*' ./net/http/

# Pass "tags"
GOROOT=/.../debian-source-packages/golang-1.x/ PATH=../bin:$PATH go test -v -tags=osusergo os/user

# Another way for internal test suites:
GOROOT=/usr/src/golang/golang-1.8-1.8.1/ PATH=../bin:$PATH go tool dist test -list
GOROOT=/usr/src/golang/golang-1.8-1.8.1/ PATH=../bin:$PATH go tool dist test -run go_test:cmd/go
# If errors don't make sense:
GOROOT=/usr/src/golang/golang-1.8-1.8.1/ PATH=../bin:$PATH go tool dist test -run go_test:net/http -rebuild

Rdeps availability on buildds

The buildds should pick your updated Go libraries from the security archive for LTS suite, so as soon they are “Installed” in the buildd status.

Copyright (C) 2021, 2022, 2023, 2025 Sylvain Beucler