Technical workflows

Usually we work on a Debian stable or testing computer, but to build and test security updates, we need an environment that matches the target dist: oldstable/oldoldstable for LTS, and even older for ELTS.

There’s (often) more than one way to do it (TIMTOWTDI). Here are two main approaches. Different parts may be combined.

Note: while ELTS is not part of Debian, we include some ELTS-based instructions as it’s common to reuse ELTS work for LTS and vice-versa.

Full virtual machines

This workflow involves doing most of the work within a complete virtual machine.

• No pollution from newer environments, including when building source packages

• Security isolation at all times; e.g. credentials (Salsa) not available by default

• Match our end-users’ environment, including kernel

• Graphic support by default, including full-desktop environment

• Use tools from target dist at all times (compatibility) (autopkgtest, piuparts, lintian, etc.)

• Snapshots

• Easy multi-VMs / networking setup

• Fast iterative debugging (fix/make/fix/make) rather than lengthy rebuilds from scratch

Cons:

• Manual install

• More suited for remote-style development (SSH/text-mode)

• Credentials (Salsa) not available by default ;)

• Use tools from target dist at all times (may lack new features)

• Limited autopkgtest (no full-isolation)

• Build environment may become unclean / non-minimal as testing goes, but salsa/debusine compensates, and snapshots help

Example:

• Install libvirt and its default virbr0 bridge (192.168.122.0/24).

• Use virt-manager with the QEMU backend to easily create VMs.

• Install a full graphic system using the default Debian installation method: using an ISO file. Do this for each release you need (e.g. trixie-stable, bookworm-oldstable, bullseye-lts, buster-elts, stretch-elts).

• Install SSH and some remote development tools (emacs-nox, vim…).

• Create a SSH key and add it to Salsa with an expiration date (isolate your main SSH key, especially when testing untrusted exploits).

• Update your VM with the latest security patches, and create a clean snapshot. After you’re done with a security update, reset to the last clean snapshot, update again and create a new clean snapshot.

• Clone the VM in case you need more than 1 VM, e.g. client/server or minimal cluster setup.

• Build locally using git-buildpackage, or even plain apt-source + debuild and later gbp import-dsc.

• Run autopkgtest within the VM, possibly with an LXC setup.

• Run any target tool as usual within the VM without additional containers/chdist setup.

Note: previously VirtualBox could be used, but due to its open-core model, plus licensing issues in said core, that package didn’t left sid for many years and can’t be recommended anymore.

Light containers

This workflow does most of the work in one’s environment and start light target environments as needed.

Pros:

• Unattended install

• Uses standard develpment environment

• Credentials available by default (Git, etc.)

• Volatile environments

Cons:

• Possible pollution from newer environment

• Need to be more careful with security isolation (credentials, untrusted PoC/exploits testing)

• Harder/longer to setup a full test desktop environment

• May not match our end-users’ environment: no bootloader, no graphics, testing-oriented enviroment (e.g. default user is root).

• May use incompatible newer tooling (e.g. previous autopkgtest issues with jessie)

This may not be suitable for all kinds of testing, but this works fine for running full isolation-machine autopkgtest.

Example:

• Use your normal development environment (editors, etc.)

• Don’t run PoCs and exploits directly

• Run builds on Salsa CI, or ephemeral sbuild chroots.

• Run local builds with git-buildpackage --git-builder/--git-pbuilder

• Run autopkgtest with autopkgtest-build-qemu.

• Use chdist to check apt/packages status in other dist.

In the past we had to build packages locally, either to upload them as binaries, or to test the build in a clean environments prior to crossing fingers and sending it to official buildds. Nowadays Salsa CI and debusine do this for you, but we keep the information in case a manual rebuild is needed, typically for debugging or whenever there’s an issue/limitation with the CI.

sbuild

See:

• https://wiki.debian.org/sbuild

• https://meetings-archive.debian.net/pub/debian-meetings/2025/DebConf25/debconf25-775-using-sbuild-in-2025-extended-version.av1.webm

Be sure to install the latest (trixie / bookworm-backports) sbuild with unshare support, like Debian buildds and possibly debusine. Moreover there’s no need for chroots configuration anymore:

apt install sbuild mmdebstrap uidmap arch-test apt-cacher

autopkgtest and piuparts need to installed on the host:

sudo apt install autopkgtest autodep8
sudo apt install piuparts adequate

Some default configuration:

mkdir ~/.config/sbuild/
wget https://deb.freexian.com/extended-lts/archive-key.gpg -O /var/tmp/freexian-archive-key.gpg

cat <<'EOF' >> ~/.config/sbuild/config.pl
# Use newer unshare backend
$chroot_mode = "unshare";

# Investigate failures
$external_commands = { "build-failed-commands" => [ [ '%SBUILD_SHELL' ] ] };

# Cache chroot tarball
$unshare_mmdebstrap_keep_tarball = 1;
$unshare_mmdebstrap_max_age = 8640000;  # 10 days

# Since trixie, /tmp is tmpfs # and large builds will fail after
# filling all memory..
$unshare_tmpdir_template = '/var/tmp/tmp.sbuild.XXXXXXXXXX';

# Transparent ELTS support
push @{$unshare_mmdebstrap_extra_args},
   qr/^(buster.*|stretch.*)$/,
  ["--include=freexian-archive-keyring",
   "--keyring=/var/tmp/freexian-archive-key.gpg",
   "http://deb.freexian.com/extended-lts/"];
# TODO: same for piuparts? currently fails with ELTS

# apt-cacher
push @{$unshare_mmdebstrap_extra_args}, "*",
  ['--aptopt=Acquire::http { Proxy "http://127.0.0.1:3142"; }'];
EOF

Examples:

sbuild -d bookworm hello_2.10-3.dsc --run-autopkgtest --run-piuparts

sbuild -d buster hello_2.10-2.dsc --run-autopkgtest

sbuild -d stretch --arch=i386 hello_2.10-1+deb9u1.dsc --run-autopkgtest

# Note: autopkgtest-virt-unshare reuses sbuild's chroot tarballs cache (~/.cache/build/)
# This works directly too:
autopkgtest hello_2.10-3.dsc hello_2.10-3_amd64.changes -- unshare --release bookworm

Troubleshootings:

sbuild -d buster hello_2.10-2.dsc
# KO with default configuration: buster is archived and not available
# -> see configuration above to auto-add mmdebstrap ELTS parameters

sbuild -d bookworm hello_2.10-3.dsc --run-autopkgtest  # KO, not installed
# needs autopkgtest installed on the host

TODO: clean source package generation, in particular debusine vs. ftp-upload.d.o (-S -sa, etc.).

TODO: git-buildpackage + sbuild

pbuilder

https://wiki.debian.org/pbuilder

Simple builder, it used to be simpler than sbuild, but sbuild got more easy to use.

# Init
sudo pbuilder create --basetgz /var/cache/pbuilder/base-bullseye.tgz \
 --distribution bullseye \
 --othermirror 'deb http://security.debian.org/ bullseye-security main contrib'
sudo pbuilder update --basetgz /var/cache/pbuilder/base-bullseye.tgz

# Rebuild source packages _from bullseye_ (in extracted source)
pdebuild --buildresult .. --use-pdebuild-internal --debbuildopts '-S' \
 -- --basetgz /var/cache/pbuilder/base-bullseye.tgz
# doesn't work: sudo pbuilder debuild --basetgz /var/cache/pbuilder/base-bullseye.tgz --buildresult .. --debbuildopts '-S'
# or: just 'debuild' in a bullseye VM

# Rebuild binary packages from bullseye,
# reproducing buildd's separate build-indep/build-arch + no network
export DEB_BUILD_OPTIONS="nocheck ..."
# - first security upload:
sudo --preserve-env=DEB_BUILD_OPTIONS \
 pbuilder build --basetgz /var/cache/pbuilder/base-bullseye.tgz \
  --source-only-changes --logfile build-indep.log --buildresult . \
  --binary-indep --debbuildopts '-sa' package+deb11u1.dsc
sudo --preserve-env=DEB_BUILD_OPTIONS \
 pbuilder build --basetgz /var/cache/pbuilder/base-bullseye.tgz \
  --source-only-changes --logfile build-arch.log  --buildresult . \
  --binary-arch  --debbuildopts '-sa' package+deb11u1.dsc
# - later uploads (source tarball already present at security.d.o):
sudo --preserve-env=DEB_BUILD_OPTIONS \
 pbuilder build --basetgz /var/cache/pbuilder/base-bullseye.tgz \
  --source-only-changes --logfile build-indep.log --buildresult . \
  --binary-indep package+deb11u2.dsc
sudo --preserve-env=DEB_BUILD_OPTIONS \
 pbuilder build --basetgz /var/cache/pbuilder/base-bullseye.tgz \
  --source-only-changes --logfile build-arch.log  --buildresult . \
  --binary-arch  package+deb11u2.dsc

# to debug a failed build:
# https://pbuilder-docs.readthedocs.io/en/latest/faq.html#logging-in-to-pbuilder-to-investigate-build-failure
mkdir hooks/
cp -a /usr/share/doc/pbuilder/examples/C10shell hooks/
sudo pbuilder ... --hookdir hooks/ ...
# during build, network is disabled with 'unshare -n' + 'ifconfig lo up'

debvm

Light VMs tailored for specific testing:

apt install debvm

# LTS
debvm-create -r bullseye -o bullseye.ext4
debvm-create -r bullseye -o bullseye.ext4 \
  --size=3G -k ~/.ssh/id_xxx.pub

# ELTS requires both --keyring and freexian-archive-keyring.
# Using split archive.d.o / deb.f.c to ease Freexian bandwidth.
debvm-create -r buster -o buster-elts.ext4 \
  ...
  -- \
  --keyring=/public/path/to/archive-key.gpg \
  --include=freexian-archive-keyring \
  http://archive.debian.org/debian/ \
  "deb http://deb.freexian.com/extended-lts $DIST-lts main"
debvm-create -r stretch -o stretch-elts.ext4 \
  ...
  -- \
  --keyring=/public//path/to/archive-key.gpg \
  --include=freexian-archive-keyring \
  http://archive.debian.org/debian/ \
  "deb http://deb.freexian.com/extended-lts $DIST-lts main"

The part after -- is passed to mmdebstrap.

Optionally pre-install additional packages:

--include=strace,vim,emacs-nox,... \

For apt-cacher, provide an URL accessible from the host and the VM:

--aptopt='Acquire::http { Proxy "http://192.168.122.1:3142"; }' \

You can add the ELTS staging repo:

"deb http://deb.freexian.com/extended-lts-staging $DIST-lts-proposed main" \

Run the VM:

debvm-run -i bullseye.ext4

debvm-run -i bullseye.ext4 --sshport 2222 -- -m 2G
ssh -p 2222 -o StrictHostKeyChecking=no root@localhost

The part after -- is passed to qemu.

See Create an arm* VM for ARM-specific options.

Troubleshootings:

• The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY A07310D369055D5A

Due to unshare, ensure freexian-archive-key.gpg is in a publicly accessible (+x) hierarchy (e.g. not in your home dir).

Also check the output of gpg:

$ gpg --list-key --no-default-keyring --keyring /.../freexian-archive-key.gpg
...
pub   rsa4096 2018-05-28 [SC] [expires: 2027-12-05]
      AB597C4F6F3380BD4B2BEBC2A07310D369055D5A

• E: unable to pick chroot mode automatically (use –mode for manual selection)

Add e.g. --mode=unshare at the end of the debvm-create line, or run through sudo.

incus

TODO: some LTS team members mentioned using incus as part of their workflow.

Pros/cons?

chdist

To check the status of packages or uninstallability issues without a chroot:

chdist create bullseye http://deb.debian.org/debian bullseye main
#chdist <command> bullseye <args>
chdist apt bullseye update
chdist apt-cache bullseye policy hello
...

For ELTS:

chdist create buster-elts https://deb.freexian.com/extended-lts buster main
apt install freexian-archive-keyring
ln -s /etc/apt/trusted.gpg.d/freexian-archive-extended-lts.gpg \
   ~/.chdist/buster-elts/etc/apt/trusted.gpg.d/freexian-archive-extended-lts.gpg
# or get it from https://deb.freexian.com/extended-lts/archive-key.gpg
# Add staging area:
echo "deb https://deb.freexian.com/extended-lts-staging buster-lts-proposed main" \
  >> ~/.chdist/buster-elts/etc/apt/sources.list

Foreign architectures

i386 environments are compatible with amd64 hosts, usually with an optional like --arch i386.

ARM environments are better tested from porter boxes, unexpensive boards (ARM local debug), or through slow VMS, for instance with debvm (Create an arm* VM). arm64 can similarly run armhf and armel using --arch.

Testing

Salsa CI and debusine both offer many CI tooling.

For setting up Salsa CI, see Git workflow for the (E)LTS packages.

For debusine, see wiki:DebusineDebianNet.

See Test Suites for manual tests or build procedures for specific packages.

Manual CI tests

Salsa CI and debusine may have limitations or fail to handle some corner cases. Running them manually can help debugging, and you can use a version closer to (E)LTS.

Source checks: inspect overall source changes since last release, even if you used Git:

debdiff package+deb11u3.dsc package+deb11u4.dsc | diffstat
debdiff package+deb11u3.dsc package+deb11u4.dsc | less

If possible check the binaries too to detect missing or extra files:

debdiff --from bullseye-old/*.deb --to bullseye-new/*.deb

Lintian: check for common packaging issues in last build from extracted source after build, in a bullseye host (only check new errors).

lintian -i

piuparts: test package upgrade:

sudo piuparts -d bullseye \
  --extra-repo='deb http://security.debian.org/ bullseye-security main' \
  -l piuparts-package.log \
  -I :etc/buggy-dep \
  --single-changes-list package+deb11u4_{all,amd64}.changes \
  | grep -P '(INFO|ERROR):'
# also consider --install-remove-install

# For archived/expired dists, use an existing tarball e.g.:
DIST=stretch
sudo piuparts --keep-sources-list \
  -b /var/cache/pbuilder/base-$DIST.tgz \
  -l piuparts-package.log \
  -I :etc/buggy-dep \
  --single-changes-list package*_amd64.changes \
  | grep -P '(INFO|ERROR):'

Piuparts doesn’t handle conflicting packages in a single run (e.g. nginx-light vs. nginx-full); in this case, test each .deb individually (rather than the full changes).

With newer piuparts (1.5.0), you can avoid using root by using the unshare backend, and working in a publicly accessible (+x) hierarchy: drop sudo and add:

'--bootstrapcmd=mmdebstrap --skip=check/empty --variant=apt' \

Reverse dependencies

Identify direct and indirect reverse dependencies them with apt rdepends, e;g. apt rdepends --recurse --important --follow=Depends,PreDepends,Suggests,Recommends,Conflicts,Replaces or apt-rdepends -r.

Check dose-ceve from dose-extra for more complex cases, e.g. golang.

reverse-depends [-b] from ubuntu-dev-tools knows about reverse build-dependencies but relies on a web service and only works for Ubuntu.