=================== Technical workflows =================== Usually we work on a Debian stable or testing computer, but to build and test security updates, we need an environment that matches the target dist: oldstable/oldoldstable for LTS, and even older for ELTS. There's (often) more than one way to do it (`TIMTOWTDI `_). Here are two main approaches. Different parts may be combined. Note: while ELTS is not part of Debian, we include some ELTS-based instructions as it's common to reuse ELTS work for LTS and vice-versa. .. contents:: :depth: 3 --------------------- Full virtual machines --------------------- This workflow involves doing most of the work within a complete virtual machine. * No pollution from newer environments, including when building source packages * Security isolation at all times; e.g. credentials (Salsa) not available by default * Match our end-users' environment, including kernel * Graphic support by default, including full-desktop environment * Use tools from target dist at all times (compatibility) (autopkgtest, piuparts, lintian, etc.) * Snapshots * Easy multi-VMs / networking setup * Fast iterative debugging (fix/``make``/fix/``make``) rather than lengthy rebuilds from scratch Cons: * Manual install * More suited for remote-style development (SSH/text-mode) * Credentials (Salsa) not available by default ;) * Use tools from target dist at all times (may lack new features) * Limited autopkgtest (no ``full-isolation``) * Build environment may become unclean / non-minimal as testing goes, but salsa/debusine compensates, and snapshots help Example: .. image:: /_static/technical-workflows/virt-manager.png * Install ``libvirt`` and its default ``virbr0`` bridge (192.168.122.0/24). * Use ``virt-manager`` with the QEMU backend to easily create VMs. * Install a full graphic system using the default Debian installation method: using an ISO file. Do this for each release you need (e.g. trixie-stable, bookworm-oldstable, bullseye-lts, buster-elts, stretch-elts). * Install SSH and some remote development tools (emacs-nox, vim...). * Create a SSH key and add it to Salsa with an expiration date (isolate your main SSH key, especially when testing untrusted exploits). * Update your VM with the latest security patches, and create a clean snapshot. After you're done with a security update, reset to the last clean snapshot, update again and create a new clean snapshot. * Clone the VM in case you need more than 1 VM, e.g. client/server or minimal cluster setup. * Build locally using ``git-buildpackage``, or even plain ``apt-source`` + ``debuild`` and later ``gbp import-dsc``. * Run :doc:`/wiki/TestSuites/autopkgtest` within the VM, possibly with an LXC setup. * Run any target tool as usual within the VM without additional containers/chdist setup. Note: previously VirtualBox could be used, but due to its open-core model, plus licensing issues in said core, that package didn't left sid for many years and can't be recommended anymore. ---------------- Light containers ---------------- This workflow does most of the work in one's environment and start light target environments as needed. Pros: * Unattended install * Uses standard develpment environment * Credentials available by default (Git, etc.) * Volatile environments Cons: * Possible pollution from newer environment * Need to be more careful with security isolation (credentials, untrusted PoC/exploits testing) * Harder/longer to setup a full test desktop environment * May not match our end-users' environment: no bootloader, no graphics, testing-oriented enviroment (e.g. default user is root). * May use incompatible newer tooling (e.g. previous autopkgtest issues with jessie) This may not be suitable for all kinds of testing, but this works fine for running full `isolation-machine autopkgtest `_. Example: * Use your normal development environment (editors, etc.) * Don't run PoCs and exploits directly * Run builds on Salsa CI, or ephemeral sbuild chroots. * Run local builds with ``git-buildpackage --git-builder/--git-pbuilder`` * Run `autopkgtest `_ with ``autopkgtest-build-qemu``. * Use `chdist `_ to check apt/packages status in other dist. In the past we had to build packages locally, either to upload them as binaries, or to test the build in a clean environments prior to crossing fingers and sending it to official buildds. Nowadays Salsa CI and debusine do this for you, but we keep the information in case a manual rebuild is needed, typically for debugging or whenever there's an issue/limitation with the CI. ~~~~~~ sbuild ~~~~~~ See: * https://wiki.debian.org/sbuild * https://meetings-archive.debian.net/pub/debian-meetings/2025/DebConf25/debconf25-775-using-sbuild-in-2025-extended-version.av1.webm Be sure to install the latest (trixie / bookworm-backports) sbuild with ``unshare`` support, like Debian buildds and possibly debusine. Moreover there's no need for chroots configuration anymore: :: apt install sbuild mmdebstrap uidmap arch-test apt-cacher :doc:`wiki/TestSuites/autopkgtest` and piuparts need to installed on the host: :: sudo apt install autopkgtest autodep8 sudo apt install piuparts adequate Some default configuration: :: mkdir ~/.config/sbuild/ wget https://deb.freexian.com/extended-lts/archive-key.gpg -O /var/tmp/freexian-archive-key.gpg cat <<'EOF' >> ~/.config/sbuild/config.pl # Use newer unshare backend $chroot_mode = "unshare"; # Investigate failures $external_commands = { "build-failed-commands" => [ [ '%SBUILD_SHELL' ] ] }; # Cache chroot tarball $unshare_mmdebstrap_keep_tarball = 1; $unshare_mmdebstrap_max_age = 8640000; # 10 days # Since trixie, /tmp is tmpfs # and large builds will fail after # filling all memory.. $unshare_tmpdir_template = '/var/tmp/tmp.sbuild.XXXXXXXXXX'; # Transparent ELTS support push @{$unshare_mmdebstrap_extra_args}, qr/^(buster.*|stretch.*)$/, ["--include=freexian-archive-keyring", "--keyring=/var/tmp/freexian-archive-key.gpg", "http://deb.freexian.com/extended-lts/"]; # TODO: same for piuparts? currently fails with ELTS # apt-cacher push @{$unshare_mmdebstrap_extra_args}, "*", ['--aptopt=Acquire::http { Proxy "http://127.0.0.1:3142"; }']; EOF Examples: :: sbuild -d bookworm hello_2.10-3.dsc --run-autopkgtest --run-piuparts sbuild -d buster hello_2.10-2.dsc --run-autopkgtest sbuild -d stretch --arch=i386 hello_2.10-1+deb9u1.dsc --run-autopkgtest # Note: autopkgtest-virt-unshare reuses sbuild's chroot tarballs cache (~/.cache/build/) # This works directly too: autopkgtest hello_2.10-3.dsc hello_2.10-3_amd64.changes -- unshare --release bookworm Troubleshootings: :: sbuild -d buster hello_2.10-2.dsc # KO with default configuration: buster is archived and not available # -> see configuration above to auto-add mmdebstrap ELTS parameters sbuild -d bookworm hello_2.10-3.dsc --run-autopkgtest # KO, not installed # needs autopkgtest installed on the host TODO: clean source package generation, in particular debusine vs. ftp-upload.d.o (``-S -sa``, etc.). TODO: git-buildpackage + sbuild ~~~~~~~~ pbuilder ~~~~~~~~ https://wiki.debian.org/pbuilder Simple builder, it used to be simpler than sbuild, but sbuild got more easy to use. :: # Init sudo pbuilder create --basetgz /var/cache/pbuilder/base-bullseye.tgz \ --distribution bullseye \ --othermirror 'deb http://security.debian.org/ bullseye-security main contrib' sudo pbuilder update --basetgz /var/cache/pbuilder/base-bullseye.tgz # Rebuild source packages _from bullseye_ (in extracted source) pdebuild --buildresult .. --use-pdebuild-internal --debbuildopts '-S' \ -- --basetgz /var/cache/pbuilder/base-bullseye.tgz # doesn't work: sudo pbuilder debuild --basetgz /var/cache/pbuilder/base-bullseye.tgz --buildresult .. --debbuildopts '-S' # or: just 'debuild' in a bullseye VM # Rebuild binary packages from bullseye, # reproducing buildd's separate build-indep/build-arch + no network export DEB_BUILD_OPTIONS="nocheck ..." # - first security upload: sudo --preserve-env=DEB_BUILD_OPTIONS \ pbuilder build --basetgz /var/cache/pbuilder/base-bullseye.tgz \ --source-only-changes --logfile build-indep.log --buildresult . \ --binary-indep --debbuildopts '-sa' package+deb11u1.dsc sudo --preserve-env=DEB_BUILD_OPTIONS \ pbuilder build --basetgz /var/cache/pbuilder/base-bullseye.tgz \ --source-only-changes --logfile build-arch.log --buildresult . \ --binary-arch --debbuildopts '-sa' package+deb11u1.dsc # - later uploads (source tarball already present at security.d.o): sudo --preserve-env=DEB_BUILD_OPTIONS \ pbuilder build --basetgz /var/cache/pbuilder/base-bullseye.tgz \ --source-only-changes --logfile build-indep.log --buildresult . \ --binary-indep package+deb11u2.dsc sudo --preserve-env=DEB_BUILD_OPTIONS \ pbuilder build --basetgz /var/cache/pbuilder/base-bullseye.tgz \ --source-only-changes --logfile build-arch.log --buildresult . \ --binary-arch package+deb11u2.dsc # to debug a failed build: # https://pbuilder-docs.readthedocs.io/en/latest/faq.html#logging-in-to-pbuilder-to-investigate-build-failure mkdir hooks/ cp -a /usr/share/doc/pbuilder/examples/C10shell hooks/ sudo pbuilder ... --hookdir hooks/ ... # during build, network is disabled with 'unshare -n' + 'ifconfig lo up' ~~~~~ debvm ~~~~~ Light VMs tailored for specific testing: :: apt install debvm # LTS debvm-create -r bullseye -o bullseye.ext4 debvm-create -r bullseye -o bullseye.ext4 \ --size=3G -k ~/.ssh/id_xxx.pub # ELTS requires both --keyring and freexian-archive-keyring. # Using split archive.d.o / deb.f.c to ease Freexian bandwidth. debvm-create -r buster -o buster-elts.ext4 \ ... -- \ --keyring=/public/path/to/archive-key.gpg \ --include=freexian-archive-keyring \ http://archive.debian.org/debian/ \ "deb http://deb.freexian.com/extended-lts $DIST-lts main" debvm-create -r stretch -o stretch-elts.ext4 \ ... -- \ --keyring=/public//path/to/archive-key.gpg \ --include=freexian-archive-keyring \ http://archive.debian.org/debian/ \ "deb http://deb.freexian.com/extended-lts $DIST-lts main" The part after ``--`` is passed to ``mmdebstrap``. Optionally pre-install additional packages: .. code-block:: bash --include=strace,vim,emacs-nox,... \ For apt-cacher, provide an URL accessible from the host and the VM: .. code-block:: bash --aptopt='Acquire::http { Proxy "http://192.168.122.1:3142"; }' \ You can add the ELTS staging repo: .. code-block:: bash "deb http://deb.freexian.com/extended-lts-staging $DIST-lts-proposed main" \ Run the VM: :: debvm-run -i bullseye.ext4 debvm-run -i bullseye.ext4 --sshport 2222 -- -m 2G ssh -p 2222 -o StrictHostKeyChecking=no root@localhost The part after ``--`` is passed to ``qemu``. See :doc:`howtos/arm-vm` for ARM-specific options. Troubleshootings: * *The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A07310D369055D5A* Due to ``unshare``, ensure ``freexian-archive-key.gpg`` is in a publicly accessible (``+x``) hierarchy (e.g. not in your home dir). Also check the output of gpg: .. code:: bash $ gpg --list-key --no-default-keyring --keyring /.../freexian-archive-key.gpg ... pub rsa4096 2018-05-28 [SC] [expires: 2027-12-05] AB597C4F6F3380BD4B2BEBC2A07310D369055D5A * *E: unable to pick chroot mode automatically (use --mode for manual selection)* Add e.g. ``--mode=unshare`` at the end of the ``debvm-create`` line, or run through ``sudo``. ~~~~~ incus ~~~~~ TODO: some LTS team members mentioned using ``incus`` as part of their workflow. Pros/cons? ~~~~~~ chdist ~~~~~~ To check the status of packages or uninstallability issues without a chroot: :: chdist create bullseye http://deb.debian.org/debian bullseye main #chdist bullseye chdist apt bullseye update chdist apt-cache bullseye policy hello ... For ELTS: :: chdist create buster-elts https://deb.freexian.com/extended-lts buster main apt install freexian-archive-keyring ln -s /etc/apt/trusted.gpg.d/freexian-archive-extended-lts.gpg \ ~/.chdist/buster-elts/etc/apt/trusted.gpg.d/freexian-archive-extended-lts.gpg # or get it from https://deb.freexian.com/extended-lts/archive-key.gpg # Add staging area: echo "deb https://deb.freexian.com/extended-lts-staging buster-lts-proposed main" \ >> ~/.chdist/buster-elts/etc/apt/sources.list --------------------- Foreign architectures --------------------- i386 environments are compatible with amd64 hosts, usually with an optional like ``--arch i386``. ARM environments are better tested from porter boxes, unexpensive boards (:doc:`howtos/arm-debug`), or through slow VMS, for instance with debvm (:doc:`howtos/arm-vm`). arm64 can similarly run armhf and armel using ``--arch``. ------- Testing ------- Salsa CI and debusine both offer many CI tooling. For setting up Salsa CI, see :doc:`/git-workflow-lts`. For debusine, see `wiki:DebusineDebianNet `_. See :doc:`wiki/TestSuites` for manual tests or build procedures for specific packages. ~~~~~~~~~~~~~~~ Manual CI tests ~~~~~~~~~~~~~~~ Salsa CI and debusine may have limitations or fail to handle some corner cases. Running them manually can help debugging, and you can use a version closer to (E)LTS. **Source checks:** inspect overall source changes since last release, even if you used Git: :: debdiff package+deb11u3.dsc package+deb11u4.dsc | diffstat debdiff package+deb11u3.dsc package+deb11u4.dsc | less If possible check the binaries too to detect missing or extra files: :: debdiff --from bullseye-old/*.deb --to bullseye-new/*.deb **Lintian:** check for common packaging issues in last build from extracted source after build, in a bullseye host (only check new errors). :: lintian -i **piuparts:** test package upgrade: :: sudo piuparts -d bullseye \ --extra-repo='deb http://security.debian.org/ bullseye-security main' \ -l piuparts-package.log \ -I :etc/buggy-dep \ --single-changes-list package+deb11u4_{all,amd64}.changes \ | grep -P '(INFO|ERROR):' # also consider --install-remove-install # For archived/expired dists, use an existing tarball e.g.: DIST=stretch sudo piuparts --keep-sources-list \ -b /var/cache/pbuilder/base-$DIST.tgz \ -l piuparts-package.log \ -I :etc/buggy-dep \ --single-changes-list package*_amd64.changes \ | grep -P '(INFO|ERROR):' Piuparts doesn't handle conflicting packages in a single run (e.g. nginx-light vs. nginx-full); in this case, test each ``.deb`` individually (rather than the full changes). With newer piuparts (1.5.0), you can avoid using root by using the ``unshare`` backend, and working in a publicly accessible (``+x``) hierarchy: drop ``sudo`` and add: :: '--bootstrapcmd=mmdebstrap --skip=check/empty --variant=apt' \ ~~~~~~~~~~~~~~~~~~~~ Reverse dependencies ~~~~~~~~~~~~~~~~~~~~ Identify direct and indirect reverse dependencies them with ``apt rdepends``, e;g. ``apt rdepends --recurse --important --follow=Depends,PreDepends,Suggests,Recommends,Conflicts,Replaces`` or `apt-rdepends `_ ``-r``. Check `dose-ceve `_ from `dose-extra `_ for more complex cases, e.g. :doc:`/wiki/TestSuites/golang`. `reverse-depends `_ ``[-b]`` from `ubuntu-dev-tools `_ knows about reverse *build*-dependencies but relies on a web service and only works for Ubuntu.