===========
autopkgtest
===========

The *autopkgtest* package runs `DEP-8 <https://dep-team.pages.debian.net/deps/dep8/>`_ tests.

This page mostly aims at running these tests manually within a (E)LTS
VM, for efficiently debugging, fixing and designing tests. This is
useful when salsa-ci/ci.debian.net/ci.freexian.com fails at the end of
a lengthy process with no debug access.

*As for 2025-02 bookworm is Debian Stable, and we still support jessie
in ELTS, so we focus on stable tooling and document jessie. When
trixie is released and jessie EOL, we can upgrade the tooling and drop
jessie documentation.*

See also https://wiki.debian.org/ContinuousIntegration/autopkgtest

.. contents:: :depth: 2

---------
Local run
---------

This runs tests directly in the current host. This is enough for most
tests and easiest to debug.

This complements working within a LTS VM for fine-grained
patch-by-patch testing.

In your test VM, examples/cheat-sheet with stretch+ syntax:

.. code-block:: bash

 cd package-version/
 apt-get build-dep .
 debuild
 
 apt install autopkgtest
 # additional automatic/ecosystem tests:
 # https://manpages.debian.org/unstable/autodep8/autodep8.1.en.html
 apt install autodep8
 
 autopkgtest ../..._amd64.changes -- null
 # save the test logs (but no more colors):
 autopkgtest -o ../test.outdir ../..._amd64.changes -- null
 # test local debian/tests/ changes, without auto-rebuilding
 autopkgtest -B . -- null
 # run only selected test
 autopkgtest --test-name=TEST ...
 # run only selected testS (>=bookworm)
 autopkgtest --test-name=TEST1 --test-name=TEST2 ...
 # ensure you run test with a non-root user
 autopkgtest -u dla ...

 # run a reverse dependency's tests, simply:
 autopkgtest rdep_source_package_name -- null

In your test VM, example with `rails <https://wiki.debian.org/LTS/TestSuites/rails>`_, old (jessie) syntax:

.. code-block:: bash

 cd rails-4.1.8/
 apt-get build-dep rails
 debuild
 
 apt install autopkgtest
 apt install ruby-spring ruby-jquery-rails ruby-sqlite3 ruby-sass-rails ruby-uglifier ruby-coffee-rails ruby-turbolinks ruby-jbuilder ruby-sdoc
 apt install ruby-byebug ruby-web-console # stretch
 # TODO: autopkgtest should install dependencies automatically, possibly those weren't due to APT::Install-Recommends "false", or it's a bug somewhere
 
 adt-run -B ./ --- null
 adt-run ../rails_4.1.8-1+deb8u7_amd64.changes --- null

TODO: can we pass DEB_BUILD_OPTIONS=xxx?

------------------------------------------
Full LXC environment (isolation-container)
------------------------------------------

This is for tests that would break the VM configuration (such as
``/etc``). It is easy to create LXC containers from within a VM,
here's how.

In your test VM, example with *apache2*, new (stretch+) syntax.

LXC setup (buster and above):

.. code-block:: bash

 apt install autopkgtest
 apt install lxc lxc-templates debootstrap rsync distro-info
 systemctl status lxc-net
 # If necessary:
 cat >> /etc/lxc/default.conf <<'EOF'
 lxc.apparmor.profile = unconfined
 EOF

 # Create base container
 # Add security updates to better match target environment
 DIST=bullseye
 cat <<EOF > ~/apt-security.sh
 #!/bin/bash
 echo 'deb http://security.debian.org/debian-security $DIST-security main' >> /etc/apt/sources.list
 echo 'deb-src http://security.debian.org/debian-security $DIST-security main' >> /etc/apt/sources.list
 apt update
 apt -y upgrade
 EOF
 autopkgtest-build-lxc debian $DIST amd64 ~/apt-security.sh

TODO: adjust sources for buster/ELTS.

LXC setup (stretch):

.. code-block:: bash

 apt install autopkgtest
 apt install lxc debootstrap rsync ebtables
 # Create network
 apt install libvirt-daemon-system
 # edit /etc/libvirt/qemu/networks/default.xml if needed (IP range...)
 virsh net-autostart default
 service libvirtd restart
 cat > /etc/lxc/default.conf <<'EOF'
 lxc.network.type = veth
 lxc.network.flags = up
 lxc.network.link = virbr0
 EOF

 # Create base container
 # Add security updates to better match target environment
 cat <<'EOF' > ~/apt-security.sh
 #!/bin/bash
 echo 'deb http://security.debian.org/debian-security stretch/updates main' >> /etc/apt/sources.list
 echo 'deb-src http://security.debian.org/debian-security stretch/updates main' >> /etc/apt/sources.list
 apt update
 apt -y upgrade
 EOF
 autopkgtest-build-lxc debian stretch amd64 ~/apt-security.sh

Testing the package (buster and above):

.. code-block:: bash

 DIST=bullseye
 cd apache2-2.4.38/
 apt-get build-dep .
 debuild

 # Test the package you're working on
 autopkgtest --shell-fail ../apache2_2.4.38-3+deb10u9_amd64.changes \
   -- lxc autopkgtest-$DIST-amd64
 # Caveat: tests packages *from the archive* (not your new .deb):
 #autopkgtest -B . -- lxc autopkgtest-$DIST-amd64
 # use this instead:
 autopkgtest -B . ../*.deb -- lxc autopkgtest-$DIST-amd64

Testing the package (stretch):

.. code-block:: bash

 cd apache2-2.4.25/
 apt-get build-dep apache2
 debuild

 # Test the package you're working on
 autopkgtest ../apache2_2.4.25-3+deb9u11_amd64.changes \
   -- lxc autopkgtest-stretch-amd64
 # Caveat: tests packages *from the archive* (not your new .deb):
 #autopkgtest -B . -- lxc autopkgtest-stretch-amd64
 # use this instead:
 autopkgtest -B . ../*.deb -- lxc autopkgtest-stretch-amd64
 
Testing the package (jessie syntax):

.. code-block:: bash

 adt-build-lxc debian jessie amd64
 # TODO: apply security updates in adt-jessie-amd64
 adt-run ../apache2_2.4.10-10+deb8u19_amd64.changes \
   --- lxc adt-jessie-amd64

---------------------------------------
Full VM environment (isolation-machine)
---------------------------------------

If you really need ``isolation-machine`` (which can break the VM,
reboot it, etc.), or if you'd rather work from your host rather than
within a LTS VM, you'll need to proceed differently.

First, create an autopkgtest VM image:

.. code-block:: bash

  RELEASE=bullseye  # or jessie, stretch, buster...
  sudo autopkgtest-build-qemu --size=25G \
    $RELEASE ./autopkgtest-$RELEASE.img

If you have an apt-cacher (note: 127.0.0.1 special-cased and works
within qemu, through 10.0.2.2):

.. code-block:: bash

    --apt-proxy=http://127.0.0.1:3142 \

For non-Debian (e.g. ELTS) dists, also pass its mirror and `keyring
<https://www.freexian.com/lts/extended/docs/how-to-use-extended-lts/>`_
(requires autopkgtest >= 5.29, cf. bookworm-backports):

.. code-block:: bash

    --keyring=./freexian-archive-extended-lts.gpg \
    --mirror=http://deb.freexian.com/extended-lts/ \

Note: this was tested on bookworm and trixie/202502. Sometimes the
resulting image is truncated (< 1MB), try re-running the command.

TODO: these autopkgtest Debian VMs don't have security sources.

Then, run the tests using the ``qemu`` backend (probably from outside
your LTS VM, unless you have support for nested virtualization):

.. code-block:: bash

  autopkgtest --shell-fail --apt-upgrade [...] \
    -- qemu /path/to/autopkgtest-$RELEASE.img

e.g.:

.. code-block:: bash

  cd python2.7/
  autopkgtest -sU \
    . ../previous-run/binaries/*.deb -o ../out \
    -- qemu ../../autopkgtest-buster.img

**Troubleshooting**

As usual, if working from a source directory, clean-up it up first
(especially big files), as autopkgtest will copy it to its VM.

If the VM status becomes *[paused] (io-error)* for no reason, check
if /tmp is filled, especially if it's a tmpfs (default in trixie).

When facing a bug, the maintainers are pretty responsive. In the team,
jochensp and paride among others contributed to the toolchain.

To investigate an issue in the tests, remember the ``-s|--shell-fail``
option, which here will just pause for you to access the VM through
serial and SSH (see setup below), actually available on boot.

Serial access:

.. code-block:: bash

  minicom -D unix#/tmp/autopkgtest-qemu.xxx/ttyS0
  minicom -D unix#/tmp/autopkgtest-qemu.xxx/ttyS1
  minicom -D unix#/tmp/autopkgtest-qemu.xxx/monitor # QEMU monitor
  # also try removing -nographics in /usr/share/autopkgtest/lib/autopkgtest_qemu.py

SSH access: autopkgtest redirects port 10022 to local SSH, but you
need it installed through ``autopkgtest-build-qemu --script=...``:

.. code-block:: bash

  cat <<'EOF' > ./install-openssh-server.sh
  #!/bin/sh
  # Executed in the host, remember to use chroot
  chroot "$1" apt-get -o "Acquire::http::Proxy=$AUTOPKGTEST_SETUP_APT_PROXY" \
    install -y openssh-server < /dev/null
  chroot "$1" mkdir -m 700 /root/.ssh/
  cp -a /home/you/.ssh/yourkey.pub "$1/root/.ssh/authorized_keys"
  EOF
  chmod 755 ./install-openssh-server.sh

  autopkgtest-build-qemu ... --script=./install-openssh-server.sh

  autopkgtest ... -- qemu autopkgtest-$RELEASE.img
  ssh -p 10022 -o StrictHostKeyChecking=no root@127.0.0.1

To investigate an issue in a disk image:

.. code-block:: bash

  qemu-img convert autopkgtest-$RELEASE.img autopkgtest-$RELEASE.raw
  sudo kpartx -av autopkgtest-$RELEASE.raw
  sudo mount /dev/mapper/loop0p1 /mnt/inspection/
  ...
  sudo umount /mnt/inspection
  sudo kpartx -dv autopkgtest-$RELEASE.raw

If for some reason things don't work, an alternative is the ``ssh``
backend to connect to an existing VM, see also
``/usr/share/autopkgtest/ssh-setup/`` to handle reboot & reset.

Last resort, make a snapshot of your VM and try running with the
``null`` autopkgtest backend.

**mmdebstrap**: starting with trixie, ``mmdebstrap`` provides
 ``mmdebstrap-autopkgtest-build-qemu`` as an alternative/replacement.

Snippet for SSH setup:

.. code-block:: bash

 mmdebstrap-autopkgtest-build-qemu ... -- \
   --include=openssh-server \
   --customize-hook='mkdir -m700 -p "$1/root/.ssh"' \
   --customize-hook="upload /pass/to/key.pub /root/.ssh/authorized_keys"

-----------------
Local APT archive
-----------------

adt-run (jessie) can be confused by e.g. conflicting binary packages
from the same source package, in which case you need to setup a local
APT archive for your packages:

.. code-block:: bash

 cd /usr/src/mysourcepackage/
 apt-ftparchive packages . > Packages
 apt-ftparchive release . > Release
 echo 'deb [trusted=yes] file:///usr/src/mysourcepackage ./' >> /etc/apt/sources.list
 apt update

Alternatively, for a single machine:

.. code-block:: bash

 apt install local-apt-repository
 mkdir /srv/local-apt-repository
 cp -a *.deb /srv/local-apt-repository/
 apt update


----------- 
Backporting
----------- 

stretch -> jessie:

 * *$AUTOPKGTEST_TMP* -> *$ADTTMP*
 * *build-essential* (e.g. *gcc*) not installed by default, re-add it if necessary


| Copyright (C) 2021, 2022, 2023, 2024, 2025  Sylvain Beucler