ARM local debug

Debugging ARM build or test failures can be difficult because test environments are not readily available.

ARM VMs are quite slow on amd64, to the point they introduce new issues.

For Debian and Debian LTS we have porter boxes, which come with limitations (no root access, no graphic interface).

For archived/ELTS dists, since ~2025-08 with trixie and the new sbuild unshare mode, we now may use the Debian porter boxes for ELTS dists (as we are not limited by the DSA-supported schroot chroots).

There’s an issue in debusine to provide SSH access on failure, but it’s been backlogged.

So let’s create a cheap environment for full flexibility.

Hardware

The Raspberry Pi 400 is a convenient Raspberry Pi v4 ARM board, fan-less, with integrated keyboard and built-in WiFi, hence requiring far fewer cables.

../_images/r400.jpg

The architecture is arm64 and also handles armhf/armel (32-bit) builds. It also supports KVM virtualization. There’s 4GB RAM.

R400 supports booting from USB out of the box. We can use a USB hard-disk exclusively and avoid SD cards which tend to wear down quickly with many write operations.

It costs less than $100 with a power supply and a micro-HDMI/HDMI cable. You’ll also need an USB external hard-disk or a cheap USB/SATA adapter.

Other Raspberry Pi models: all models >= v2 can run Debian armhf or arm64. Model 1 is in-between armel and armhf, and can run Debian armel or the Raspberry Pi OS specific armhf (“armhf~”), which is incompatible with Debian armhf, so not useful here. Some other models also support USB boot but may need a firmware update first. v5 is not fully supported by mainline kernels as of 2025-09.

Possible alternatives:

Base system

The raspi.debian.net (pre-built images) appears to be dead-ish. There’s a lengthy manual installation method at wiki:RaspberryPi4 which I didn’t try.

Conversely Raspberry Pi OS (ex-raspbian) uses Debian arm64 directly for latest Raspberry Pi models.

Download the latest full version and flash it on a fast hard-disk:

xzcat 2025-10-01-raspios-trixie-arm64.img.xz \
  | sudo dd of=/dev/sdX bs=64k status=progress oflag=sync

Unplug any SD card, plug your hard-disk to an USB port, and boot.

Optional hardening:

  • Menu > Preferences > Control Centre (or rpcc, or sudo raspi-config)

  • Disable auto-login:

    • System > Console/Desktop auto login > [uncheck]

  • Enable SSH:

    • Interfaces > SSH > [check]

  • Save your screen:

    • Display > Screen Blanking > [check]

  • Adjust password-less sudo for the main user:

    visudo /etc/sudoers.d/010_pi-nopasswd
# drop NOPASSWD:

  • Controlled package installation:

    echo 'APT::Install-Recommends "false";' > /etc/apt/apt.conf.d/00InstallRecommends

  • Debian backports are compatible with Raspberry Pi OS:

    echo "deb http://deb.debian.org/debian trixie-backports main" >> /etc/apt/sources.list

  • Space-efficient persistent logs (journald is configured as volatile):

    apt install rsyslog

Emergency UI shortcuts in the default LXDE-based UI:

  • Super for menu (killall wf-panel-pi to restart)

  • Ctrl+Alt+T for a terminal

  • Ctrl+Alt+Del for shutdown menu

Check your disk’s performances with Menu > Accessories > Raspberry Pi Diagnostics (or agnostics).

Support check:

$ dpkg --print-architecture
arm64
$ arch-test
arm64
armel
armhf

TODO: purer Debian base install, either with the manual procedure, or using mmdebstrap. Though, we won’t compile or test any package in this base system, only use it to run ARM chroot/unshare/qemu containers, so it doesn’t matter much.

sbuild/unshare

We’ll install the latest sbuild with unshare support, like Debian buildds and debusine. Moreover there’s no need for chroots configuration anymore:

apt install sbuild mmdebstrap uidmap arch-test
apt install autopkgtest autodep8  # for --run-autopkgtest
apt install piuparts adequate  # for --run-piuparts

Then you can use sbuild as usual (see technical workflows for the setup).

This also works on ARM porter boxes such as amdahl.debian.org aka arm64-porterbox.debian.net.

Tests (unshare mode):

sbuild -d unstable hello

sbuild -d bookworm hello  # KO, can't auto-add multiple source lines
dget http://deb.debian.org/debian/pool/main/h/hello/hello_2.10-3.dsc
sbuild -d bookworm hello_2.10-3.dsc

sbuild -d buster hello_2.10-2.dsc
sbuild -d buster --arch=armhf hello_2.10-2.dsc
sbuild -d buster gimp_2.10.8-2+deb10u3.dsc  # OK, 30min

sbuild -d stretch hello_2.10-1+deb9u1.dsc  # KO, no stretch/arm64 support
sbuild -d stretch --arch=armhf hello_2.10-1+deb9u1.dsc  # OK
sbuild -d stretch --arch=armhf mercurial_4.0-1+deb9u3.dsc  # OK, 30min

sbuild -d bookworm hello_2.10-3.dsc --run-autopkgtest
# Note: uses the host's autopkgtest, unlike lintian which is installed in the chroot

# Note: autopkgtest-virt-unshare reuses sbuild's chroot tarballs cache (~/.cache/build/)
# This works directly too:
autopkgtest hello_2.10-3.dsc hello_2.10-3_arm64.changes -- unshare --release bookworm --arch arm64

sbuild -d buster hello_2.10-2.dsc --run-autopkgtest
sbuild -d buster --arch=armhf hello_2.10-2.dsc --run-autopkgtest
sbuild -d stretch --arch=armhf hello_2.10-1+deb9u1.dsc --run-autopkgtest  # no tests but OK
sbuild -d stretch --arch=armhf mercurial_4.0-1+deb9u3.dsc --run-autopkgtest
# TODO: autopkgtest [09:51:16]: ERROR: testbed failure: testbed auxverb failed with exit code 255

apt install piuparts adequate
sbuild -d bookworm hello_2.10-3.dsc --run-piuparts

apt install git-buildpackage pristine-tar
gbp clone https://salsa.debian.org/lts-team/packages/gimp --debian-branch debian/buster
cd gimp/
gbp buildpackage --git-builder=sbuild --no-clean-source

mv *.build ..
git checkout debian/stretch
gbp buildpackage --git-builder=sbuild --no-clean-source --arch=armhf

Troubleshootings:

sbuild -d stretch mercurial_4.0-1+deb9u3.dsc  # KO, no stretch/arm64 support
# add --arch=armhf; requires 'arch-test'

QEMU/KVM

We can run full ARM VMs on the R400, with native speed, the original Debian kernels, graphics support, etc.

# basic support
apt install qemu-system-arm
# for the full virt-manager GUI and network
apt install virt-manager gir1.2-spiceclientglib-2.0 gir1.2-spiceclientgtk-3.0 \
  libvirt-daemon-system qemu-utils qemu-efi-aarch64
# for remote graphic access
apt install netcat-openbsd
# default virbr0 bridge
virsh net-autostart default
virsh net-start default

You can user virt-manager locally, or add a new QEMU/KVM connection from your main host, using qemu+ssh://.

Trixie VM install: run a classical install from virt-manager with debian-13.1.0-arm64-netinst.iso (local ISO, KVM/aarch64/virt on virbr0).

TODO: difference between machine types “virt” and “raspi4b”.

Older debian dists may need work-arounds such as extracting vmlinuz/initrd (ex: https://0xca7.github.io/notes/virt-manager-arm/).

To test a graphic system, add some virtual hardware:

  • Details > Add Hardware

    • Video > Virtio

    • Graphics > Spice server

    • Input > USB Keyboard (VirtIO doesn’t work in early phases like GRUB)

    • Input > EvTouch USB Graphics Tablet

# TODO: test buster/stretch ISOs

debvm works too, see Create an arm* VM for more detailed instructions.

apt install debvm

debvm-create -r buster -o buster-arm64.ext4 ... \
  -- \
    --include=freexian-archive-keyring \
    --keyring=/var/tmp/freexian-archive-key.gpg \
    ... \
    http://deb.freexian.com/extended-lts/
#   ~20min with graphics support
debvm-run -i buster-arm64.ext4 ...

debvm-create -r stretch ... -- \
  --arch=armhf --include=linux-image-5.10-armmp ...

TODO: arm64 VMs are KVM-accelerated, but not armhf VMs (emulated 32-bit CPU). KVM arm32-on-arm64 is possible with some debvm-run tweaking, see https://wiki.debian.org/Arm64Qemu and https://salsa.debian.org/helmutg/debvm/-/issues/25 .

TODO: graphics: blank screen with debvm+stretch+graphics.

TODO: graphics: how to change default english keymap.

TODO: KVM access is granted through a graphical login. ACLs (setfacl -m ... /dev/kvm) even gets reset. When working remotely exclusively (SSH) this prevents working as an unprivileged user, unless you keep the graphical autologin.

Copyright (C) 2025 Sylvain Beucler