========= docker.io ========= This covers testing docker swarm, especially to verify the overlay network encrypts data send among the containers. Three nodes are needed: one manager and two workers VMs setup --------- This test relies on debvm, and a virbr0 network bridge already configured. */etc/qemu/bridge.conf* should have .. allow virbr0 Enable network bridge for being used by qemu: .. code:: sh sudo chown root:kvm /etc/qemu/bridge.conf sudo chmod 640 /etc/qemu/bridge.conf sudo chmod u+s /usr/lib/qemu/qemu-bridge-helper `apt-cacher-ng` is also running on the host machine. Adjust MIRROR URL if not is your case. .. code:: sh export RELEASE=buster debvm-create --size=3GB --release=$RELEASE -- http://192.168.122.1:3142/deb.debian.org/debian mv rootfs.ext4 rootfs-manager.ext4 debvm-create --size=3GB --release=$RELEASE -- http://192.168.122.1:3142/deb.debian.org/debian mv rootfs.ext4 rootfs-worker1.ext4 debvm-create --size=3GB --release=$RELEASE -- http://192.168.122.1:3142/deb.debian.org/debian mv rootfs.ext4 rootfs-worker2.ext4 debvm-run -i rootfs-manager.ext4 -- -netdev bridge,id=net1,br=virbr0 -device virtio-net-pci,netdev=net1,mac=52:54:00:12:34:00 -smp 2 debvm-run -i rootfs-worker1.ext4 -- -netdev bridge,id=net1,br=virbr0 -device virtio-net-pci,netdev=net1,mac=52:54:00:12:34:01 -smp 2 debvm-run -i rootfs-worker2.ext4 -- -netdev bridge,id=net1,br=virbr0 -device virtio-net-pci,netdev=net1,mac=52:54:00:12:34:02 -smp 2 In the example documented here, the machines have these network interfaces and addresses: .. code:: sh #virsh net-dhcp-leases default Expiry Time MAC address Protocol IP address Hostname Client ID or DUID ----------------------------------------------------------------------------------------------------------------------------------------------- 2023-08-01 09:41:00 52:54:00:12:23:01 ipv4 192.168.122.32/24 worker-1 ff:c2:72:f6:09:00:02:00:00:ab:11:18:aa:a0:aa:74:b7:84:be 2023-08-01 09:42:06 52:54:00:12:23:02 ipv4 192.168.122.33/24 worker-2 ff:c2:72:f6:09:00:02:00:00:ab:11:06:b4:b1:fd:ba:22:08:91 2023-08-01 09:44:12 52:54:00:12:45:56 ipv4 192.168.122.35/24 manager ff:c2:72:f6:09:00:02:00:00:ab:11:4a:c0:f0:69:cf:3d:9b:1e Docker ------ On each VM ^^^^^^^^^^^^^^^^ * Ajust their hostnames .. code:: sh hostnamectl set-hostname {manager,worker-1,worker-2} if `hostnamectl` is available (as in buster), or manually, otherwise. * Install docker.io .. code:: sh apt install -y docker.io * On bookworm and trixie, you need to aditionally install e2fsprogs .. code:: sh apt install -y e2fsprogs * Ensure the xt_u32 linux module is loaded. Needed for https://github.com/moby/moby/security/advisories/GHSA-33pg-m6jh-5237 .. code:: sh modprobe -v xt_u32 On manager: ^^^^^^^^^^^ * pull nginx (used to test) .. code:: sh docker image pull nginx * Init the swarm .. code:: sh docker swarm init --advertise-addr=$MANAGER_IP The output should give you the swarm token :: To add a worker to this swarm, run the following command: docker swarm join --token SWMTKN-1-4788k6slks7r8zc6v8vilr4m453v1uohv57s0ynbg95etkxj6o-cc7kalnbuo0xnbi44fzeubu7j 192.168.122.35:2377 In any case, it is possible to get the token later with: .. code:: sh docker swarm join-token worker * create a network (encrypted): .. code:: sh docker network create -d overlay --opt encrypted nginx-net On the runners: ^^^^^^^^^^^^^^^ .. code:: sh docker swarm join --token SWMTKN-1-4788k6slks7r8zc6v8vilr4m453v1uohv57s0ynbg95etkxj6o-cc7kalnbuo0xnbi44fzeubu7j 192.168.122.35:2377 On manager: ^^^^^^^^^^^^ * Workers should be visible: .. code:: sh docker node ls ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION kl59ohp9xlggbtktgtblwdth7 * manager Ready Active Leader 18.09.1 g2yc5o0k9162kggu9rc5nnakk worker-1 Ready Active 18.09.1 t0yhj5l6fl4i3k90lztw3oo0h worker-2 Ready Active 18.09.1 It may be useful to label workers for being able to apply service deployment constraints. .. code:: sh docker node update --label-add foo=bar worker-1 docker node update --label-add foo=bar worker-2 * create a service: .. code:: sh docker service create --name my-nginx --publish target=80,published=80 --replicas=3 --network nginx-net nginx Or better, with deployment constraints: .. code:: sh docker service create --name my-nginx --publish target=80,published=80 --replicas=2 --network nginx-net --constraint node.labels.foo==bar nginx * Verify the workers are serving the service: .. code:: sh docker service ps my-nginx The worker nodes should be listed in the Nodes column, and their status should be Running. * On the host, use *tcpcump* to sniff the traffic, e.g. on 4789/udp port (docker network overlay data) .. code:: sh sudo tcpdump -vvv --print "udp port 4789" -i virbr0 -w capture.pcap * Access the web site on the hosts IPs from the external network. Here: wget -O - http://192.168.122.35 References ---------- * https://docs.docker.com/engine/swarm/swarm-tutorial/create-swarm/ * https://docs.docker.com/engine/swarm/networking/ * https://docs.docker.com/network/drivers/overlay/ * https://docs.docker.com/config/daemon/ | Copyright (C) 2023 Santiago Ruano Rincón