Armada

The ansible scripts used to maintain the fleet of machines for my labgrid lab.

Adding Inventory

While these ansible scripts help us maintain the fleet they do expect a few things before a host can be properly on-boarded.

• These scripts assume the host is a stable debian based system
• The ansible user must already exist on the system
• The host's ssh keys must be signed by the lab's CA
• The host must be publicly accessable by the lab's jumpbox

Debian Based System

Live images of Debian can be found here: https://www.debian.org/CD/live/

During the install of debian:

• The domain for the lab is lab.0x42.sh with most hosts having a random 10 alpha-numeric string. For example 29j2vurmjy.lab.0x42.sh unless the host is expected to be public facing.

• These scripts will use the twig user to connect to the hosts using a signed ssh key by the lab's CA. (remote_user property in ansible.cfg)

• For ansible to run on this new host some additional packages must be installed. Currently these packages are:

  • python3
  • sudo

Initial SSH Access

To keep key/password management to a minimum the lab is configured to work with CA certificates to sign and manage ssh keys. So long as the host is configured to accept the signed ssh keys these scripts will be able to connect to the host.

First copy over the lab's public CA key to the new host:

scp /path/to/lab-ca.pub twig@[hostname].lab.0x42.sh:/etc/ssh/

These scripts will also only connect to hosts with keys signed by the lab's CA. For new hosts this command can be used to sign each of the host's ssh keys

scp twig@[hostname].lab.0x42.sh:/etc/ssh/ssh_host_rsa_key.pub .
sudo ssh-keygen -s /path/to/lab-ca \
    -I [hostname].lab.0x42.sh \
    -h -n [hostname].lab.0x42.sh \
    -V +52w /path/to/ssh_host_rsa_key.pub
scp /path/to/ssh_host_rsa_key-cert.pub twig@[hostname].lab.0x42.sh:/etc/ssh/

Then, on the host system, add the needed settings to configure sshd:

echo "TrustedUserCAKeys /etc/ssh/lab-ca.pub" >> /etc/ssh/sshd_config"
echo "HostCertificate /etc/ssh/lab_host_rsa_key-cert.pub" >> /etc/ssh/sshd_config"

Inspecting SSH Keys

Because all SSH keys in the lab have an expiration date it's often useful to check when the certificates will expire. We can use ssh-keygen to dump all information about the public SSH key.

ssh-keygen -L -f /path/to/ssh_key_cert.pub
/path/to/ssh_key_cert.pub:
        Type: ssh-ed25519-cert-v01@openssh.com user certificate
        Public key: ED25519-CERT SHA256:AvMWvmZ1vmVHfHV1IU6AJ2GpOZaFo9S/qrQ31QWtBHE
        Signing CA: ED25519 SHA256:yYVS991YQgEuq9baRicgAVtOD+svDToxYKn0uhF3Vt8 (using ssh-ed25519)
        Key ID: "some cool id"
        Serial: 0
        Valid: from 2026-01-19T00:53:00 to 2026-04-19T00:54:16
        Principals:
                nobody
        Critical Options: (none)
        Extensions:
                permit-X11-forwarding
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc

Install Tailscale

All of the exporters are located behind firewalls to reduce their exposure to bots. A publicly accessable jumpbox which then uses tailscale to tunnel through the firewall to the host allows us to connect to these hosts living behind these firewalls.

To install tailscale on the debian system run:

apt upgrade curl
curl -fsSL https://tailscale.com/install.sh | sh

Next we add it to the tailscale network:

sudo tailscale up

From here we need a signing node to add it to the tailscale network. From there we can add the lab ACL tag to give it access to the lab network.