Imagining a Device Provisioning Process

2026-03-07

This is a kind of follow up to a prior post about integrating hardware devices in a robust, repeatable way. Here I'm looking into ways of provisioning devices (computers) that are sometimes referred to as "industrial PCs", "IoT gateways", or just "on-site computers". I've been irked by how convoluted and messy the process can become when it feels like there is an 80% solution just laying in the building blocks of a standard Linux installation. To finally put these thoughts to rest I thought I'd try buidling such a system.

I'm interested in trying out mkosi after having seen it when I looked into portable services. It describes itself as:

A fancy wrapper around dnf --installroot, apt, pacman and zypper that generates customized disk images with a number of bells and whistles.

About the simplest possible configuration for a working mkosi is as such:

$ cat mkosi.conf 
[Distribution]
Distribution=fedora
Release=43

[Output]
Format=disk
OutputDirectory=mkosi.output
ImageId=device-golden-image

[Partitions]
MakeEFI=yes

[Content]
Bootable=yes
Packages=
    kernel-core
    kernel-modules
    systemd
    systemd-boot-unsigned
    libbpf
    openssh-server

Here I'm building a basic Fedora 43 image into a directory that doesn't yet exist, called mkosi.output. It will build a raw disk image at the end of the process which is bootable with qemu or systemd-nspawn.

$ tree
.
├── mkosi.conf
└── mkosi.output
    ├── device-golden-image -> device-golden-image.raw
    ├── device-golden-image.initrd
    ├── device-golden-image.raw
    └── device-golden-image.vmlinuz

2 directories, 5 files

This isn't a particularly useful machine image but seeing it boot is encouraging:

$ sudo systemd-nspawn -i mkosi.output/device-golden-image -b -x
...

$ machinectl list
MACHINE                              CLASS     SERVICE        OS VERSION ADDRESSES
device-golden-image-d94d89a78245feef container systemd-nspawn -  -       -        

1 machines listed.

The first issue I'd like to address is disk management in the face of this raw disk image. It should be apparent that it behooves me to keep the image a reasonable size, here it is around a gigabyte. Because it is a raw disk image it takes up all of that space on the host machine where I'm building it and will be transferred byte-for-byte to whatever media I intend.

[root@fedora ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb2       1.3G  474M  689M  41% /
devtmpfs        858M     0  858M   0% /dev
tmpfs           968M     0  968M   0% /dev/shm
efivarfs        256K  8.9K  243K   4% /sys/firmware/efi/efivars
tmpfs           388M  660K  387M   1% /run
tmpfs           1.0M   12K 1012K   2% /run/credentials/@system
tmpfs           1.0M     0  1.0M   0% /run/credentials/systemd-journald.service
tmpfs           968M     0  968M   0% /tmp
/dev/vda        1.0T  5.8M 1022G   1% /var/tmp
/dev/vdb1       511M  230M  282M  45% /boot
tmpfs           1.0M     0  1.0M   0% /run/credentials/getty@tty1.service
tmpfs           1.0M     0  1.0M   0% /run/credentials/serial-getty@hvc0.service
tmpfs           194M  4.0K  194M   1% /run/user/0

Launching the machine under qemu you can verify it is quite small. What I'd prefer is for the machine to resize to whatever the destination disk size is. In that way I can try to keep my images small but utilize all the resources available on the edge PC. This is achievable with systemd-repart, here's a definition I'm working with:

[Partition]
Type=root
Format=btrfs
SizeMinBytes=1024M
GrowFileSystem=yes

$ tree
.
├── mkosi.conf
├── mkosi.extra
│   └── usr
│       └── lib
│           └── repart.d
│               └── 10-root.conf
└── mkosi.output
    ├── device-golden-image -> device-golden-image.raw
    ├── device-golden-image.initrd
    ├── device-golden-image.raw
    └── device-golden-image.vmlinuz

Placing that into a mkosi.extra directory ensures it is loaded into the disk image when built and on first launch the operating system grows the filesystem to the full disk size. I checked this using qemu, first creating a "big" disk of 20G, dd-ing the contents of the golden machine image so it takes up the first ~1GB of space, and then loading that growable disk image under qemu:

$ qemu-img create -f raw mkosi.output/big-disk.img 20G
Formatting 'mkosi.output/big-disk.img', fmt=raw size=21474836480

$ dd if=mkosi.output/device-golden-image.raw of=mkosi.output/big-disk.img bs=4M conv=notrunc status=progress
1841299456 bytes (1.8 GB, 1.7 GiB) copied, 4 s, 460 MB/s
441+1 records in
441+1 records out
1851805696 bytes (1.9 GB, 1.7 GiB) copied, 4.09456 s, 452 MB/s

$ qemu-system-x86_64 -machine q35,accel=kvm -cpu host -m 2G -drive if=pflash,format=raw,readonly=on,file=/usr/share/edk2/ovmf/OVMF_CODE.fd -drive format=raw,file=mkosi.output/big-disk.img -nographic
...
[root@fedora ~]# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda      8:0    0   20G  0 disk
├─sda1   8:1    0  512M  0 part /boot
└─sda2   8:2    0 19.5G  0 part /
sr0     11:0    1 1024M  0 rom

Let's Include Some Software

While I could create or use a container and define a service to run podman or similar, I thought it might be more interesting to return to a slightly more familiar path to developing and deploying software on linux and that is to just use the package management facilities of the distribution. It has been years since I last used .deb packages for work but I don't remember it being especially difficult. I'm using Fedora for this example so I'll take the opportunity to see what creating an RPM locally is like. So as to not get too ambitious I'm going to try packaging up the arduino-monitor script I wrote about previously and include the udev rules and service files:

full file listing

./sources/99-arduino-monitor.rules

SUBSYSTEM=="tty", \
  ATTRS{idVendor}=="2341", \
  ATTRS{idProduct}=="0043", \
  ATTRS{manufacturer}=="Arduino (www.arduino.cc)", \
  TAG+="systemd", \
  ENV{SYSTEMD_WANTS}="arduino-monitor@%k.service"

./sources/arduino-monitor

#!/usr/bin/env tclsh

set port     [lindex $argv 0]
set baudrate 9600

proc sysfs_attr {path attr} {
    set f [file join $path $attr]
    if {![file readable $f]} { return "" }
    set fd [open $f r]
    set val [string trim [read $fd]]
    close $fd
    return $val
}

proc print_device_info {device_name} {
    set tty_device [file join /sys/class/tty $device_name device]
    set resolved [exec readlink -f $tty_device]
    set usb_path [file dirname $resolved]

    puts "--- device: /dev/$device_name ---"
    foreach attr {idVendor idProduct manufacturer serial} {
        set val [sysfs_attr $usb_path $attr]
        if {$val ne ""} { puts "  $attr: $val" }
    }
    puts "---"
}

set device_name [file tail $port]
print_device_info $device_name

set fd [open $port r+]
fconfigure $fd \
    -mode     "$baudrate,n,8,1" \
    -handshake none             \
    -buffering line             \
    -translation auto           \
    -blocking  0

proc readLine {fd} {
    if {[eof $fd]} {
        puts stderr "serial port closed."
        close $fd
        exit 0
    }
    set line [gets $fd]
    if {$line ne ""} { puts $line }
}

fileevent $fd readable [list readLine $fd]
puts "listening on $port at $baudrate baud..."
vwait forever

./sources/arduino-monitor@.service

[Unit]
Description=Arduino Uno monitor on /dev/%i
BindsTo=dev-%i.device
After=dev-%i.device

[Service]
Type=simple
ExecStart=/usr/bin/arduino-monitor /dev/%i
Restart=no
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=dev-%i.device

./arduino-monitor.spec

Name:           arduino-monitor
Version:        0.1
Release:        1%{?dist}
Summary:        arduino uno serial monitor daemon
License:        none
BuildArch:      noarch
BuildRequires:  systemd-rpm-macros
Requires:       tcl

%description
Monitors an arduino uno over serial and logs output to the journal.
Triggered automatically via udev when the device is connected.

%prep

%build

%install
install -Dm755 %{_sourcedir}/arduino-monitor \
    %{buildroot}%{_bindir}/arduino-monitor

install -Dm644 %{_sourcedir}/99-arduino-monitor.rules \
    %{buildroot}%{_prefix}/lib/udev/rules.d/99-arduino-monitor.rules

install -Dm644 %{_sourcedir}/arduino-monitor@.service \
    %{buildroot}%{_unitdir}/arduino-monitor@.service

%post
%systemd_post arduino-monitor@.service

%preun
%systemd_preun arduino-monitor@.service

%postun
%systemd_postun arduino-monitor@.service

%files
%{_bindir}/arduino-monitor
%{_prefix}/lib/udev/rules.d/99-arduino-monitor.rules
%{_unitdir}/arduino-monitor@.service

Building the package is a small matter of: rpmbuild -bb --define "_sourcedir $(pwd)/sources" --define "_rpmdir $(pwd)/rpms" arduino-monitor.spec which is just a little long to circumvent the default sources and rpms directory and include them in the same directory as the spec file for simplicity.

With an RPM built I can include the package in my mkosi definition:


$ cat mkosi.conf 
[Distribution]
Distribution=fedora
Release=43

[Output]
Format=disk
OutputDirectory=mkosi.output
ImageId=device-golden-image

[Partitions]
MakeEFI=yes

[Content]
KernelCommandLine=console=ttyS0
Bootable=yes
Packages=
    kernel-core
    kernel-modules
    systemd
    systemd-boot-unsigned
    libbpf
    openssh-server
    arduino-monitor

And now I can build my machine image with some custom software loaded on it:

$ mkosi --package-dir=/home/nolan/sources/arduino-monitor/rpms/noarch/

A Bootable Machine Image in Action

Now, I'll admit I am not exactly an expert at qemu. I spent quite a while trying to untangle StackOverflow answers and the documentation to explain the dependencies to achieve USB passthrough to a VM from the host and what I came up with is this:

$ sudo qemu-system-x86_64 -enable-kvm -m 1024 -machine q35 \
-drive if=pflash,format=raw,readonly=on,file=/usr/share/edk2/ovmf/OVMF_CODE.fd \
-usb -device usb-host,hostbus=1,hostport=3 \
mkosi.output/big-disk.img

Which I freely admit looks a bit awful. I sort of understand the need to change the default BIOS due to the UKI that mkosi is producing but don't really know what the OVMF_CODE.fd is or does, and I think I understand how the specific USB device is named from the output of lsusb... but I don't really get why I need sudo. It isn't critical because I'm not especially interested in qemu at the moment but what it does allow me to verify is that it worked!

[root@fedora ~]# cat /proc/cmdline 
initrd=\device-golden-image\initrd initrd=\device-golden-image\6.18.16-200.fc43.x86_64\kernel-modules.initrd console=ttyS0

[root@fedora ~]# systemctl status arduino-monitor@ttyACM0
● arduino-monitor@ttyACM0.service - Arduino Uno monitor on /dev/ttyACM0
     Loaded: loaded (/usr/lib/systemd/system/arduino-monitor@.service; d
isabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: active (running) since Sat 2026-03-07 23:37:44 UTC; 54s ago
 Invocation: 110784e70260457f9151523606994b75
   Main PID: 468 (arduino-monitor)
      Tasks: 1 (limit: 869)
     Memory: 3.6M (peak: 3.6M)
        CPU: 16ms
     CGroup: /system.slice/system-arduino\x2dmonitor.slice/arduino-monitor@ttyACM0.service
             └─468 /usr/bin/tclsh /usr/bin/arduino-monitor /dev/ttyACM0

Mar 07 23:38:28 fedora arduino-monitor[468]: 422
Mar 07 23:38:29 fedora arduino-monitor[468]: 422
Mar 07 23:38:30 fedora arduino-monitor[468]: 362
Mar 07 23:38:31 fedora arduino-monitor[468]: 287
Mar 07 23:38:32 fedora arduino-monitor[468]: 362
Mar 07 23:38:33 fedora arduino-monitor[468]: 362
Mar 07 23:38:34 fedora arduino-monitor[468]: 499
Mar 07 23:38:35 fedora arduino-monitor[468]: 361
Mar 07 23:38:36 fedora arduino-monitor[468]: 426
Mar 07 23:38:37 fedora arduino-monitor[468]: 361

So we've got a machine image generation process that integrates some custom software that triggers based on hardware interaction (USB plug in event), and it grows the machine partition automatically to utilize the full disk capacity of the target machine. That's feeling pretty good so far!