Documenting a simple system for process management in the context of integrating hardware into a linux environment in the face of multiple devices and hot-plugging.
This whole exercise is prompted by a low grade annoyance I've had for years with how to manage a kind of internet of things configuration of devices on edge computers. I'll skip all the gory details and instead focus on some high level context and goals.
That's it! I expect you're thinking "that doesn't sound so hard" and you are right! It was the idea that this shouldn't be so hard that made so many convoluted solutions so painful.
Instead of getting too into the weeds of any particular piece of hardware or protocol I'm going grab some Arduino Uno R3 devices I have lying around and crib an example from the Arduino IDE. I'll perform an analog read of a pin (A0), here with nothing attached to it, and write it out to a 9600 baud serial line with a newline terminator. The board is in fact totally empty so the pin is reading a floating voltage that will drift around. This is sufficiently interesting and requires no real explaining, you get to use your imagination for what it would look like to decode a real protocol with actual useful information in it.
void setup() {
// initialize serial communication at 9600 bits per second:
Serial.begin(9600);
}
void loop() {
// read the input on analog pin 0
int sensorValue = analogRead(A0);
Serial.println(sensorValue);
delay(1000);
}
Plugging in one of these boards to my laptop produces a bit of
chatter that can be seen in dmesg:
[ 9016.084542] usb 1-3: new full-speed USB device number 41 using xhci_hcd
[ 9016.244959] usb 1-3: New USB device found, idVendor=2341, idProduct=0043, bcdDevice= 0.01
[ 9016.244970] usb 1-3: New USB device strings: Mfr=1, Product=2, SerialNumber=220
[ 9016.244976] usb 1-3: Manufacturer: Arduino (www.arduino.cc)
[ 9016.244980] usb 1-3: SerialNumber: 9543334393335151F0D2
[ 9016.290954] cdc_acm 1-3:1.0: ttyACM0: USB ACM device
that ttyACM0 is the character device node that becomes
available on connect, you can find it under the device tree
(/dev/ttyACM0). You can also find the device under
the sys file system with the data that it carries from
the device itself. Getting at that is a little circuitous because of
the symlinking involved in the sys tree but can be seen with the
actual location here:
$ ls -al /sys/class/tty/ttyACM0/device
lrwxrwxrwx. 1 root root 0 Feb 28 19:14 /sys/class/tty/ttyACM0/device -> ../../../1-3:1.0
$ readlink -f /sys/class/tty/ttyACM0/device
/sys/devices/pci0000:00/0000:00:08.1/0000:05:00.3/usb1/1-3/1-3:1.0
$ cat $(readlink -f /sys/class/tty/ttyACM0/device)/../manufacturer
Arduino (www.arduino.cc)
Part of that indirection becomes obvious when you start to plug in
more devices. The second device plugged in would be
named ttyACM1 but the destination depends on the
physical USB port used. On my laptop 1-3 is the furthest port on the
left hand side for example.
So let's contrive a program that should read device information out when it is plugged in, you might imagine this is for monitoring or some kind of inventory tracking. After it has been plugged in the device will be doing work and producing data to be read off the serial communication line. I will, of course, be using TCL for this because it is simply delightful how easy it is to read a serial device compared to something like Python unless you use a third-party package like pyserial:
#!/usr/bin/env tclsh
set port [lindex $argv 0]
set baudrate 9600
proc sysfs_attr {path attr} {
set f [file join $path $attr]
if {![file readable $f]} { return "" }
set fd [open $f r]
set val [string trim [read $fd]]
close $fd
return $val
}
proc print_device_info {device_name} {
set tty_device [file join /sys/class/tty $device_name device]
set resolved [exec readlink -f $tty_device]
set usb_path [file dirname $resolved]
puts "--- device: /dev/$device_name ---"
foreach attr {idVendor idProduct manufacturer serial} {
set val [sysfs_attr $usb_path $attr]
if {$val ne ""} { puts " $attr: $val" }
}
puts "---"
}
set device_name [file tail $port]
print_device_info $device_name
set fd [open $port r+]
fconfigure $fd \
-mode "$baudrate,n,8,1" \
-handshake none \
-buffering line \
-translation auto \
-blocking 0
proc readLine {fd} {
if {[eof $fd]} {
puts stderr "serial port closed."
close $fd
exit 0
}
set line [gets $fd]
if {$line ne ""} { puts $line }
}
fileevent $fd readable [list readLine $fd]
puts "listening on $port at $baudrate baud..."
vwait forever
With that saved someplace plausible and marked executable it is possible to test it out with one of these junk-data-generators plugged in:
$ /usr/local/bin/arduino_monitor /dev/ttyACM0
--- Device: /dev/ttyACM0 ---
idVendor: 2341
idProduct: 0043
manufacturer: Arduino (www.arduino.cc)
serial: 55730323831351806100
---
Listening on /dev/ttyACM0 at 9600 baud...
406
1
397
394
3388
387
385
...
So far, so easy. Now to integrate it into the process management system I want to hook the plug-in event using udev. The idea is that I want to act only when one of these specific devices is plugged in because there's little guarantee what else might be running or temporarily attached to the machine. The way udev works this is quite easy with a rule definition based on those attributes enumerated above:
$ cat /etc/udev/rules.d/99-arduino-monitor.rules
SUBSYSTEM=="tty", \
ATTRS{idVendor}=="2341", \
ATTRS{idProduct}=="0043", \
ATTRS{manufacturer}=="Arduino (www.arduino.cc)", \
TAG+="systemd", \
ENV{SYSTEMD_WANTS}="arduino-monitor@%k.service"
Of course you may question whether that information is really
available or how I know what to call it. It's a simple thing to
query it out with udevadm:
$ udevadm info -n /dev/ttyACM0
P: /devices/pci0000:00/0000:00:08.1/0000:05:00.3/usb1/1-4/1-4:1.0/tty/ttyACM0
M: ttyACM0
R: 0
J: c166:0
U: tty
D: c 166:0
N: ttyACM0
L: 0
S: serial/by-id/usb-Arduino__www.arduino.cc__0043_55730323831351806100-if00
S: serial/by-path/pci-0000:05:00.3-usb-0:4:1.0
S: serial/by-path/pci-0000:05:00.3-usbv2-0:4:1.0
E: DEVPATH=/devices/pci0000:00/0000:00:08.1/0000:05:00.3/usb1/1-4/1-4:1.0/tty/ttyACM0
E: DEVNAME=/dev/ttyACM0
E: MAJOR=166
E: MINOR=0
E: SUBSYSTEM=tty
E: USEC_INITIALIZED=14408635103
E: ID_BUS=usb
E: ID_MODEL=0043
E: ID_MODEL_ENC=0043
E: ID_MODEL_ID=0043
E: ID_SERIAL=Arduino__www.arduino.cc__0043_55730323831351806100
E: ID_SERIAL_SHORT=55730323831351806100
E: ID_VENDOR=Arduino__www.arduino.cc_
E: ID_VENDOR_ENC=Arduino\x20\x28www.arduino.cc\x29
E: ID_VENDOR_ID=2341
E: ID_REVISION=0001
E: ID_TYPE=generic
E: ID_USB_MODEL=0043
E: ID_USB_MODEL_ENC=0043
E: ID_USB_MODEL_ID=0043
E: ID_USB_SERIAL=Arduino__www.arduino.cc__0043_55730323831351806100
E: ID_USB_SERIAL_SHORT=55730323831351806100
E: ID_USB_VENDOR=Arduino__www.arduino.cc_
E: ID_USB_VENDOR_ENC=Arduino\x20\x28www.arduino.cc\x29
E: ID_USB_VENDOR_ID=2341
E: ID_USB_REVISION=0001
E: ID_USB_TYPE=generic
E: ID_USB_INTERFACES=:020201:0a0000:
E: ID_USB_INTERFACE_NUM=00
E: ID_USB_DRIVER=cdc_acm
E: ID_USB_CLASS_FROM_DATABASE=Communications
E: ID_VENDOR_FROM_DATABASE=Arduino SA
E: ID_MODEL_FROM_DATABASE=Uno R3 (CDC ACM)
E: ID_PATH_WITH_USB_REVISION=pci-0000:05:00.3-usbv2-0:4:1.0
E: ID_PATH=pci-0000:05:00.3-usb-0:4:1.0
E: ID_PATH_TAG=pci-0000_05_00_3-usb-0_4_1_0
E: ID_MM_CANDIDATE=1
E: SYSTEMD_WANTS=arduino-monitor@ttyACM0.service
E: DEVLINKS=/dev/serial/by-id/usb-Arduino__www.arduino.cc__0043_55730323831351806100-if00 /dev/serial/by-path/pci-0000:05:00.3-usb-0:4:1.0 /dev/serial/by-path/pci-0000:05:00.3-usbv2-0:4:1.0
E: TAGS=:systemd:
E: CURRENT_TAGS=:systemd:
The rule
specifies ENV{SYSTEMD_WANTS}="arduino-monitor@%k.service"
which is a nifty integration into the service management of
systemd. The %k is a template or parameter to be
substituted with a value from the device that matched the rule,
specifically it will be what is
(confusingly) referred
to as the "kernel name" for the device, in this example
"ttyACM0". Additionally, the "TAGS" will cause systemd to create
a .device for this matched device, which pretty much
supplies everything needed to tie together the process
management. Writing a service file that binds to that device file
looks like this:
$ cat /etc/systemd/system/arduino-monitor@.service
[Unit]
Description=arduino uno monitor on /dev/%i
BindsTo=dev-%i.device
After=dev-%i.device
[Service]
Type=simple
ExecStart=/usr/local/bin/arduino_monitor /dev/%i
Restart=no
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=dev-%i.device
A slight systemd quirk is the substitution of / values with hyphens
(which I've dealt with
before) that turns dev/ttyACM0
into dev-ttyACM0 when naming systemd things like
devices and mounts.
With the service file added and the udev rule loaded (udevadm
control --reload-rules) things pretty much just work to
launch the process on plug in:
$ journalctl -f -u arduino-monitor@ttyACM*
Feb 28 20:04:35 eta systemd[1]: Started arduino-monitor@ttyACM0.service - arduino uno monitor on /dev/ttyACM0.
Feb 28 20:04:35 eta arduino_monitor[37433]: --- Device: /dev/ttyACM0 ---
Feb 28 20:04:35 eta arduino_monitor[37433]: idVendor: 2341
Feb 28 20:04:35 eta arduino_monitor[37433]: idProduct: 0043
Feb 28 20:04:35 eta arduino_monitor[37433]: manufacturer: Arduino (www.arduino.cc)
Feb 28 20:04:35 eta arduino_monitor[37433]: serial: 55730323831351806100
Feb 28 20:04:35 eta arduino_monitor[37433]: ---
Feb 28 20:04:35 eta arduino_monitor[37433]: Listening on /dev/ttyACM0 at 9600 baud...
Feb 28 20:04:37 eta arduino_monitor[37433]: 433
Feb 28 20:04:38 eta arduino_monitor[37433]: 426
Feb 28 20:04:39 eta arduino_monitor[37433]: 419
Feb 28 20:04:40 eta arduino_monitor[37433]: 414
Feb 28 20:04:41 eta arduino_monitor[37433]: 410
Feb 28 20:04:42 eta arduino_monitor[37433]: 406
Feb 28 20:04:43 eta arduino_monitor[37433]: 404
More interesting to me though is plugging in a second device:
Feb 28 20:06:46 eta systemd[1]: Started arduino-monitor@ttyACM1.service - arduino uno monitor on /dev/ttyACM1.
Feb 28 20:06:47 eta arduino_monitor[37665]: --- Device: /dev/ttyACM1 ---
Feb 28 20:06:47 eta arduino_monitor[37665]: idVendor: 2341
Feb 28 20:06:47 eta arduino_monitor[37665]: idProduct: 0043
Feb 28 20:06:47 eta arduino_monitor[37665]: manufacturer: Arduino (www.arduino.cc)
Feb 28 20:06:47 eta arduino_monitor[37665]: serial: 9543334393335151F0D2
Feb 28 20:06:47 eta arduino_monitor[37665]: ---
Feb 28 20:06:47 eta arduino_monitor[37665]: Listening on /dev/ttyACM1 at 9600 baud...
Feb 28 20:06:47 eta arduino_monitor[37433]: 377
Feb 28 20:06:48 eta arduino_monitor[37665]: 378
Feb 28 20:06:49 eta arduino_monitor[37433]: 378
Feb 28 20:06:50 eta arduino_monitor[37665]: 380
Feb 28 20:06:51 eta arduino_monitor[37433]: 380
Feb 28 20:06:52 eta arduino_monitor[37665]: 380
Feb 28 20:06:53 eta arduino_monitor[37433]: 380
Feb 28 20:06:54 eta arduino_monitor[37433]: 379
$ systemctl list-units arduino-monitor@*
UNIT LOAD ACTIVE SUB DESCRIPTION
arduino-monitor@ttyACM0.service loaded active running arduino uno monitor on /dev/ttyACM0
arduino-monitor@ttyACM1.service loaded active running arduino uno monitor on /dev/ttyACM1
I didn't do a great job differentiating the log lines so you have to squint to see that I'm logging two different services (seen in the process IDs) and they're both producing data at once. Unplugging one of them looks like this:
Feb 28 20:06:51 eta arduino_monitor[37433]: 380
Feb 28 20:06:52 eta arduino_monitor[37665]: 380
Feb 28 20:06:53 eta arduino_monitor[37433]: 380
Feb 28 20:06:54 eta arduino_monitor[37433]: 379
Feb 28 20:06:57 eta systemd[1]: arduino-monitor@ttyACM0.service: Deactivated successfully.
$ systemctl list-units arduino-monitor@*
UNIT LOAD ACTIVE SUB DESCRIPTION
arduino-monitor@ttyACM1.service loaded active running arduino uno monitor on /dev/ttyACM1
Plugging in the device launches the service and unplugging the device shuts it down cleanly. I won't belabor the point but unplugging and replugging a bunch more times but you can trust that I've done it dozens of time in writing this and delighted in watching it all just work.