Monitor changes to paths and files using systemd

Introduction

It's common to want to take some action when a specific file or any file under a path or directory has changed. Perhaps a configuration needs to be adjusted every time some file changes (maybe you want to append a custom section to a config file managed by tiup), or perhaps you want to keep track of all changes to a file under some specific directory (maybe you want to turn a tiup cluster definition into a git repository and automatically track all changes to the cluster definition).

Historically, Linux has used inotify to monitor for changes to paths and files. The inotify kernel API can be used by programs that want to be notified about changes to files. To use the API, a separate userspace program must run to request notification from the API. This means installing and maintaining additional tools.

Fortunately, systemd includes native support for "path" units. A path unit tells systemd to monitor a specific path for changes, and when any changes are detected an associated "service unit" is executed. See https://www.freedesktop.org/software/systemd/man/systemd.path.html for the full definition of path unit syntax, and https://www.freedesktop.org/software/systemd/man/systemd.service.html for service unit syntax.

systemd concepts

In systemd, there is a system service manager that's responsible for starting system services. This is used by tiup to manage the services that comprise a TiDB cluster. Look at the *.service files in the config-cache directory of your cluster. For example:

$ ls ~/.tiup/storage/cluster/clusters/test/config-cache/*.service
/home/ubuntu/.tiup/storage/cluster/clusters/test/config-cache/blackbox_exporter-127.0.0.1-9115.service
/home/ubuntu/.tiup/storage/cluster/clusters/test/config-cache/grafana-127.0.0.1-3000.service
/home/ubuntu/.tiup/storage/cluster/clusters/test/config-cache/node_exporter-127.0.0.1-9100.service
/home/ubuntu/.tiup/storage/cluster/clusters/test/config-cache/pd-127.0.0.1-2379.service
/home/ubuntu/.tiup/storage/cluster/clusters/test/config-cache/prometheus-127.0.0.1-9090.service
/home/ubuntu/.tiup/storage/cluster/clusters/test/config-cache/tidb-127.0.0.1-4000.service
/home/ubuntu/.tiup/storage/cluster/clusters/test/config-cache/tiflash-127.0.0.1-9000.service
/home/ubuntu/.tiup/storage/cluster/clusters/test/config-cache/tikv-127.0.0.1-20160.service

$ cat ~/.tiup/storage/cluster/clusters/test/config-cache/tidb-127.0.0.1-4000.service
[Unit]
Description=tidb service
After=syslog.target network.target remote-fs.target nss-lookup.target

[Service]
LimitNOFILE=1000000
LimitSTACK=10485760
User=ubuntu
ExecStart=/home/ubuntu/.tiup/cluster/tidb-deploy/tidb-4000/scripts/run_tidb.sh
Restart=always

RestartSec=15s

[Install]
WantedBy=multi-user.target

$ systemctl status tidb-4000.service
● tidb-4000.service - tidb service
     Loaded: loaded (/etc/systemd/system/tidb-4000.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2021-10-05 16:52:11 UTC; 1h 8min ago
   Main PID: 35109 (tidb-server)
      Tasks: 14 (limit: 37560)
     Memory: 45.6M
     CGroup: /system.slice/tidb-4000.service
             └─35109 bin/tidb-server -P 4000 --status=10080 --host=0.0.0.0 --advertise-address=127.0.0.1 --store=tikv --path=127.0.0.1:2379 --log-slow-query=/home/ubuntu/.tiup/cluster/tidb-deploy/t>

Oct 05 16:52:11 ip-10-0-81-2 systemd[1]: Started tidb service.

systemd user service manager

Systemd also has a user service manager for each user that logs in to the system. This manages certain aspects of their terminal sessions, windowing system, and other miscellaneous things. An unprivileged user can create their own systemd units that are managed by their user's service manager, but an unprivileged user cannot create units that are managed by the system service manager — that requires sudo/root access.

Systemd user service manager unit files are stored in ~/.config/systemd/user/, so make sure that directory exists:

mkdir -p ~/.config/systemd/user/

Note: By default, the user service manager only runs when a user is logged in!

If a specific user's service manager should run when the user is not logged in, that can be achieved by enabling "linger" for that user using loginctl: loginctl enable-linger ubuntu. This is only required if you need a user's systemd services to be active even when the user is not actively logged in. We don't need that for this task, because we can assume that people will be logged in when executing tiup. If you want to improve resiliency, you could enable linger for the user or implement this as a system service so that a nefarious agent cannot disable the service.

systemd path units

Systemd has a fundamental concept of a "unit". A systemd unit file is just a text file that tells systemd what to do.

There are a number of different kinds of systemd units. For example, services (these files end in .service), timers (they end in .timer, and you can use these instead of cron), paths (these end in .path), sockets, mounts, etc.

Read more about the details of the generic "unit" concept and find links to the different types of units at https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Description.

This article will focus on the path unit, and will also use an associated service unit.

git + tiup

Git is the obvious choice for tracking the revision history of a set of files. We can turn the tiup cluster definition directory into a git repository. It's as easy as git init/add/commit, for a cluster named "test":

git -C ~/.tiup/storage/cluster/clusters/test/ init
git -C ~/.tiup/storage/cluster/clusters/test/ add .
git -C ~/.tiup/storage/cluster/clusters/test/ commit -m 'initial commit'

This doesn't mean that changes made by tiup will automatically be added and committed to the git repository; for that we will rely on a systemd path unit to monitor changes to this path and execute a systemd service that executes git commands.

tiup + systemd path unit

First, let's create a systemd path unit to monitor the directory that holds the tiup data about our cluster. As above, make sure that the ~/.config/systemd/user/ directory exists. Then create a file ~/.config/systemd/user/tiup-test.path:

[Path]
PathModified=/home/ubuntu/.tiup/storage/cluster/clusters/test/

[Install]
WantedBy=default.target

The PathModified directive tells systemd to monitor any changes to the path under which tiup stores the configuration and access data for the "test" cluster. But this doesn't yet tell systemd to do anything when the path is modified. "By default, a service by the same name as the path (except for the suffix) is activated", so we will create a matching tiup-test.service file:

[Unit]
Description=tiup test cluster config monitor

[Service]
Type=oneshot
WorkingDirectory=/home/ubuntu/.tiup/storage/cluster/clusters/test/
ExecStart=git add .
ExecStart=git commit -m 'automatic commit from changed path'

Any time you make changes to systemd unit files, you should execute systemd daemon-reload. Because we have created unit files that are managed by the systemd user service manager, we invoke systemctl with the --user flag:

systemctl --user daemon-reload

Now we should see these unit files in the output of list-unit-files:

$ systemctl --user list-unit-files tiup-\*
UNIT FILE         STATE    VENDOR PRESET
tiup-test.path    disabled enabled
tiup-test.service static   enabled

2 unit files listed.

Note that the tiup-test.path unit is disabled. We must enable this unit file so that systemd begins monitoring the path:

$ systemctl --user enable --now tiup-test.path
$ systemctl --user list-unit-files tiup-\*
UNIT FILE         STATE   VENDOR PRESET
tiup-test.path    enabled enabled
tiup-test.service static  enabled

2 unit files listed.

Now that our path unit is enabled, we can do a simple test by creating an empty file under the monitored path:

$ touch ~/.tiup/storage/cluster/clusters/test/empty
$ git -C ~/.tiup/storage/cluster/clusters/test/ log --summary
commit 2db237b92973cad0fb60d1060df6af10eea1b204 (HEAD -> master)
Author: Ubuntu <ubuntu@ip-10-0-81-2.us-west-2.compute.internal>
Date:   Tue Oct 5 18:48:42 2021 +0000

    automatic commit from changed path

 create mode 100644 empty

commit c33170248d56580f6f2d96d4b45138a1df99e625
Author: Ubuntu <ubuntu@ip-10-0-81-2.us-west-2.compute.internal>
Date:   Tue Oct 5 18:19:08 2021 +0000

    initial commit
...

We can see that the new file empty has been added to the git repository, which confirms that our path and service units work as expected. Now let's try a more complex tiup operation to see how that works with this new setup.

$ cat scale-out.yaml
tiflash_servers:
  - host: 127.0.0.1
$ tiup cluster scale-out test scale-out.yaml --yes
Starting component `cluster`: /home/ubuntu/.tiup/components/cluster/v1.5.6/tiup-cluster scale-out test scale-out.yaml --yes
...
Scaled cluster `test` out successfully

$ git -C ~/.tiup/storage/cluster/clusters/test/ log --name-status
commit fdd1575404781bd87c12c1b649912982fd02f7dc (HEAD -> master)
Author: Ubuntu <ubuntu@ip-10-0-81-2.us-west-2.compute.internal>
Date:   Tue Oct 5 18:55:45 2021 +0000

    automatic commit from changed path

A       backup/meta-2021-10-05T18:55:45.213720986Z.yaml
M       meta.yaml

s3 bucket versioning

As an alternative to managing a local git repository for tiup cluster metadata, you could set up your systemd path unit & service to sync cluster metadata to an s3 bucket. An s3 bucket can optionally have "bucket versioning" enabled. Read more about s3 bucket versioning at https://docs.aws.amazon.com/AmazonS3/latest/userguide/manage-versioning-examples.html.

You can have your systemd service use the aws s3 sync command instead of git to sync the contents of the tiup cluster metadata to the s3 bucket, like this:

  aws s3 sync --quiet ~/.tiup/storage/cluster/clusters/test/ s3://<bucket>/tiup-test/

Summary

In order to track changes made to cluster topology configurations via tiup (or via direct manipulation of tiup's cluster topology metadata), we turned the tiup cluster metadata directory into a git repository. Then we created a pair of system unit files to monitor the path and automatically commit changes to the git repository.

A further extension to this approach could be to add a remote for the git repository and automatically push changes to the remote so that they are not lost if the tiup management node is destroyed.