systemd

systemd is "a suite of basic building blocks for a Linux system. It provides a system and service manager that runs as PID 1 and starts the rest of the system."

It contains an init process, essentially the first process launched by the Linux Kernel as a daemon, which continues running until the computer is shut down.

It can launch services, including via d-bus and on-demand upon a socket connection, a logging daemon, utilities to control system options like the hostname, date and time, network options, and can schedule tasks, among other features. When it first came out as a replacement to the classic SysVinit, it was fairly controversial due to certain design decisions like logging in a binary format rather than plain-text, but it is now widely adopted, and enabled by default on Arch Linux and Ubuntu.

It also provides journald, a logging daemon that replaces the syslog daemon.

Units

Unit files, ini-style text files which commonly end in .service for services, .socket for sockets, .timer for timers, etc. are used to define units. You may occasionally need to create new unit files, or modify existing ones, but a lot of packages come with reasonable defaults. You can list the loaded units with the systemctl list-units command.

You can see the defined unit files with the systemctl list-unit-files command.

As you can see, units may include:

Automounts
Mounts
Paths
Scopes
Services
Slices
Sockets
Swap
Targets
Timers

We will not be covering all of these, but you can read their respective manpages, e.g. with man systemd.automount for automounts, or man systemd.slice for cgroup slices.

You can get the status of a unit with the systemctl status command.

$ systemctl status sshd.service
● sshd.service - OpenSSH Daemon
     Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; preset: disabled)
     Active: active (running) since ...

Note: you need to pass the complete unit name including the extension, but if you omit it, it will default to .service. In this case, systemctl status sshd would be equivalent, but it would not work for other types of units.

$ systemctl status dbus.socket
● dbus.socket - D-Bus System Message Bus Socket
     Loaded: loaded (/usr/lib/systemd/system/dbus.socket; static)
     Active: active (running) since ...

Services

Services, sometimes known as daemons, are programs that run in the background. Some common services include:

bluetooth.service: used to connect devices via Bluetooth on desktops and laptops.
dbus-broker.service: the desktop message bus used to pass messages from one application to another. This is useful for desktop notifications, among other things.
NetworkManager.service: the network manager daemon.
sshd.service: the SSH daemon.
systemd-*.service: many services used by systemd itself.

Starting and stopping services

Services can be started and stopped with the systemctl command. For example, you could start, restart, and stop the sshd service with the following commands:

$ sudo systemctl start sshd.service
$ sudo systemctl restart sshd
$ sudo systemctl stop sshd

To start services automatically on boot, you can use the systemctl enable and systemctl disable commands. This will have no effect until you restart the machine, but you can add the --now flag to also start or stop the service right away. As such, systemctl enable sshd followed by systemctl start sshd can be accomplished with one command, systemctl enable --now sshd.

Important units for Django

Some of the most important units for our use case are:

docker.service: the Docker daemon, used for running containers.
gunicorn.service: the Gunicorn WSGI server.
gunicorn.socket: the Gunicorn socket.
nginx.service: the web server.
postgresql.service: the PostgreSQL database.
sshd.service: the SSH daemon.
systemd-timesyncd.service: the systemd time synchronization daemon which prevents clock drift.

Notes:

Computer hardware clocks are not very precise, and can drift away from the actual time as the computer runs. As such, we can use the `time sync daemon to synchronize the clock with more accurate time servers using the _network time protocol.
As we will see in later sections, we can also run services like gunicorn, nginx, and postgresql in containers, in which case enabling them in systemd will not be required.
gunicorn does not provide service and socket files out of the box. We will cover their creation in a later chapter.

User units

While systemd is often used to control system processes, it can also manage user services and other units. systemd generally starts a per-user process for each user when they log in. User units cannot depend on system units.

User units can also be controlled with the systemctl command, but also require a --user flag.

Writing a service unit file

To define a new service, you have to write a service file in the appropriate location. System units are typically defined in /etc/systemd/system/, and user units are typically defined in /etc/systemd/user/ for units shared among all users, or ~/.config/systemd/user/ for units specific to each user.

The service unit file is described in the systemd.unit and systemd.service man pages, and I invite you to read those for more details, but we will cover some basics here.

The unit file is split into multiple sections, with a header in square brackets, such as [Unit], [Service], or [Install]. The lines below set options for that specific section.

The [Unit] section defines generic information about the unit, such as its description, documentation links, dependencies, conflicts, etc.

The optional [Install] section may provide additional information about how to install the unit. It is used by the enable and disable commands, and can be used to define aliases, units that depend on this one, or a default instance.

The [Service] section defines the service itself.

Let's create a completely useless service that simply displays the current date and time. While it has no practical applications, it will at least be useful as an example. We will see more realistic uses in later chapters.

Let's create a new file called ~/.config/systemd/user/now.service with the following content:

[Unit]
Description=Log the current date and time

[Service]
Type=oneshot
ExecStart=/usr/bin/date

[Install]
WantedBy=timers.target

Note: we added the WantedBy option to specify the timers.target unit. This will allow us to schedule the service to run at specific times in the task scheduling chapter.

After saving the file, we can now run our service with systemctl start now.service --user. As it is a user, not a system unit, we need to specify the --user flag. As we run it, we see... nothing! That is by design, as systemd captures the standard output.

We can see the output with the systemctl status now.service --user command.

$ systemctl status now.service --user
● now.service - Log the current date and time
     Loaded: loaded (...)
     Active: inactive (dead)

Feb 12 14:44:07 machine-name date[pid]: Mon 12 Feb 2024 02:44:07 PM

There will likely be a bit more information, but notice that the service is not running, as it has exited normally and is of type oneshot, but more importantly, it has logged the current date and time.

systemd Journal

The systemd journal is a log of all events in the system. It is useful for troubleshooting and debugging. You can read it with the journalctl command.

The journalctl command is well documented in the man pages and the Arch Linux wiki, so we will not be covering it in details here, but some of the more useful options include:

-b, the boot flag. -b 0 filters the journal to only show the messages since the last system start, -b -1 shows the messages from the penultimate's system start until the last restart, etc.
--user, the user flag. Like the systemctl command, it can be used to specify user units rather than system units.
--system, the system flag. As it's the default, it is probably not that useful.
-u, the unit flag. This will filter the journal to only show messages from the specified unit. Alternatively, you can use the --user-unit flag to only show a specific user's unit.
-S and -U, the since and until flags which allow you to filter the journal to only show messages after and/or before a specific time.

For example, we can see the log for the service we wrote via journalctl -u now.service --user or journalctl --user-unit now.service.