Skip to content

Real-time Applications

Basic prerequisites

Linux Kernel

To run applications with real-time, low-latency requirements on Industrial Edge, use a real-time kernel. Most Linux distributions provide suitable packages. On Debian, for example, run apt-get install linux-image-rt-amd64. To verify the setup, check whether uname -a contains PREEMPT_RT or PREEMPT RT, or whether /sys/kernel/realtime exists and contains 1.

Cgroups v2

The CPU isolation plugin relies on kernel control groups (cgroups) to control the CPU sets available to apps. There currently exist two interfaces, v1 and v2, and in general, a system can be operated in one of three ways:

  • cgroup v1 only
  • cgroup v2 only
  • hybrid, with both v1 and v2 enabled

The cgroup v1 interface is deprecated. Major projects such as systemd have already announced that they will drop support for it. For this reason, CPU isolation supports cgroup v2 only. To verify the configuration, run mount | grep cgroup. You should see one entry of type cgroup2 that is mounted at /sys/fs/cgroup, for example:

$ mount | grep cgroup
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)

Legacy setups that mount cgroup v2 at /sys/fs/cgroup/unified are not supported.

Power Management

In addition to cgroup v2 support, Power Management Quality of Service (PM QoS) should be available on the host. The corresponding device /dev/cpu_dma_latency should exist so that the system can mount it into containers. This gives applications an interface to control power management. If this is not possible, configure power management system-wide so that timing variations stay low, for example by reducing sleep-state effects.

System tuning

To achieve stable real-time behavior, apply additional measures at system level. We refer to these measures as system tuning. The best tuning settings depend on the platform and the applications. There is no single configuration that fits every system. Industrial Edge runs on different platforms and is designed for many application types. For this reason, the following recommendations are a compromise:

  • They aim to provide acceptable latencies for most apps and were developed using measurements for selected benchmark applications.
  • They focus on generic (non device-dependent) measures that are not overly intrusive, i.e., don't require global changes/settings that can impact other applications in the system.
    Hence, we change as few system defaults as possible.

Depending on your platform, you can apply additional and more aggressive tuning measures to reduce latency further.

TuneD

We recommend TuneD for system tuning. TuneD offers several technical and operational advantages:

  • TuneD is an established Open Source project, and using it means you don't have to implement and rely on own custom-built tuning scripts.
  • Tuning profiles offer a flexible way of describing tuning measures.
  • TuneD monitors the system for thread-creation events.
    This way, it can adapt the affinity and/or priority of newly created kernel threads (cf. the last item on the list above).
  • TuneD can perform other tuning tasks beyond CPU isolation.
    If your platform and/or application requires additional measures, you can simply extend the tuning profile.
  • TuneD enables applications to dynamically tune the per-cpu kernel threads of isolated cores and the network I/O path of isolated networks.

The installation steps depend on your Linux distribution. The minimum supported version is v2.20.0. On Debian Bookworm and later, you can install TuneD with apt-get install tuned. On Debian Bullseye, the included TuneD version v2.15.0 is not supported. Install v2.20.0 or later from bullseye-backports.

To let applications tune per-CPU kernel threads and the network I/O path dynamically, the CPU and network isolation plug-ins require a newer TuneD version that includes the kthread plug-in. At the time of writing, no official TuneD release includes the kthread plug-in. Build a custom Debian package based on v2.24.0 and use the kthread plug-in pull request.

Linux Perf

To monitor the system and get notified whenever a new process or thread is created, TuneD uses Linux perf. The functionality offered by perf is implemented in part in the kernel itself, and in part by supporting userland libraries and tools. It's those libraries and tools that we're talking about here.

On Debian 12 (Bookworm), install the package linux-perf. On Debian 11 (Bullseye), use the package from bullseye-backports because the default Debian 11 package does not work. In both cases, you can skip the following subsections and continue with enabling the tuning profile.

Python Bindings

TuneD is implemented in Python. For this reason, it also requires the Python bindings of the Linux perf userland libraries and tools. Whether these bindings are included depends on the distribution. For example, RHEL, CentOS, and Fedora include the bindings, while Ubuntu does not. The Debian change that adds these bindings is available here.

To test if you have the Python bindings (after installing perf in the way preferred by your distribution), run:

python3 -c "import perf"

If the response is for example

$ python3 -c "import perf"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'perf'

TuneD will not work as expected in this case.

If your distribution package does not include the bindings, build your own perf package with the required patch.

Version Compatibility and Versioned Packages

The source code of the userland perf libraries and tools is part of the Linux kernel source tree. Its build and packaging are tightly coupled to the kernel build. For this reason, the required steps depend on how you build and integrate the kernel on your platform.

For kernels up to versions of around 5.16 or 5.18, the version (major.minor) of the userland libraries and tools must match that of the kernel. For more recent kernel versions, that strict requirement has been removed, so a newer version of the userland libraries and tools (the package linux-perf) will also work on older kernels. In case of Debian that means that the situation has improved as Debian 11 (Bullseye, with bullseye-backported kernel 6.1) and Debian 12 (Bookworm, with kernel 6.1) use the aforementioned more recent kernel versions.

Debian supports the installation of multiple kernel packages at the same time. For this reason, Debian also ships versioned Linux perf packages so that the binaries do not conflict. For example, packages such as linux-perf_4.19 and linux-perf_6.1 can be installed at the same time. The system selects the correct binaries at runtime based on the running kernel. If you use a kernel earlier than 5.16 or 5.18 and you need to build your own linux-perf package on Debian, build a versioned package. As noted in the prerequisites, the minimum supported kernel version for the Resource Manager is 5.12. If you do not use the Debian kernel sources, apply the following two Debian patches:

  • Find the patches Debian applies for your kernel version, e.g., for kernel 6.1.
  • You need the patches tools-perf-install.patch and tools-perf-version.patch.

If you use a versioned linux-perf package, also install a wrapper that allows import perf from Python and selects the correct version for the running kernel. You can find the file perf.py in this merge request. Install it in the Python path, for example in /usr/lib/python3/dist-packages/.

Tuning Profile

The ie-cpu-resource-plugin package includes the iedge TuneD profile in /lib/tuned/iedge/tuned.conf. Configure the profile in /etc/tuned/iedge-variables.conf, where you define the relevant CPU sets. The ie-cpu-partition service keeps this configuration file up to date and synchronized with the ie-cpu-resource-plugin configuration.

After installation, activate the iedge tuning profile:

tuned-adm profile iedge