Skip to content

CPU Isolation

Use the CPU isolation plug-in to allocate selected CPUs exclusively to one application. This avoids interference from other workloads and is important for real-time applications.

NOTICE

Installing the plug-in from resource-enablement-cpu.tar.gz alone does not guarantee real-time behavior. Prepare the device for real-time applications.

Configuring CPU isolation

You can control how CPUs are allocated. Define the device-builder default configuration in /usr/lib/ie-cpu-resource-plugin/ie-cpu-resource-plugin.yaml. During operation, changes are written only to /etc/ie-resource-plugins/ie-cpu-resource-plugin.yaml.

The CPU isolation plug-in reads the static parameters from the device-builder configuration and the non-static parameters from the user configuration. Users can change only the non-static parameters. The plug-in merges both sets into the effective configuration. After a firmware update, the system keeps the user configuration and merges it with the updated device-builder configuration.

NOTICE

Make sure that /etc/ie-resource-plugins/ is empty before shipping the image.

The following settings are supported:

# CPUs that should be reserved for system tasks and are not isolatable
cpuset_system: "2"

# Statically configured system CPUs
cpuset_system_static: "0-1"

# Statically configured CPUs that should be ignored by Industrial Edge
cpuset_ignore_static: "0,4"

Each setting uses a CPU set as a comma-separated list of ranges. A range can also contain a single CPU. Examples are 0, 0,2,4, and 0-1,3,8-9. Do not use spaces. Do not use descending ranges. For example, 0, 4-3 is invalid for both reasons.

NOTICE

The device administrator can change cpuset_system dynamically. The static options cpuset_system_static and cpuset_ignore_static can only be changed in the configuration file. This lets you define the base configuration while still allowing administrators to adjust the running system.

In the following sections, cpuset_system_effective means the union of cpuset_system and cpuset_system_static.

  • cpuset_system_effective defines the set of CPUs that are reserved for housekeeping and are not given to Industrial Edge apps as isolated CPUs.
    They are, however, allowed to run Industrial Edge apps that aren't given dedicated isolated CPUs.
  • cpuset_ignore_static defines the set of CPUs that must not be touched by Industrial Edge altogether.
    If this option is not set (or set to the empty string), it defaults to the empty set. The set of CPUs usable by Industrial Edge is the difference between the set of online CPUs and the CPUs to ignore (online CPU set cpuset_ignore); it therefore includes the CPUs in cpuset_system_effective. All Industrial Edge CPUs, except for the CPUs in cpuset_system_effective, are isolatable.

NOTICE

cpuset_system_effective and cpuset_ignore_static may overlap. CPUs that are in both sets are used to run system processes, but they are not used to run Industrial Edge apps.

The following constraints exist on the aforementioned configuration options:

  • cpuset_system_static must contain CPU 0 and therefore is never empty (CPU 0 is often special as it may run important system processes and kernel threads, or handle interrupts or hardware-related tasks)
  • cpuset_system_effective must neither equal cpuset_ignore_static nor be a subset thereof (if all Industrial Edge CPUs were to be isolated, regular apps that do not require isolation would no longer be able to run)

For most use cases, use the following default values. They give administrators a high level of flexibility. The values do not account for hyperthreading, which is discussed later.

cpuset_system: "", cpuset_system_static: "0", cpuset_ignore_static: ""

If the target device must run special tasks on dedicated CPUs, exclude these CPUs from Industrial Edge by using cpuset_ignore_static. For example, if the device has many CPUs and each CPU can run a real-time application, reserve at least one CPU for system tasks such as interrupt handling:

cpuset_system: "", cpuset_system_static: "0,1", cpuset_ignore_static: "0"

In this example, cpuset_system_static also includes CPU 1.

NOTICE

Restart the system after you change the configuration. The ie-cpu-partition service uses these values during startup to set the allowed CPU sets of the default systemd cgroups and to update the TuneD profile.

Hyperthreaded systems

The CPU sets in cpuset_system, cpuset_system_static, and cpuset_ignore_static refer to logical CPUs. These are the processing units that the operating system sees and schedules. Logical CPUs do not always match physical cores. Depending on the hardware and configuration, they can also represent hyperthreads.

The set of online logical CPUs is listed under /sys/devices/system/cpu/online. The values found under /sys/devices/system/cpu/cpu<number>/topology/thread_siblings_list, where cpu<number> is any of the online logical CPUs, provide information on the association of hyperthreads (also called thread siblings) with the system's physical cores.

For example, /sys/devices/system/cpu/cpu0/topology/thread_siblings_list shows the thread siblings of logical CPU 0, including itself. If the file contains one CPU only, the logical CPU maps directly to one physical core. If it contains multiple logical CPUs, those CPUs share one physical core. For example, if /sys/devices/system/cpu/cpu0/topology/thread_siblings_list contains 0-1, logical CPU 0 and logical CPU 1 share the same physical core.

NOTICE

It need not be the case that consecutive logical CPUs are each other's thread siblings (e.g., logical CPU 1 and logical CPU 5 could be thread siblings).

The CPU resource plug-in isolates physical cores only. It does not split thread siblings across isolation boundaries. This means that if cpuset_isolatable contains only some, but not all, thread siblings of one physical core, the plug-in treats all of these logical CPUs as non-isolatable.

For example, on a quad-core hyperthreaded system (with /sys/devices/system/cpu/online listing CPU set 0-7), if each two consecutive logical CPUs are thread siblings (i.e., logical CPU 0 and logical CPU 1 are thread siblings, logical CPU 2 and logical CPU 3 are thread siblings, etc.), and if the CPU resource plugin is configured with cpuset_system: "2,4", cpuset_system_static: "0", and cpuset_ignore_static: "6", then, in effect, no logical CPU is considered isolatable, because 0-7 \ 6 \ 0,2,4 = 1,3,5,7, and no combination of logical CPUs from 1,3,5,7 resolves to all of the thread siblings of a physical core.

When the plug-in isolates a CPU on a hyperthreaded system, it intentionally assigns only one thread sibling of the physical core. This prevents a real-time application from sharing the same core with another process. The other thread siblings remain unused. The plug-in does not assign them to the application and also removes them from the best-effort set.