isolcpus kernel option appears to break taskset

Question

I have a laptop with an Intel Core i7-12700H CPU, running Ubuntu 24.04 LTS. This CPU has 6 “performance“ cores, each one running 2 threads, and 8 “efficient” cores.

Most of the time, the “efficient” cores can provide much more power than I need, so I’d rather keep the “performance” cores idle, to save power, and only use the “efficient” ones. Occasionally, I have to run and benchmark some intensive (multi-threaded) computation code. Then I wan to use the “performance” cores, without being disturbed by other processes. I may want either to run a single thread per core, to get the maximum per-thread performance, or to use both threads of each core.

At first, I considered using cgroup’s cpusets, with the cset tool. I wrote the following script to create the sets:

cset set -c 12-19 --cpu_exclusive -s efficient 
cset set -c 0-11 --cpu_exclusive -s performance 
cset set -c 0,2,4,6,8,10 -s performance/no-ht 
cset proc -m -f root -t efficient

It works as expected, but I find it troublesome since I need root to escape the efficient cgroup, with ugly commands like

sudo cset proc --exec performance/no-ht -- sudo -u $(whoami) ~/bin/my_code

Moreover, because it switches to root, I can’t wrap that whole command in time or a profiler. I can still run time or the profiler within the performance cgroup, but then they share the same cores as the code under test…

Hence I considered taskset, with the following command:

/usr/bin/time taskset -c 0-11:2 ~/bin/my_code

It works as expected, does not need root, but other processes may use the “performance“ cores.

One solution is to use both systems, with an awful command like:

sudo cset proc --exec root -- sudo -u $(whoami) taskset -c 12-19 /usr/bin/time taskset -c 0-11:2 ~/bin/my_code

It works fine but looks over-complicated and still needs root…

Then I read about the isolcpus kernel parameter and, despite being deprecated, it looked great to me. So I added isolcpus=0-11 to the GRUB_CMDLINE_LINUX_DEFAULT parameter in /etc/default/grub, ran sudo update-grub and rebooted.

At first sight it looked fine, only a few kernel threads were running on CPUs 0 to 11, all userspace processes were running on CPUs 12 to 19. So I tried:

/usr/bin/time taskset -c 0-11:2 ~/bin/my_code

As expected, OpenMP correctly discovered it could use 6 cores and launched 6 threads, but I figured out all of them were running on CPU 0… I checked the Cpus_allowed_list line in /proc/<pid>/status and it was correct (0,2,4,6,8,10). Running taskset to allow each thread on a single core works, but it’s not very convenient and does not allow to share load if there are more threads than cores…

Any idea why the threads seem to be stuck on the first allowed CPU when using the isolcpus kernel option?

Is there a better way to keep everything running on CPUs 12 to 19 by default, like if everything was run through taskset -c 12-19, without using cgroup jails than need root to escape?

Answer 1

I finally found another way to proceed, so I’m documenting it here. That solution is to set CPUAffinity=12-19 in the [Manager] section in /etc/systemd/system.conf.

Then systemd (pid 1) is running as if running though taskset -c 12-19 and all userspace programs inherit from those defaults.

The negative point is that it does not affect the kernel’s own threads, such as kswapd0 which sometimes uses lots of CPU. Of course, there are some kernel threads that need to be tied to a specific core, such as cpuhp/0, ksoftirqd/0, migration/0 or idle_inject/0, but many other ones can run on any core and might be tied to cores 12 to 19.

I can use taskset to change the CPU affinity of kernel threads like kswapd0. Since All the kernel threads are children of kthreadd, I guess that, if I could change its CPU affinity early in the boot process (before it starts other kernel treads), then all kernel threads that don’t need to be tied to a specific CPU set would inherit from kthreadd’s CPU affinity.

Hence, I don’t intend to accept this answer, and I’m still interested by an answer that also work for kernel threads, and an explanation why the isolcpus kernel parameter does not behave as I expected.