6

I was thinking if there is "canonical" way to have this?

Background & description

I have to install some program on live server. Although I do trust the vendor (FOSS, Github, multiple authors...) I would rather to ensure avoiding not entirely impossible scenario of script falling in some trouble and exhausting system resources and leaving server unresponsive. I had the case of installing amavis which was started right after and because of some messy configuration it have produced loadavg of >4 and system barely responsive.

My first taught was nice - nice -n 19 thatscript.sh. This may and may not help, but I was thinking that it would be best that I write and activate script that would do following:

  • run as daemon, on (example) 500ms-2s

  • check for labeled process with ps and grep

  • if labeled process(es) (or any other process) takes too much CPU (yet to be defined) - kill them with SIGKILL

My second taught was - it would not be the first time that I'm reinventing the wheel.

So, is there any good way to "jail" the program and processes produced by it into some predefined limited amount of system resources or automated kill if some threshold was reached by them?

Improve this question
8
  • Thanks. Any hint about this nice ways? What would it be Commented Jun 19, 2017 at 15:40
  • This is done already by the system at a per-user level with /etc/security/limits.conf; the current settings can be observed with ulimit. Give your process a role user, and apply appropriate resource limits. Commented Jun 19, 2017 at 15:52
  • I reread your question. The only limit you want is on the total amount of CPU time that a process or set of processes accumulate? Or are there other criteria, such as percentage of CPU used over a certain time period, memory usage, disk space usage, disk bandwidth, network usage, etc. Commented Jun 19, 2017 at 17:11
  • @DopeGhoti sounds nice but installing a program is usually done with root Commented Jun 19, 2017 at 21:45
  • @MarkPlotnick well, I think total amount of CPU time constraining would do the job. Unresponsive machine usually runs out of this Commented Jun 20, 2017 at 5:14

2 Answers 2

Reset to default
5

Alternative #1: Monitor your process with monit

Install M/Monit and create a configuration file based on this template:

check process myprogram
matching "myprogram.*"
start program = "/usr/bin/myprogram" with timeout 10 seconds
stop program = "/usr/bin/pkill thatscript"
if cpu > 99% for 2 cycles then stop
if loadavg (5min) > 80 for 10 cycles then stop

Alternative #2: Limit process CPU usage with cgroups

The most native Linux specific solution of them all. Offers a lot of options and complexity.

Example:

sudo cgcreate -g cpu:/cpulimited
sudo cgset -r cpu.shares=512 cpulimited
sudo cgexec -g cpu:cpulimited /usr/bin/myprogram > /dev/null &

I encourage you to read more at:

DigitalOcean - How-to: Limit resources using cgroups on CentOS 6

RedHat - Resource Management Guide

Oracle - List of cgroups subsystems

Alternative #3: Limit process CPU usage with cpulimit

Get latest version of cpulimit from your package manager of choice, or by getting the source available at GitHub.

Limit the CPU usage to 90%: cpulimit -l 90 /usr/bin/myprogram > /dev/null &

Sidenote:

You can also pin a certain process to use certain CPU core(s) to ensure that you always have some free CPU power.

Improve this answer
5
+50

systemd can limit a target/service's resources. From the man page:

CPUQuota:

Assign the specified CPU time quota to the processes executed. Takes a percentage value, suffixed with "%". The percentage specifies how much CPU time the unit shall get at maximum, relative to the total CPU time available on one CPU. Use values > 100% for allotting CPU time on more than one CPU. This controls the "cpu.max" attribute on the unified control group hierarchy and "cpu.cfs_quota_us" on legacy. For details about these control group attributes, see cgroup-v2.txt and sched-design-CFS.txt.

Example: CPUQuota=20% ensures that the executed processes will never get more than 20% CPU time on one CPU.

Implies "CPUAccounting=true".

Since using this implies CPUAccounting I'll include that as well

CPUAccounting:

Turn on CPU usage accounting for this unit. Takes a boolean argument. Note that turning on CPU accounting for one unit will also implicitly turn it on for all units contained in the same slice and for all its parent slices and the units contained therein. The system default for this setting may be controlled with DefaultCPUAccounting= in systemd-system.conf(5).

I'll also quote from Slice:

The name of the slice unit to place the unit in. Defaults to system.slice for all non-instantiated units of all unit types (except for slice units themselves see below). Instance units are by default placed in a subslice of system.slice that is named after the template name.

So by default everything will get thrown into the same slice, which means everything in a single resource pool.

There's also MemoryHigh to look at:

MemoryHigh:

Specify the high limit on memory usage of the executed processes in this unit. Memory usage may go above the limit if unavoidable, but the processes are heavily slowed down and memory is taken away aggressively in such cases. This is the main mechanism to control memory usage of a unit.

Takes a memory size in bytes. If the value is suffixed with K, M, G or T, the specified memory size is parsed as Kilobytes, Megabytes, Gigabytes, or Terabytes (with the base 1024), respectively. Alternatively, a percentage value may be specified, which is taken relative to the installed physical memory on the system. If assigned the special value "infinity", no memory limit is applied. This controls the "memory.high" control group attribute. For details about this control group attribute, see cgroup-v2.txt.

Implies "MemoryAccounting=true".

This setting is supported only if the unified control group hierarchy is used and disables MemoryLimit=.

You can easily toss a script into a systemd service.

Assuming /usr/local/thatscript.sh is the script:

/usr/lib/systemd/system/thatscript.service

[Unit]
Description=This runs "thatscript"
ConditionFileNotEmpty=/usr/local/thatscript.sh

[Service]
Type=simple
ExecStart=/usr/local/thatscript.sh
CPUQuota=20%

[Install]
WantedBy=multi-user.target

Then you'll need systemctl daemon-reload to read in the new service file, then you can systemctl enable thatscript.service if you want it to run at boot, or systemctl start thatscript.service if you want to start it manually.

Improve this answer
1
  • Yes, systemd is available - I'm using system that supports it. Thanks for answer, I'll review it carefully Commented Jun 22, 2017 at 12:29

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.