I finally got this working! The trick was to create my own slice and set it in the service file, like this:

[Service]
# Everything else as in the original question
Slice=my_service_limit.slice

And create a slice unit file /lib/systemd/system/my_service_limit.slice that looks like this:

[Unit]
Description=Slice that limits memory for all my services

[Slice]
# MemoryHigh works only in "unified" cgroups mode, NOT in "hybrid" mode
MemoryHigh=500M
# MemoryMax works in "hybrid" cgroups mode, too
MemoryMax=600M

Note: be careful with the naming of the slice, as - is a hierarchy separator, as explained in https://systemd.io/CGROUP_DELEGATION - a very helpful page for anyone trying to configure this. You can check whether the service is really using the configured slice by looking at the output of systemctl status myservice - it should say:

  CGroup: /my_service_limit.slice/[email protected]

It was not necessary to set systemd.unified_cgroup_hierarchy=1 (as per Ryutaroh Matsumoto's answer) for MemoryMax to work, but it was necessary for MemoryHigh - even in "hybrid" mode (the default in Ubuntu 18.04) it's silently ignored.

Also worth noting that these only apply to the physical RAM used - they do not include swap space used. (There is a separate MemorySwapMax setting, but no MemorySwapHigh, it seems.)

I finally got this working! The trick was to create my own slice and set it in the service file, like this:

[Service]
# Everything else as in the original question
Slice=my_service_limit.slice

And create a slice unit file /lib/systemd/system/my_service_limit.slice that looks like this:

[Unit]
Description=Slice that limits memory for all my services

[Slice]
# MemoryHigh works only in "unified" cgroups mode, NOT in "hybrid" mode
MemoryHigh=500M
# MemoryMax works in "hybrid" cgroups mode, too
MemoryMax=600M

Note: be careful with the naming of the slice, as - is a hierarchy separator, as explained in https://systemd.io/CGROUP_DELEGATION - a very helpful page for anyone trying to configure this. You can check whether the service is really using the configured slice by looking at the output of systemctl status myservice - it should say:

  CGroup: /my_service_limit.slice/[email protected]

It was not necessary to set systemd.unified_cgroup_hierarchy=1 (as per Ryutaroh Matsumoto's answer) for MemoryMax to work, but it was necessary for MemoryHigh - even in "hybrid" mode (the default in Ubuntu 18.04) it's silently ignored.

Also worth noting that these only apply to the physical RAM used - they do not include swap space used. (There is a separate MemorySwapMax setting, but no MemorySwapHigh, it seems.)

I finally got this working! The trick was to create my own slice and set it in the service file, like this:

[Service]
# Everything else as in the original question
Slice=my_service_limit.slice

And create a slice unit file /lib/systemd/system/my_service_limit.slice that looks like this:

[Unit]
Description=Slice that limits memory for all my services

[Slice]
MemoryHigh=500M
MemoryMax=600M

Note: be careful with the naming of the slice, as - is a hierarchy separator, as explained in https://systemd.io/CGROUP_DELEGATION - a very helpful page for anyone trying to configure this. You can check whether the service is really using the configured slice by looking at the output of systemctl status myservice - it should say:

  CGroup: /my_service_limit.slice/[email protected]

It was not necessary to set systemd.unified_cgroup_hierarchy=1 on Ubuntu 18.04, which runs cgroups in "hybrid" mode by default.

I finally got this working! The trick was to create my own slice and set it in the service file, like this:

[Service]
# Everything else as in the original question
Slice=my_service_limit.slice

And create a slice unit file /lib/systemd/system/my_service_limit.slice that looks like this:

[Unit]
Description=Slice that limits memory for all my services

[Slice]
# MemoryHigh works only in "unified" cgroups mode, NOT in "hybrid" mode
MemoryHigh=500M
# MemoryMax works in "hybrid" cgroups mode, too
MemoryMax=600M

Note: be careful with the naming of the slice, as - is a hierarchy separator, as explained in https://systemd.io/CGROUP_DELEGATION - a very helpful page for anyone trying to configure this. You can check whether the service is really using the configured slice by looking at the output of systemctl status myservice - it should say:

  CGroup: /my_service_limit.slice/[email protected]

It was not necessary to set systemd.unified_cgroup_hierarchy=1 (as per Ryutaroh Matsumoto's answer) for MemoryMax to work, but it was necessary for MemoryHigh - even in "hybrid" mode (the default in Ubuntu 18.04) it's silently ignored.

Also worth noting that these only apply to the physical RAM used - they do not include swap space used. (There is a separate MemorySwapMax setting, but no MemorySwapHigh, it seems.)