133

Before all the unit files were in /etc/systemd/system/ but now some are showing up in /usr/lib/systemd/system (<- on CentOS, or /lib/systemd/system <- on Debian/Ubuntu), what is the difference between these folders?

Improve this question
0

3 Answers 3

Reset to default
88

This question is already answered in man 7 file-hierarchy which comes with systemd (there is also online version):

        /etc
           System-specific configuration.
 (…)
 VENDOR-SUPPLIED OPERATING SYSTEM RESOURCES
       /usr
            Vendor-supplied operating system resources. 
            Usually read-only, but this is not required. Possibly 
            shared between multiple hosts. This directory should not
            be modified by the administrator, except when installing 
            or removing vendor-supplied packages.

Basically, files that ships in packages downloaded from distribution repository go into /usr/lib/systemd/. Modifications done by system administrator (user) go into /etc/systemd/system/.

System-specific units override units supplied by vendors. Using drop-ins, you can override only specific parts of unit files, leaving the rest to vendor (drop-ins are available since the very beginning of systemd, but were properly documented only in v219; see man systemd.unit).

Improve this answer
5
  • It seems at least sometimes one overwrite the other one. For e.g., I tweak /etc/systemd/system/multi-user.target.wants/apache2.service, then sudo systemctl daemon-reload and changes are copied to /lib/systemd/system/apache2.service (a Debian 9). Commented Nov 10, 2022 at 22:45
  • @PabloA Services in /etc/systemd/system/multi-user.target.wants/ are symbolic links to the real services. Commented Feb 16, 2023 at 14:18
  • 2
    Ok, that explains /etc/ vs /usr/lib/ but on my Ubuntu 23.04 I have: /lib/systemd/ and /usr/lib/systemd and /etc/systemd Commented Apr 25, 2023 at 17:09
  • 1
    @Bram on Fedora, /lib is symlink to /usr/lib. This is remnant from ancient times where /bin, /lib and /sbin were supposed to be minimal, self-contained part of OS able to bring up the entire thing, including mounting network drives. See freedesktop.org/wiki/Software/systemd/separate-usr-is-broken Commented Apr 28, 2023 at 9:17
  • This does not answer the OPs question AFAICT. Commented Mar 5 at 0:32
75

Background

If you look at the man page man systemd.unit it has a table that explains the differences. This is from a CentOS 7.x system.

   UNIT LOAD PATH
          Unit files are loaded from a set of paths determined during 
          compilation, described in the two tables below. Unit files found 
          in directories listed earlier override files with the same name 
          in directories lower in the list.

           Table 1.  Load path when running in system mode (--system).
           ┌────────────────────────┬─────────────────────────────┐
           │Path                    │ Description                 │
           ├────────────────────────┼─────────────────────────────┤
           │/etc/systemd/system     │ Local configuration         │
           ├────────────────────────┼─────────────────────────────┤
           │/run/systemd/system     │ Runtime units               │
           ├────────────────────────┼─────────────────────────────┤
           │/usr/lib/systemd/system │ Units of installed packages │
           └────────────────────────┴─────────────────────────────┘

When they say "installed packages" they're referring to anything which was installed via an RPM. The same can be assumed for Debian/Ubuntu as well where a DEB file would be the "installed package".

NOTE: the table above from a Debian/Ubuntu system is slightly different.

 Table 1.  Load path when running in system mode (--system).
       ┌────────────────────┬─────────────────────────────┐
       │Path                │ Description                 │
       ├────────────────────┼─────────────────────────────┤
       │/etc/systemd/system │ Local configuration         │
       ├────────────────────┼─────────────────────────────┤
       │/run/systemd/system │ Runtime units               │
       ├────────────────────┼─────────────────────────────┤
       │/lib/systemd/system │ Units of installed packages │
       └────────────────────┴─────────────────────────────┘

Analyzing /usr/lib/systemd/system

You can tell what packages own which unit files in /usr/lib/systemd/system like this on a CentOS/Fedora/RHEL system:

$ rpm -qf /usr/lib/systemd/system/* |sort -u | head
abrt-2.1.11-50.el7.centos.x86_64
abrt-addon-ccpp-2.1.11-50.el7.centos.x86_64
abrt-addon-kerneloops-2.1.11-50.el7.centos.x86_64
abrt-addon-pstoreoops-2.1.11-50.el7.centos.x86_64
abrt-addon-vmcore-2.1.11-50.el7.centos.x86_64
abrt-addon-xorg-2.1.11-50.el7.centos.x86_64
accountsservice-0.6.45-7.el7.x86_64
acpid-2.0.19-8.el7.x86_64
alsa-utils-1.1.3-2.el7.x86_64
anaconda-core-21.48.22.134-1.el7.centos.x86_64

Analyzing /etc/systemd/system

If we do the same against /etc/systemd/system, we'd expect to find no files owned by an RPM (Which is in fact the case on my CentOS 7.x system. ):

$ rpm -qf /etc/systemd/system/* /etc/systemd/system/*/* | grep -v 'not owned'
$

Outliers

Keep in mind that you may find occasional stray files under /usr/lib/systemd/system, such as with Virtualbox (vboxadd*):

$ rpm -qf /usr/lib/systemd/system/* |sort -u | grep 'not owned'
file /usr/lib/systemd/system/initrd.target.wants is not owned by any package
file /usr/lib/systemd/system/shutdown.target.wants is not owned by any package
file /usr/lib/systemd/system/vboxadd.service is not owned by any package
file /usr/lib/systemd/system/vboxadd-service.service is not owned by any package
file /usr/lib/systemd/system/vboxadd-x11.service is not owned by any package

There are others.

Conclusions

The expectation is that /usr/lib/systemd/system is a directory that should only contain systemd unit files which were put there by the package manager (YUM/DNF/RPM/APT/etc).

Files in /etc/systemd/system are manually placed here by the operator of the system for ad-hoc software installations that are not in the form of a package. This would include tarball type software installations or home grown scripts.

Improve this answer
5
  • 7
    I was reluctant to click this google result because I was curious about /lib/systemd/system vs. /usr/lib/systemd/system. I'm glad I found this answer. Commented Aug 6, 2018 at 1:52
  • 2
    Placing a service definition in /etc/systemd/system generates an error if you mask it: Failed to execute operation: Invalid argument; systemd tries to replace the file with a symlink to /dev/null. Not saying this answer is incorrect, just something to remember. Commented Jan 22, 2019 at 17:59
  • @BrunoBronosky Debian actually uses both /lib/systemd/system and /usr/lib/systemd/system, therefore I asked the question separately unix.stackexchange.com/questions/550001/… Commented Nov 2, 2019 at 6:57
  • This should be made the real answer. Thanks Commented Jan 4, 2021 at 9:45
  • I don't feel this answers the OPs question - that is why I've downvoted. Commented Mar 5 at 0:34
1

The runtime files in /run/systemd/system stem from the ability to make modifications to a process (unit) during the current boot without that change/modification persisting across a reboot.

From systemctl.1:

--runtime
           When used with enable, disable, edit, (and related commands),
           make changes only temporarily, so that they are lost on the
           next reboot. This will have the effect that changes are not
           made in subdirectories of /etc/ but in /run/, with identical
           immediate effects, however, since the latter is lost on
           reboot, the changes are lost too.

           Similarly, when used with set-property, make changes only
           temporarily, so that they are lost on the next reboot.

Debian systems you can use dpkg-query -S to look for unit files with or without a package.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.