How is syslog entangled with journald?

Question

I set up my journald to limit the log size to 5GB in /etc/systemd/journald.conf and thought everything was fine. Then, yesterday, I realized that my /var/log/syslog file is in fact 12GB big.

What I believed was that journalctl is reading the messages from different log locations, and that the SystemMaxUse option is meant to limit their size. But as it seems, it is not.

I ended up using logrotate now with the size option but I would prefer to understand the concept for the future.

It looks like I have not really understood how the logs in /var/log are entangled with journald/journalctl. Please enlighten me!

Answer 1

The way it works is that systemd-journald can forward logs to syslog (aka to the file /var/log/syslog). These forwarded logs are not controlled by systemd-journald so that is why the SystemMaxUse= directive does not apply to /var/log/syslog. The SystemMaxUse= only applies to persistent systemd-journald controlled files which are usually in /var/log/journal.

The files in /var/log/journal are meant to be viewed via journalctl. Journald provides additional features not found with syslog, including structured logging, indexing logs for fast search, access control and signed messages.

Conventional syslog files like /var/log/syslog can be opened and read with any text viewing program. Syslog files have a different format than journald logs. Depending on your settings, the forwarded syslogs may contain all the logs that journald has itself.

Details on forwarding logs to syslog

Some of this may be specific to Ubuntu Server 24.04

Forwarding logs from systemd-journald to syslog is controlled by the ForwardToSyslog= directive. In Ubuntu, this is set to yes in a systemd drop-in file that is created by default for you: /usr/lib/systemd/journald.conf.d/syslog.conf:

[Journal]
ForwardToSyslog=yes

The logs are actually forwarded to rsyslog (systemctl status rsyslog.service) which is mostly configured in /etc/rsyslog.d/50-default.conf. There you can see /var/log/syslog and other conventional log file locations used to output the forwarded logs:

#
# First some standard log files.  Log by facility.
#
auth,authpriv.*                 /var/log/auth.log
*.*;auth,authpriv.none          -/var/log/syslog
#cron.*                         /var/log/cron.log
#daemon.*                       -/var/log/daemon.log
kern.*                          -/var/log/kern.log
#lpr.*                          -/var/log/lpr.log
mail.*                          -/var/log/mail.log
#user.*                         -/var/log/user.log

#
# Logging for the mail system.  Split it up so that
# it is easy to write scripts to parse these files.
#
#mail.info                      -/var/log/mail.info
#mail.warn                      -/var/log/mail.warn
mail.err                        /var/log/mail.err

The logs from systemd-journald are forwarded over a Unix socket that Rsyslog reads from. The socket is located at /run/systemd/journal/syslog and configured by the systemd socket unit syslog.socket (systemctl status syslog.socket):

# /usr/lib/systemd/system/syslog.socket
[Socket]
ListenDatagram=/run/systemd/journal/syslog
# ...

On Ubuntu, Rsyslog uses a module called imuxsock that utilizes the Unix socket at /run/systemd/journal/syslog where journald can write logs to if the ForwardToSyslog= configuration option is enabled. The imuxsock module is configured in /etc/rsyslog.conf:

module(load="imuxsock") # provides support for local system logging
# ...

Rsyslog should by default be configured for systemd support on all platforms that usually run systemd (which means most Linux distributions, but not, for example, Solaris).

Rsyslog is able to coexist with systemd with minimal changes on the part of the local system administrator. While the systemd journal now assumes full control of the local /dev/log system log socket, systemd provides access to logging data via the /run/systemd/journal/syslog log socket. This log socket is provided by the syslog.socket file that is shipped with systemd.

The imuxsock module can still be used in this setup and provides superior performance over imjournal, the alternative journal input module.

Coexistence with systemd

Answer 2

systemd-journald also writes in /run/log/journal.

--flush

Asks the journal daemon to flush any log data stored in /run/log/journal/ into /var/log/journal/, if persistent storage is enabled. This call does not return until the operation is complete.

Note that this call is idempotent: the data is only flushed from /run/log/journal/ into /var/log/journal/ once during system runtime (but see --relinquish-var below), and this command exits cleanly without executing any operation if this has already happened. This command effectively guarantees that all data is flushed to /var/log/journal/ at the time it returns.

--relinquish-var

Asks the journal daemon for the reverse operation to --flush: if requested the daemon will write further log data to /run/log/journal/ and stops writing to /var/log/journal/. A subsequent call to --flush causes the log output to switch back to /var/log/journal/, see above.

Answer 3

Journald limit (SystemMaxUse) puts a limit only on journald own files in /var/log/journal. It doesn't know or care about other log files. It's other applications responsibility.