29

I just noticed this verify option for journalctl and decided to give it a go, it's showing corruption, what might cause that? and what if anything should I do about it? should I investigate further?

journalctl --verify
PASS: /var/log/journal/19184893a1d645c7a43729e79b10a876/user-1000.journal
Invalid object contents at 3733856░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░   0%
File corruption detected at /var/log/journal/19184893a1d645c7a43729e79b10a876/system.journal:3733856 (of 91734016, 4%).
FAIL: /var/log/journal/19184893a1d645c7a43729e79b10a876/system.journal (Bad message)
Invalid object contents at 21575496░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  45%
File corruption detected at /var/log/journal/19184893a1d645c7a43729e79b10a876/system@60e058db556e4de4b256d0b1ff176aa4-0000000000000001-0004e0b436d20aa1.journal:21575496 (of 44052480, 48%).
FAIL: /var/log/journal/19184893a1d645c7a43729e79b10a876/system@60e058db556e4de4b256d0b1ff176aa4-0000000000000001-0004e0b436d20aa1.journal (Bad message)
PASS: /var/log/journal/19184893a1d645c7a43729e79b10a876/user-1000@60e058db556e4de4b256d0b1ff176aa4-0000000000000a91-0004e0b4ff9a949a.journal
PASS: /var/log/journal/19184893a1d645c7a43729e79b10a876/user-1001.journal
Improve this question

3 Answers 3

Reset to default
22
+25

Currently, journalctl can detect corrupt logs but has no "fsck" type command to attempt repair. The journald will automatically switch to writing a new "clean" file as soon as it detects the problem, so theoretically data loss should be minimal.

Until there is a file-repairing command, finding the corrupt journal file and removing it is the only cure. You can find more on this in our Fedora mega-thread over making journal-only logging the default:

For tail corruptions your normal journalctl tool will provide you with as much information as is possible to salvage from the file. It will output the last complete log line and then finish. This is pretty close to how good you can get.

Things are different for corruptions in the middle. We have no nice tool for salvaging data from such corruption, but they could be written relatively easily. However, since they are highly unlikely due to the "append-only" model of the journal this hasn't been on our TODO list.

Of course, if you can identify what caused the problem initially and report it, that would be nice.

Improve this answer
7
  • what is an FAA? Commented Aug 23, 2013 at 4:40
  • @xenoterracide some sort of weird phone typo is what that is. :) Commented Aug 23, 2013 at 12:09
  • 1
    @slm Yeah, there's a long bus ride from the office. :) Sorry for the time wasting -- hope you at least learned something about aviation. Commented Aug 23, 2013 at 14:44
  • 1
    on my system at least corrupt journals seem to be common... is there any advice about how to identify what caused the problem? Commented Aug 23, 2013 at 21:43
  • 2
    On every systemd system I have seen, corrupt journal logs are common. This answer is now almost 5 years old and as far as I can tell, there is no solution for this issue. Commented Mar 26, 2018 at 19:37
3

It seems to be related to this thread on the ArchLinux wiki, titled: journalctl issues. It seems to have something to do with this setting SystemMaxUse in /etc/systemd/journald.conf.

The thread isn't conclusive but several people were having some luck in either clearing out the logs under /var/log/journal/* or increasing the value for SystemMaxUse.

Improve this answer
2
  • unfortunately playing with that value doesn't seem to have helped. I'm suspect clearing logs helps because it deletes the invalid logs. Commented Aug 23, 2013 at 0:14
  • @xenoterracide - I'll keep looking. Yeah cleaning out the logs seemed nonsensical to me too, but people were reporting it so I figured it was worth a shot. Commented Aug 23, 2013 at 1:10
2 
sudo journalctl --flush
sudo journalctl --verify

From manual:

Asks the journal daemon to flush any log data stored in /run/log/journal/ into /var/log/journal/, if persistent storage is enabled. This call does not return until the operation is complete. Note that this call is idempotent: the data is only flushed from /run/log/journal/ into /var/log/journal/ once during system runtime (but see --relinquish-var below), and this command exits cleanly without executing any operation if this has already happened. This command effectively guarantees that all data is flushed to /var/log/journal/ at the time it returns.

Added in version 217.

There is a related service, systemd-journal-flush.service.

Improve this answer
2
  • What does the journalctl --flush command do ? Commented Aug 2, 2023 at 9:13
  • i just quote from man journalctl : (--flush) == >Asks the journal daemon to flush any log data stored in /run/log/journal into /var/log/journal, if persistent storage is enabled. This call does not return until the operation is complete. Note that this call is idempotent: the data is only flushed from /run/log/journal into /var/log/journal once during system runtime, and this command exits cleanly without executing any operation if this has already happened. This command effectively guarantees that all data is flushed to /var/log/journal at the time it returns. Commented Aug 4, 2023 at 12:01

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.