0

The Problem

I have several identical servers all running identical logrotate configurations rotating a custom log file. On at least one of the servers my logs are not getting rotated correctly.

I have been troubleshooting this problem off and on for weeks now to no avail. Here is some more info and the steps that I have taken so far. Any help would be appreciated.

I am basically running out of ways to troubleshoot this problem and would love some ideas to try to see what is going wrong here.

What I've Tried

(Configuration files are below)

  1. Check configuration and permissions on the main logrotate file. Everything looks good to me there.
  2. Set permissions on the target directory, derp, to what logrotate needs (the directory can not be world writable afaik).

  3. Force logrotate to run with the following:

    logrotate --force /etc/logrotate.d/derp

    This command rotates the logs correctly but the next day they will not run.

  4. Manually change the date to sometime in the past in /var/lib/logrotate/status and then see if the log gets correctly run the next time logrotate kicks off or with:

    logrotate -vv /etc.logrotate.d/derp

    This will rotate the log once but the next time the log is up for rotation it does not get rotated. Here is the output of a successful manual rotate:

    rotating pattern: /derp/*.log  after 1 days (14 rotations)
empty log files are not rotated, old logs are removed
considering log /derp/access.log
log needs rotating
considering log /derp/php_errors.log
log does not need rotating
rotating log /derp/access.log, log->rotateCount is 14
dateext suffix '-20150814'
...
files getting renamed and moved
...
running prerotate script
renaming /derp/access.log to /derp/access.log.1
creating new /derp/access.log mode = 0774 uid = 0 gid = 4
running postrotate script

    Life is good, no? The only problem is that the next day, when it needs rotating (i.e., it always has data and is never empty) nothing happens.

  5. I have consulted many SEU questions among which was this question which was helpful in the past dealing with other logrotate issues.

Config

The system is running Debian 7 up to date. The process generating the log files is run as root. The process generating the log files is a php application and spawns multiple children.

Here are the relevant configurations, where derp is the location of my custom log files:

/etc/logrotate.d/derp

Permissions on the config file:

-rw-r--r-- 1 root root 265 Jul 9 2014 derp

/derp/*.log {
                daily
                missingok
                rotate 14
                compress
                delaycompress
                notifempty
                create 774 root adm
                sharedscripts
                postrotate
                                nohup /derp/daemonrestart.sh > /dev/null 2>&1
                endscript
                prerotate
                                killall -9 php > /dev/null 2>&1
                endscript
}

The prerotate and the postrotate scripts basically start the process (a socket server) or stops it and work fine on all of other servers. I am pretty sure that there are no problems there.

/var/lib/logrotate/status

(redacted to show relevant info)

logrotate state -- version 2
"/var/log/kern.log" 2015-7-20
...
"/derp/php_errors.log" 2015-8-11
...
"/derp/access.log" 2015-8-14
...

/derp/

Permissions on the target directory:

drwxr-xr-x 2 root root 4096 Jun 18 10:05 derp

And the relevant log located in \derp\:

-rwxrwxr-- 1 root adm 3755558 Aug 14 07:57 access.log

Improve this question
2
  • logrotate runs via a cron job (see dpkg -L logrotate for the file created in /etc/cron.daily). Is cron running correctly? If you have systemd, can you run systemctl status cron.service and verify that it's running? Commented Aug 14, 2015 at 14:06
  • I do see a logrotate file inside of /etc/cron.daily/ that looks valid. Additionally when the logs fail to get rotated as expected the date does get updated in /var/lib/logrotate/status. Commented Aug 14, 2015 at 14:13

2 Answers 2

Reset to default
1

You wrote:

Force logrotate to run with the following: logrotate --force /etc/logrotate.d/derp. This command rotates the logs correctly but the next day they will not run.

The problem here is that the daily attribute actually means "24 hours". So unless the cron entry for logrotate (probably in /etc/cron.d/daily) is more than 24 hours away from the time you manually rotate the log files, the log files will not be rotated until the day after that.

On my Debian system the entries in /etc/cron.daily are run daily at 06:25, so it's probable that any manual log rotation done during the normal working day will prevent a log rotation occurring until two days later.

Improve this answer
2
  • So the logs never rotate correctly whether or not I do the logrotate --force or not. I have also manually set the values in /var/lib/logrotate/status to indicate that the log needs rotating. Still nothing. This issue began to show up about 6 months ago on new systems created from a FAI (Fully Automated Install) install of linux. The last 10 systems out of over 40 started exhibiting this behavior. Commented Dec 8, 2015 at 18:14
  • @datUser incorrect date/time on the suspect systems? Commented Dec 9, 2015 at 16:35
1

Ok I gave up trying to fix this. Cron to the rescue.

sudo crontab -e

Then add the following line to the super user's crontab

# Logrotate they said...
3 6 * * *               /usr/sbin/logrotate -f /etc/logrotate.d/derp &> /dev/null

The -f flag forces logrotate to rotate the logs. The file located at /etc/logrotate.d/derp is the corresponding logrotate configuration for my application.

Its ugly, but it was the only way I could fix this bug.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.