2

Several services are restarting without a cause I can determine. The Ubuntu 20.04.4 Server is running Nexus which is getting caught up in the restart frenzy. It is not a new VM and have been running on Ubuntu 20.04.4 for a while with patches applied last week. It is running on AWS.

I encountered the same issue about 8 months ago after setting up a new internal VMWare Ubuntu 20.04.x running Atlassian Bamboo. Pretty much the same log output as below except for replace Nexus service with Bamboo service. I was not able to determine the root cause, so I stood up a new VM and started fresh. It was not production yet, so I had that option.

At this point, I feel I need to determine the root cause and solution since it might be a re-occurring issue.

I am unable to determine the cause or troubleshooting steps to proceed.

This is an hour snippet of the logs, but it's pretty much the same for hours. Has anyone encountered this before or can provide advice on how to investigate this?

Aug  2 10:54:01 beta-nexus dbus-daemon[585]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.1989' (uid=0 pid=362880 comm="/usr/bin/hostnamectl " label="unconfined")
Aug  2 10:54:01 beta-nexus systemd[1]: Starting Hostname Service...
Aug  2 10:54:02 beta-nexus dbus-daemon[585]: [system] Successfully activated service 'org.freedesktop.hostname1'
Aug  2 10:54:02 beta-nexus systemd[1]: Started Hostname Service.
Aug  2 10:54:06 beta-nexus dbus-daemon[585]: [system] Activating via systemd: service name='org.freedesktop.PackageKit' unit='packagekit.service' requested by ':1.1991' (uid=0 pid=363670 comm="/usr/bin/gdbus call --system --dest org.freedeskto" label="unconfined")
Aug  2 10:54:06 beta-nexus systemd[1]: Starting PackageKit Daemon...
Aug  2 10:54:06 beta-nexus PackageKit: daemon start
Aug  2 10:54:06 beta-nexus dbus-daemon[585]: [system] Successfully activated service 'org.freedesktop.PackageKit'
Aug  2 10:54:06 beta-nexus systemd[1]: Started PackageKit Daemon.
Aug  2 10:54:11 beta-nexus systemd[1]: Stopping Nexus Service...
Aug  2 10:54:11 beta-nexus nexus[363835]: Shutting down nexus
Aug  2 10:54:14 beta-nexus nexus[363835]: Stopped.
Aug  2 10:54:14 beta-nexus systemd[1]: nexus.service: Succeeded.
Aug  2 10:54:14 beta-nexus systemd[1]: Stopped Nexus Service.
Aug  2 10:54:16 beta-nexus systemd[1]: Starting Nexus Service...
Aug  2 10:54:16 beta-nexus nexus[364093]: Starting nexus
Aug  2 10:54:16 beta-nexus systemd[1]: Started Nexus Service.
Aug  2 10:54:32 beta-nexus systemd[1]: systemd-hostnamed.service: Succeeded.
Aug  2 10:59:12 beta-nexus PackageKit: daemon quit
Aug  2 10:59:12 beta-nexus systemd[1]: packagekit.service: Succeeded.
Aug  2 11:17:01 beta-nexus CRON[364689]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Aug  2 11:24:22 beta-nexus dbus-daemon[585]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.2000' (uid=0 pid=364954 comm="/usr/bin/hostnamectl " label="unconfined")
Aug  2 11:24:22 beta-nexus systemd[1]: Starting Hostname Service...
Aug  2 11:24:22 beta-nexus dbus-daemon[585]: [system] Successfully activated service 'org.freedesktop.hostname1'
Aug  2 11:24:22 beta-nexus systemd[1]: Started Hostname Service.
Aug  2 11:24:27 beta-nexus dbus-daemon[585]: [system] Activating via systemd: service name='org.freedesktop.PackageKit' unit='packagekit.service' requested by ':1.2002' (uid=0 pid=365742 comm="/usr/bin/gdbus call --system --dest org.freedeskto" label="unconfined")
Aug  2 11:24:27 beta-nexus systemd[1]: Starting PackageKit Daemon...
Aug  2 11:24:27 beta-nexus PackageKit: daemon start
Aug  2 11:24:27 beta-nexus dbus-daemon[585]: [system] Successfully activated service 'org.freedesktop.PackageKit'
Aug  2 11:24:27 beta-nexus systemd[1]: Started PackageKit Daemon.
Aug  2 11:24:31 beta-nexus systemd[1]: Stopping Nexus Service...
Aug  2 11:24:31 beta-nexus nexus[365908]: Shutting down nexus
Aug  2 11:24:34 beta-nexus nexus[365908]: Stopped.
Aug  2 11:24:34 beta-nexus systemd[1]: nexus.service: Succeeded.
Aug  2 11:24:34 beta-nexus systemd[1]: Stopped Nexus Service.
Aug  2 11:24:36 beta-nexus systemd[1]: Starting Nexus Service...
Aug  2 11:24:36 beta-nexus nexus[366166]: Starting nexus
Aug  2 11:24:36 beta-nexus systemd[1]: Started Nexus Service.
Aug  2 11:24:53 beta-nexus systemd[1]: systemd-hostnamed.service: Succeeded.
Aug  2 11:29:32 beta-nexus PackageKit: daemon quit
Aug  2 11:29:32 beta-nexus systemd[1]: packagekit.service: Succeeded.
Aug  2 11:31:17 beta-nexus snapd[601]: storehelpers.go:722: cannot refresh: snap has no updates available: "amazon-ssm-agent", "core", "core18", "core20"
Aug  2 11:31:17 beta-nexus snapd[601]: autorefresh.go:539: auto-refresh: all snaps are up-to-date
Aug  2 11:39:45 beta-nexus systemd[1]: Starting Message of the Day...
Aug  2 11:39:46 beta-nexus 50-motd-news[366751]:  * Ubuntu ....
Aug  2 11:39:46 beta-nexus systemd[1]: motd-news.service: Succeeded.
Aug  2 11:39:46 beta-nexus systemd[1]: Finished Message of the Day.
Aug  2 11:54:44 beta-nexus dbus-daemon[585]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.2011' (uid=0 pid=367055 comm="/usr/bin/hostnamectl " label="unconfined")
Aug  2 11:54:44 beta-nexus systemd[1]: Starting Hostname Service...
Aug  2 11:54:44 beta-nexus dbus-daemon[585]: [system] Successfully activated service 'org.freedesktop.hostname1'
Aug  2 11:54:44 beta-nexus systemd[1]: Started Hostname Service.
Aug  2 11:54:48 beta-nexus dbus-daemon[585]: [system] Activating via systemd: service name='org.freedesktop.PackageKit' unit='packagekit.service' requested by ':1.2013' (uid=0 pid=367843 comm="/usr/bin/gdbus call --system --dest org.freedeskto" label="unconfined")
Aug  2 11:54:48 beta-nexus systemd[1]: Starting PackageKit Daemon...
Aug  2 11:54:48 beta-nexus PackageKit: daemon start
Aug  2 11:54:48 beta-nexus dbus-daemon[585]: [system] Successfully activated service 'org.freedesktop.PackageKit'
Aug  2 11:54:48 beta-nexus systemd[1]: Started PackageKit Daemon.
Aug  2 11:54:55 beta-nexus systemd[1]: Stopping Nexus Service...
Aug  2 11:54:55 beta-nexus nexus[368017]: Shutting down nexus
Aug  2 11:54:58 beta-nexus nexus[368017]: Stopped.
Aug  2 11:54:58 beta-nexus systemd[1]: nexus.service: Succeeded.
Aug  2 11:54:58 beta-nexus systemd[1]: Stopped Nexus Service.
Aug  2 11:55:00 beta-nexus systemd[1]: Starting Nexus Service...
Aug  2 11:55:00 beta-nexus nexus[368276]: Starting nexus
Aug  2 11:55:00 beta-nexus systemd[1]: Started Nexus Service.
Aug  2 11:55:14 beta-nexus systemd[1]: systemd-hostnamed.service: Succeeded.

This was originally posted in stackoverflow, but a friendly suggestion was made to post it here.

EDIT: Added nexus.service file. I have confirmed that it is identical to the other two Nexus servers running on Ubuntu 20.04.4 that are not having this issue.

[Unit]
Description = Nexus Service
After = network.target

[Service]
Type = forking
LimitNOFILE = 65536
WorkingDirectory = /opt/sonatype/nexus
ExecStart = /opt/sonatype/nexus/bin/nexus start
ExecStop = /opt/sonatype/nexus/bin/nexus stop
User = nexus
Group = nexus
Restart = on-abort

[Install]
WantedBy = multi-user.target

EDIT: Additional information that may or may not be helpful... I have three Nexus servers. All maintained by Chef. All Ubuntu 20.04.4. One on internal VMWare and two on AWS. The VMWare and the other AWS setups are running normally. Also, I need to correct myself that it was NOT recently upgraded to Ubuntu 20.04.4, but patched last week along with the others.

EDIT: Added timers listed from systemctl list-timers:

NEXT                        LEFT          LAST                        PASSED       UNIT                         ACTIVATES                     
Wed 2022-08-03 06:59:48 EDT 1min 4s left  Tue 2022-08-02 06:51:03 EDT 24h ago      apt-daily-upgrade.timer      apt-daily-upgrade.service     
Wed 2022-08-03 07:10:24 EDT 11min left    Tue 2022-08-02 20:07:47 EDT 10h ago      fwupd-refresh.timer          fwupd-refresh.service         
Wed 2022-08-03 10:19:46 EDT 3h 21min left Wed 2022-08-03 04:04:26 EDT 2h 54min ago ua-timer.timer               ua-timer.service              
Wed 2022-08-03 14:12:57 EDT 7h left       Tue 2022-08-02 14:12:57 EDT 16h ago      systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Wed 2022-08-03 16:48:43 EDT 9h left       Tue 2022-08-02 18:26:57 EDT 12h ago      apt-daily.timer              apt-daily.service             
Wed 2022-08-03 21:01:28 EDT 14h left      Wed 2022-08-03 04:30:02 EDT 2h 28min ago motd-news.timer              motd-news.service             
Thu 2022-08-04 00:00:00 EDT 17h left      Wed 2022-08-03 00:00:37 EDT 6h ago       logrotate.timer              logrotate.service             
Thu 2022-08-04 00:00:00 EDT 17h left      Wed 2022-08-03 00:00:37 EDT 6h ago       man-db.timer                 man-db.service                
Sun 2022-08-07 03:10:37 EDT 3 days left   Sun 2022-07-31 03:10:29 EDT 3 days ago   e2scrub_all.timer            e2scrub_all.service           
Mon 2022-08-08 00:00:00 EDT 4 days left   Mon 2022-08-01 00:00:47 EDT 2 days ago   fstrim.timer                 fstrim.service

EDIT: Added systemd journal snippet. The dominos start with the same entry:

Aug 03 06:36:35 beta-nexus.******.tld dbus-daemon[599]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.365' (uid=0 pid=65537 comm="/usr/bin/hostnamectl " label="unconfined")
Aug 03 06:36:35 beta-nexus.******.tld systemd[1]: Starting Hostname Service...
-- Subject: A start job for unit systemd-hostnamed.service has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit systemd-hostnamed.service has begun execution.
-- 
-- The job identifier is 7793.
Aug 03 06:36:35 beta-nexus.******.tld dbus-daemon[599]: [system] Successfully activated service 'org.freedesktop.hostname1'
Aug 03 06:36:35 beta-nexus.******.tld systemd[1]: Started Hostname Service.
-- Subject: A start job for unit systemd-hostnamed.service has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit systemd-hostnamed.service has finished successfully.
-- 
-- The job identifier is 7793.
Aug 03 06:36:39 beta-nexus.******.tld dbus-daemon[599]: [system] Activating via systemd: service name='org.freedesktop.PackageKit' unit='packagekit.service' requested by ':1.367' (uid=0 pid=66327 comm="/usr/bin/gdbus call --system --dest org.freedeskto" label="unconfined")
Aug 03 06:36:39 beta-nexus.******.tld systemd[1]: Starting PackageKit Daemon...
-- Subject: A start job for unit packagekit.service has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit packagekit.service has begun execution.
-- 
-- The job identifier is 7861.
Aug 03 06:36:39 beta-nexus.******.tld PackageKit[66330]: daemon start
Aug 03 06:36:39 beta-nexus.******.tld dbus-daemon[599]: [system] Successfully activated service 'org.freedesktop.PackageKit'
Aug 03 06:36:39 beta-nexus.******.tld systemd[1]: Started PackageKit Daemon.
-- Subject: A start job for unit packagekit.service has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit packagekit.service has finished successfully.
-- 
-- The job identifier is 7861.
Aug 03 06:36:46 beta-nexus.******.tld systemd[1]: Stopping Nexus Service...
-- Subject: A stop job for unit nexus.service has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A stop job for unit nexus.service has begun execution.
-- 
-- The job identifier is 7929.
Aug 03 06:36:46 beta-nexus.******.tld nexus[66504]: Shutting down nexus
Aug 03 06:36:49 beta-nexus.******.tld nexus[66504]: Stopped.
Aug 03 06:36:49 beta-nexus.******.tld systemd[1]: nexus.service: Succeeded.
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit nexus.service has successfully entered the 'dead' state.
Aug 03 06:36:49 beta-nexus.******.tld systemd[1]: Stopped Nexus Service.
-- Subject: A stop job for unit nexus.service has finished
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A stop job for unit nexus.service has finished.
-- 
-- The job identifier is 7929 and the job result is done.
Aug 03 06:36:51 beta-nexus.******.tld systemd[1]: Starting Nexus Service...
-- Subject: A start job for unit nexus.service has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit nexus.service has begun execution.
-- 
-- The job identifier is 7930.
Aug 03 06:36:51 beta-nexus.******.tld nexus[66763]: Starting nexus
Aug 03 06:36:51 beta-nexus.******.tld systemd[1]: Started Nexus Service.
-- Subject: A start job for unit nexus.service has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit nexus.service has finished successfully.
-- 
-- The job identifier is 7930.
Aug 03 06:37:05 beta-nexus.******.tld systemd[1]: systemd-hostnamed.service: Succeeded.
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit systemd-hostnamed.service has successfully entered the 'dead' state.
Aug 03 06:41:44 beta-nexus.******.tld PackageKit[66330]: daemon quit
Aug 03 06:41:44 beta-nexus.******.tld systemd[1]: packagekit.service: Succeeded.
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit packagekit.service has successfully entered the 'dead' state.
Improve this question
3
  • What does the systemd service file for Nexus look like? Does any other service file mention Nexus? Commented Aug 2, 2022 at 19:27
  • @thrig, I've updated the original comment with the service file. Not other service file references nexus. I have 3 Nexus servers maintained by Chef and I have manually confirmed the nexus.service files on all three servers are the same. Commented Aug 2, 2022 at 19:45
  • 1
    Anything when listing timers systemctl list-timers? Commented Aug 2, 2022 at 20:09

1 Answer 1

Reset to default
1

It seems like it's related to packagekit. The restart of nexus is about 5 seconds after the start of packagekit.

I wonder if you can figure out what this PID is:

Aug 2 11:54:48 beta-nexus dbus-daemon[585]: ... (uid=0 pid=367843 comm="/usr/bin/gdbus call --system --dest org.freedeskto" label="unconfined")
Improve this answer
1
  • Thank you for your suggestion. Following your suggestion of investigating the pid, I used ps ax which showed that chef-client was running as a daemon and executing every 30 minutes. This must have been configured automatically when I downgraded the client using a deb package. /opt/chef/embedded/bin/ruby --disable-gems /usr/bin/chef-client -d -P /var/run/chef/client.pid -c /etc/chef/client.rb -i 1800 -s 20 -L /var/log/chef/client.log Commented Aug 3, 2022 at 15:23

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.