0

I am getting this error what ever I want to do on my server

cannot create temp file for here-document: Read-only file system

cpanel, whm and all websites are Down

All my system files are read-only. I search for anything that might help but it is not working :(

I have 200G of free disk space. My OS is Centos with WHM (Cpanel) It is on the Hetzner data center Rescue did not help

I cannot edit /etc/fstab because it is read-only.

vim /etc/fstab

my disks are

[root@server ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs         32G     0   32G   0% /dev
tmpfs            32G   25M   32G   1% /run
tmpfs            32G     0   32G   0% /sys/fs/cgroup
/dev/md2        874G  545G  285G  66% /
tmpfs           6.3G     0  6.3G   0% /run/user/0

I tried to remount, but it does not work either.

mount -o remount,rw /

output of 'dmesg | tail'

[    3.576436] IPv6: ADDRCONF(NETDEV_UP): enp0s31f6: link is not ready
[    6.969821] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[    6.970035] IPv6: ADDRCONF(NETDEV_CHANGE): enp0s31f6: link becomes ready
[   10.124036] ip6_tables: (C) 2000-2006 Netfilter Core Team
[   10.140747] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
[   42.683041] systemd-readahead[566]: Failed to open pack file: Read-only file system
[   60.065749] ip_set: protocol 7
[  302.063222] EXT4-fs (md2): error count since last fsck: 354
[  302.063288] EXT4-fs (md2): initial error at time 1622194023: ext4_xattr_block_get:321: inode 37361099
[  302.063461] EXT4-fs (md2): last error at time 1660967530: ext4_xattr_block_get:321: inode 37358850

output of 'journalctl -xe'

[root@server ~]# journalctl -xe
Jan 20 19:31:31 server.mm.com rsyslogd[1205]: action 'action 1' suspended, next retry is Fri Jan 20 19:32:01 2023 [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2007 ]
Jan 20 19:31:31 server.mm.com rsyslogd[1205]: imjournal: fopen() failed for path: '/var/lib/rsyslog/imjournal.state.tmp': Read-only file system [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2013 ]
Jan 20 19:31:31 server.mm.com PAM-hulk[5183]: failed to connect stream socket
Jan 20 19:31:31 server.mm.com sshd[5183]: pam_unix(sshd:auth): check pass; user unknown
Jan 20 19:31:31 server.mm.com sshd[5183]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=46.101.248.68
Jan 20 19:31:31 server.mm.com rsyslogd[1205]: imjournal: fopen() failed for path: '/var/lib/rsyslog/imjournal.state.tmp': Read-only file system [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2013 ]
Jan 20 19:31:31 server.mm.com rsyslogd[1205]: imjournal: fopen() failed for path: '/var/lib/rsyslog/imjournal.state.tmp': Read-only file system [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2013 ]
Jan 20 19:31:32 server.mm.com sshd[5183]: Failed password for invalid user amssys from 46.101.248.68 port 38188 ssh2
Jan 20 19:31:32 server.mm.com sshd[5183]: Received disconnect from 46.101.248.68 port 38188:11: Bye Bye [preauth]
Jan 20 19:31:32 server.mm.com sshd[5183]: Disconnected from 46.101.248.68 port 38188 [preauth]
Jan 20 19:31:33 server.mm.com PAM-hulk[5182]: failed to connect stream socket
Jan 20 19:31:33 server.mm.com sshd[5182]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.177.173.39  user=root
Jan 20 19:31:33 server.mm.com sshd[5182]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Jan 20 19:31:33 server.mm.com rsyslogd[1205]: imjournal: fopen() failed for path: '/var/lib/rsyslog/imjournal.state.tmp': Read-only file system [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2013 ]
Jan 20 19:31:34 server.mm.com sshd[5189]: Invalid user netadmin from 43.134.40.99 port 42568
Jan 20 19:31:34 server.mm.com sshd[5189]: input_userauth_request: invalid user netadmin [preauth]
Jan 20 19:31:34 server.mm.com PAM-hulk[5189]: failed to connect stream socket
Jan 20 19:31:34 server.mm.com sshd[5189]: pam_unix(sshd:auth): check pass; user unknown
Jan 20 19:31:34 server.mm.com sshd[5189]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=43.134.40.99
Jan 20 19:31:35 server.mm.com sshd[5182]: Failed password for root from 61.177.173.39 port 58331 ssh2
Jan 20 19:31:36 server.mm.com PAM-hulk[5182]: failed to connect stream socket
Jan 20 19:31:36 server.mm.com sshd[5182]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Jan 20 19:31:37 server.mm.com sshd[5189]: Failed password for invalid user netadmin from 43.134.40.99 port 42568 ssh2
Jan 20 19:31:37 server.mm.com rsyslogd[1205]: imjournal: fopen() failed for path: '/var/lib/rsyslog/imjournal.state.tmp': Read-only file system [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2013 ]
Jan 20 19:31:37 server.mm.com sshd[5189]: Received disconnect from 43.134.40.99 port 42568:11: Bye Bye [preauth]
Jan 20 19:31:37 server.mm.com sshd[5189]: Disconnected from 43.134.40.99 port 42568 [preauth]
Jan 20 19:31:38 server.mm.com sshd[5182]: Failed password for root from 61.177.173.39 port 58331 ssh2
Jan 20 19:31:38 server.mm.com PAM-hulk[5182]: failed to connect stream socket
Jan 20 19:31:38 server.mm.com sshd[5182]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Jan 20 19:34:36 server.mm.com rsyslogd[1205]: file '/var/log/messages': open error: Read-only file system [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2433 ]
Jan 20 19:34:36 server.mm.com systemd[1]: Started clamd antivirus daemon.
-- Subject: Unit clamd.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit clamd.service has finished starting up.
-- 
-- The start-up result is done.
Jan 20 19:34:36 server.mm.com rsyslogd[1205]: action 'action 0' resumed (module 'builtin:omfile') [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2359 ]
Jan 20 19:34:36 server.mm.com rsyslogd[1205]: file '/var/log/messages': open error: Read-only file system [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2433 ]
Jan 20 19:34:36 server.mm.com rsyslogd[1205]: action 'action 0' resumed (module 'builtin:omfile') [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2359 ]
Jan 20 19:34:36 server.mm.com rsyslogd[1205]: file '/var/log/messages': open error: Read-only file system [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2433 ]
Jan 20 19:34:36 server.mm.com rsyslogd[1205]: action 'action 0' resumed (module 'builtin:omfile') [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2359 ]
Jan 20 19:34:36 server.mm.com rsyslogd[1205]: file '/var/log/messages': open error: Read-only file system [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2433 ]
Jan 20 19:34:36 server.mm.com rsyslogd[1205]: action 'action 0' resumed (module 'builtin:omfile') [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2359 ]
Jan 20 19:34:36 server.mm.com rsyslogd[1205]: file '/var/log/messages': open error: Read-only file system [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2433 ]
Jan 20 19:34:36 server.mm.com rsyslogd[1205]: action 'action 0' resumed (module 'builtin:omfile') [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2359 ]
Jan 20 19:34:36 server.mm.com rsyslogd[1205]: file '/var/log/messages': open error: Read-only file system [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2433 ]
Jan 20 19:34:36 server.mm.com rsyslogd[1205]: action 'action 0' resumed (module 'builtin:omfile') [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2359 ]
Jan 20 19:34:36 server.mm.com rsyslogd[1205]: file '/var/log/messages': open error: Read-only file system [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2433 ]
Jan 20 19:34:36 server.mm.com rsyslogd[1205]: action 'action 0' resumed (module 'builtin:omfile') [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2359 ]
Jan 20 19:34:36 server.mm.com rsyslogd[1205]: file '/var/log/messages': open error: Read-only file system [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2433 ]
Jan 20 19:34:36 server.mm.com rsyslogd[1205]: action 'action 0' resumed (module 'builtin:omfile') [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2359 ]
Jan 20 19:34:36 server.mm.com rsyslogd[1205]: file '/var/log/messages': open error: Read-only file system [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2433 ]
Jan 20 19:34:36 server.mm.com rsyslogd[1205]: action 'action 0' suspended, next retry is Fri Jan 20 19:35:06 2023 [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2007 ]
Jan 20 19:34:36 server.mm.com rsyslogd[1205]: imjournal: fopen() failed for path: '/var/lib/rsyslog/imjournal.state.tmp': Read-only file system [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2013 ]
Jan 20 19:34:36 server.mm.com rsyslogd[1205]: imjournal: fopen() failed for path: '/var/lib/rsyslog/imjournal.state.tmp': Read-only file system [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2013 ]
Jan 20 19:34:36 server.mm.com rsyslogd[1205]: imjournal: fopen() failed for path: '/var/lib/rsyslog/imjournal.state.tmp': Read-only file system [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2013 ]

Solved: I contacted Hetzner support, and they fixed it for me appreciate!

Please use the following wiki guide to run the file system check:

-----------------%<-----------------
https://docs.hetzner.com/robot/dedicated-server/troubleshooting/filesystem-check/
Improve this question
4
  • Could you possibly include the actual error message you see when trying to open or save the file using vim, exactly as it is shown? It's currently unclear where the message in the title of your question comes from. Commented Jan 20, 2023 at 19:22
  • "/etc/fstab" E212: Can't open file for writing Commented Jan 20, 2023 at 19:25
  • @Kusalananda thank you for your comment I just edited the question and added the actual error Commented Jan 20, 2023 at 19:28
  • @kusalananda I added some details Commented Jan 20, 2023 at 19:52

1 Answer 1

Reset to default
0

if remount with rw permissions didn't work, try to boot the server in rescue mode and run a filesystem check(fsck) on the partition, and then try to remount it again. You can also try to repair the file system using the command e2fsck -p /dev/sda1 (assuming that /dev/sda1 is the partition that is having the problem).

Improve this answer
3
  • devtmpfs 32G 0 32G 0% /dev tmpfs 32G 25M 32G 1% /run tmpfs 32G 0 32G 0% /sys/fs/cgroup /dev/md2 874G 545G 285G 66% / tmpfs 6.3G 0 6.3G 0% /run/user/0 Commented Jan 20, 2023 at 19:36
  • These are my partitions; what command should I use for repair? Commented Jan 20, 2023 at 19:37
  • /dev/md2 is your physical device, execute fsck on that one, but it is recommended to do it on rescue mode or from an external system like a live linux installer. Commented Jan 20, 2023 at 19:55

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.