3

I am running Nextcloud on Debian 11. Recently, I tried upgrading using php-updater, and afterwards MariaDB stopped working properly.

System details:

Debian 11

MariaDB 11.3.2

Apache with mpm_prefork (no php-fpm)

Nextcloud data in /var/www/owncloud/ (I am in the middle of a migration. Owncloud to Nextcloud worked so far, however, I wanted to use the php-updater script for the following Nextcloud upgrades)

Database: owncloud (visible via SHOW DATABASES; when MariaDB works)

What happened:

Right before this issue happened, I used php-updater. It said

It looks like both the 'Apache 2.0 Handler SAPI' libapache2-mod-php7.4 and the 'FastCGI Process Manager SAPI' php7.4-fpm are connected to it. This is a serious misconfiguration which must be fixed.
I: The Apache2 server has loaded the 'prefork' multi-processing module required by the Apache 2.0 Handler SAPI
   and the 'event' module, which is required for operation with php-fpm, is not loaded.

I used the option disconnect apache2 from php*-fpm, without changing the loaded modules. , MariaDB takes minutes to stop/start, or fails to start.

Log shows:

[ERROR] Unknown/unsupported storage engine: InnoDB
[ERROR] Aborting
  • My /var/lib/mysql/ currently contains:
ibdata1.bak (618M)
ibtmp1
undo001, undo002, undo003
(no ib_logfile0 / ib_logfile1 present anymore)
  • I tried adding a low-memory config (60-lowmem.cnf):
[mysqld]
innodb_force_recovery=6
innodb_buffer_pool_size = 16M
innodb_log_file_size    = 16M
innodb_log_buffer_size  = 8M
max_connections         = 10
table_open_cache        = 100
tmp_table_size          = 8M
max_heap_table_size     = 8M
skip-name-resolve
  • With innodb_force_recovery=6, MariaDB briefly started once, but after restoring configs, it fails again.

What I tried so far

  • Restored ibdata1 from ibdata1.bak
  • Tried creating fresh empty ib_logfile0 / ib_logfile1 (owned by mysql:mysql)
  • Adjusted permissions for all /var/lib/mysql/*
  • Tried different innodb_force_recovery levels (1–6)

Current state

  • MariaDB always fails with Unknown/unsupported storage engine: InnoDB.
  • I cannot get it to start reliably, even in recovery mode.
  • Nextcloud is down because the database is inaccessible.

My questions

  1. How can I force MariaDB to accept the existing ibdata1.bak (InnoDB system tablespace)?
  2. Is there a safe way to recreate the missing ib_logfile* files so MariaDB will start?
  3. If recovery is impossible, what’s the recommended way to salvage data from ibdata1.bak and the /var/lib/mysql/owncloud/ directory? I also have some backup data - at least before the Nextcloud migration, the relevant directories and a database backup were done. After the migration I backed up the data directory and once the database, when I once was able to restart the maria database.

Any hints or experience with recovering MariaDB after missing/corrupted InnoDB logs would be very much appreciated.

1 Answer 1

0

I was not able to solve the task by forcing MariaDB to accept the existing ibdata1.bak or recreate the ib_logfile*.

Solution that worked for me

In the process I restored from a clean backup. However, due to my new configuration or because it was wrongly configured elsewhere, I got a character set error after that. A workaround or solution of this issue helped.

  1. Remove database datadir

    As root, I (re-)moved /var/lib/mysql/mysql.

    mv /var/lib/mysql/mysql /var/lib/mysql/mysql.broken.$(date +%s)
    

    Then

    mariadb-install-db --user=mysql --basedir=/usr --datadir=/var/lib/mysql
    
    systemctl start mariadb
    
  2. Restore from a clean backup
    I restored from my latest mysqldump backup. I still have the old owncloud database user. Maybe the character set should have been configured here.

    mariadb -u root -p -e "DROP DATABASE IF EXISTS owncloud; CREATE DATABASE owncloud CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;"
    mariadb -u owncloud -p owncloud < owncloud_backup.sql
    

    However, then I got

    # sudo -u www-data php occ maintenance:mode --on 
    An unhandled exception has been thrown: 
    Doctrine\DBAL\Exception: Failed to connect to the database: An exception occurred in the driver: SQLSTATE[HY000] [2054] Server sent charset unknown to the client. Please, report to the developers in /var/www/owncloud/lib/private/DB/Connection.php:139
    

    Before a new import, I replaced all utf8mb3 collations in the dump with plain utf8 (or utf8mb4):

    sed -i 's/utf8mb3/utf8/g' owncloud_backup.sql
    
  3. Fix MariaDB server character set config
    In /etc/mysql/mariadb.conf.d/50-server.cnf, I reconfigured the character set:

    character-set-server = utf8mb4
    collation-server     = utf8mb4_general_ci
    skip-character-set-client-handshake
    

    Then restarted:

    systemctl restart mariadb
    
  4. Verify Nextcloud connectivity
    After the config change, Nextcloud was able to connect again. Maintenance mode could be enabled/disabled:

    sudo -u www-data php occ maintenance:mode --on
    sudo -u www-data php occ maintenance:mode --off
    

    Login worked afterwards.

Key takeaways

  • If MariaDB reports unknown/unsupported storage engine InnoDB, chances are the InnoDB system tablespace is corrupted beyond repair. Restore from mysqldump.
  • With SQLSTATE[HY000] [2054] Server sent charset unknown to the client, check for utf8mb3 leftovers in your dump. One can try character-set-server=utf8mb4 is set in MariaDB config.
  • Always keep a working backup routine — that’s what saved my Nextcloud instance here.

Hope this helps anyone hitting the same wall after an InnoDB crash.

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.