Error establishing a database connection after upgrade

[gah67]

I upgraded both my server and ee last night and since the reboot my wordpress site is down with 'Error establishing a database connection'.

The Mariadb container is continuously restarting and the reason is that there seems to be a queued transaction that it doesn't know how to deal with

The suggested solution in the error log is to run with the --tc-heuristic-recover flag

I have very little experience with Docker, but I understand that I need to somehow create a new Mariadb image with this flag on the run command and then override ee with the new container for the db (or possibly stop the existing container continually restarting (somehow) and run with just a shell and then manually run the command with the flag & repair)

All of the rest of the ee sites are working fine, just the wordpress ones are dead and I am pretty sure the database is intact other than the single error so I'd prefer to recover the system rather than trying to rebuild from scratch!

Any hints or pointers to further info would be gratefully received

I have found the similar error ( post 1503), where solutions were explained but the OP chose to reinstall I presume as no conclusion to that post

[dorain47]

Share MariaDB logs

[gah67]

2022-12-26 22:18:58+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.8.2+mariafocal started.
2022-12-26 22:18:58+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2022-12-26 22:18:58+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.8.2+mariafocal started.
2022-12-26 22:18:58+00:00 [Note] [Entrypoint]: MariaDB upgrade information missing, assuming required
2022-12-26 22:18:58+00:00 [Note] [Entrypoint]: MariaDB upgrade (mariadb-upgrade) required, but skipped due to $MARIADB_AUTO_UPGRADE setting
2022-12-26 22:18:58 0 [Note] mariadbd (server 10.8.2-MariaDB-1:10.8.2+maria~focal-log) starting as process 1 ...
2022-12-26 22:18:58 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-12-26 22:18:58 0 [Note] InnoDB: Number of transaction pools: 1
2022-12-26 22:18:58 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2022-12-26 22:18:58 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2022-12-26 22:18:59 0 [Note] InnoDB: Using Linux native AIO
2022-12-26 22:18:59 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
2022-12-26 22:18:59 0 [Note] InnoDB: Completed initialization of buffer pool
2022-12-26 22:18:59 0 [Note] InnoDB: Buffered log writes (block size=512 bytes)
2022-12-26 22:18:59 0 [Note] InnoDB: Transaction 5182566 was in the XA prepared state.
2022-12-26 22:18:59 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 0 row operations to undo
2022-12-26 22:18:59 0 [Note] InnoDB: Trx id counter is 5182567
2022-12-26 22:18:59 0 [Note] InnoDB: 128 rollback segments are active.
2022-12-26 22:18:59 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
2022-12-26 22:18:59 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
2022-12-26 22:18:59 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
2022-12-26 22:18:59 0 [Note] InnoDB: Rollback of non-prepared transactions completed
2022-12-26 22:18:59 0 [Note] InnoDB: log sequence number 5701367542; transaction id 5182568
2022-12-26 22:18:59 0 [Note] Plugin 'FEEDBACK' is disabled.
2022-12-26 22:18:59 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2022-12-26 22:18:59 0 [Note] InnoDB: Starting recovery for XA transactions...
2022-12-26 22:18:59 0 [Note] InnoDB: Transaction 5182566 in prepared state after recovery
2022-12-26 22:18:59 0 [Note] InnoDB: Transaction contains changes to 1 rows
2022-12-26 22:18:59 0 [Note] InnoDB: 1 transactions in prepared state after recovery
2022-12-26 22:18:59 0 [Note] Found 1 prepared transaction(s) in InnoDB
2022-12-26 22:18:59 0 [ERROR] Found 1 prepared transactions! It means that server was not shut down properly last time and critical recovery information (last binlog or tc.log file) was manually deleted after a crash. You have to start server with --tc-heuristic-recover switch to commit or rollback pending transactions.
2022-12-26 22:18:59 0 [ERROR] Aborting`

[mrrobot47]

Try the following:

ee service disable db
cd /opt/easyengine/services
docker-compose run --rm global-db mysqld --tc-heuristic-recover=COMMIT

If COMMIT does not work try using

docker-compose run --rm global-db mysqld --tc-heuristic-recover=ROLLBACK

Once, this recovers and you have exited from the docker-compose run command, run:

ee service enable db

[gah67]

Thank you so much ... worked perfectly

[mrrobot47]

Resolved via #1759 (reply in thread)