Failure to connect to MySQL while attempting Nutmeg -> Olive update

[ABC]

On a different server, I am for some reason having issues when running tutor local launch after updating from tutor Nutmeg (14.0.3) to Olive (via pip install --upgrade "tutor[full]==v15.3.7").

The observed error is:

Initialising MySQL...
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2005 (HY000): Unknown MySQL server host 'mysql' (11)
    [1/10] Waiting for MySQL service (this may take a while)...
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2005 (HY000): Unknown MySQL server host 'mysql' (11)
[cut]
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2005 (HY000): Unknown MySQL server host 'mysql' (11)
    [9/10] Waiting for MySQL service (this may take a while)...
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2005 (HY000): Unknown MySQL server host 'mysql' (11)
    [10/10] Waiting for MySQL service (this may take a while)...
MySQL initialisation error

And the logged errors seem to be a rather severe crash:

mysql_1                      | 2023-10-10 13:00:11+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.35-1debian10 started.
mysql_1                      | 2023-10-10T13:00:12.255591Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
mysql_1                      | 2023-10-10T13:00:12.258620Z 0 [Note] mysqld (mysqld 5.7.35) starting as process 1 ...
mysql_1                      | 2023-10-10T13:00:12.273067Z 0 [Note] InnoDB: PUNCH HOLE support available
mysql_1                      | 2023-10-10T13:00:12.273172Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
mysql_1                      | 2023-10-10T13:00:12.273206Z 0 [Note] InnoDB: Uses event mutexes
mysql_1                      | 2023-10-10T13:00:12.273263Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
mysql_1                      | 2023-10-10T13:00:12.273324Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
mysql_1                      | 2023-10-10T13:00:12.273359Z 0 [Note] InnoDB: Using Linux native AIO
mysql_1                      | 2023-10-10T13:00:12.275422Z 0 [Note] InnoDB: Number of pools: 1
mysql_1                      | 2023-10-10T13:00:12.276474Z 0 [Note] InnoDB: Using CPU crc32 instructions
mysql_1                      | 2023-10-10T13:00:12.279330Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
mysql_1                      | 2023-10-10T13:00:12.295446Z 0 [Note] InnoDB: Completed initialization of buffer pool
mysql_1                      | 2023-10-10T13:00:12.299116Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
mysql_1                      | 2023-10-10T13:00:12.319902Z 0 [ERROR] [FATAL] InnoDB: Table flags are 0 in the data dictionary but the flags in file ./ibdata1 are 0x4000!
mysql_1                      | 2023-10-10 13:00:12 0x7fa91cc42740  InnoDB: Assertion failure in thread 140364308817728 in file ut0ut.cc line 921
mysql_1                      | InnoDB: We intentionally generate a memory trap.
mysql_1                      | InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
mysql_1                      | InnoDB: If you get repeated assertion failures or crashes, even
mysql_1                      | InnoDB: immediately after the mysqld startup, there may be
mysql_1                      | InnoDB: corruption in the InnoDB tablespace. Please refer to
mysql_1                      | InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
mysql_1                      | InnoDB: about forcing recovery.
mysql_1                      | 13:00:12 UTC - mysqld got signal 6 ;
mysql_1                      | This could be because you hit a bug. It is also possible that this binary
mysql_1                      | or one of the libraries it was linked against is corrupt, improperly built,
mysql_1                      | or misconfigured. This error can also be caused by malfunctioning hardware.
mysql_1                      | Attempting to collect some information that could help diagnose the problem.
mysql_1                      | As this is a crash and something is definitely wrong, the information
mysql_1                      | collection process might fail.
mysql_1                      | 
mysql_1                      | key_buffer_size=8388608
mysql_1                      | read_buffer_size=131072
mysql_1                      | max_used_connections=0
mysql_1                      | max_threads=151
mysql_1                      | thread_count=0
mysql_1                      | connection_count=0
mysql_1                      | It is possible that mysqld could use up to 
mysql_1                      | key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68197 K  bytes of memory
mysql_1                      | Hope that's ok; if not, decrease some variables in the equation.
mysql_1                      | 
mysql_1                      | Thread pointer: 0x0
mysql_1                      | Attempting backtrace. You can use the following information to find out
mysql_1                      | where mysqld died. If you see no messages after this, something went
mysql_1                      | terribly wrong...
mysql_1                      | stack_bottom = 0 thread_stack 0x40000
mysql_1                      | mysqld(my_print_stacktrace+0x2c)[0x55a0a32311bc]
mysql_1                      | mysqld(handle_fatal_signal+0x501)[0x55a0a2b31bc1]
mysql_1                      | /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7fa91d1a3730]
mysql_1                      | /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7fa91cc7e7bb]
mysql_1                      | /lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7fa91cc69535]
mysql_1                      | mysqld(+0x6c3a8b)[0x55a0a2af8a8b]
mysql_1                      | mysqld(+0x6c3e69)[0x55a0a2af8e69]
mysql_1                      | mysqld(+0x11ca23c)[0x55a0a35ff23c]
mysql_1                      | mysqld(+0x11ca86e)[0x55a0a35ff86e]
mysql_1                      | mysqld(_Z6fil_ioRK9IORequestbRK9page_id_tRK11page_size_tmmPvS8_+0x312)[0x55a0a3607372]
mysql_1                      | mysqld(+0x118ac98)[0x55a0a35bfc98]
mysql_1                      | mysqld(_Z13buf_read_pageRK9page_id_tRK11page_size_t+0x31)[0x55a0a35c02c1]
mysql_1                      | mysqld(_Z16buf_page_get_genRK9page_id_tRK11page_size_tmP11buf_block_tmPKcmP5mtr_tb+0x409)[0x55a0a3594249]
mysql_1                      | mysqld(_Z31trx_rseg_get_n_undo_tablespacesPm+0x13e)[0x55a0a3530e9e]
mysql_1                      | mysqld(+0x10cdb47)[0x55a0a3502b47]
mysql_1                      | mysqld(_Z34innobase_start_or_create_for_mysqlv+0x2f67)[0x55a0a3506707]
mysql_1                      | mysqld(+0xfa3dd0)[0x55a0a33d8dd0]
mysql_1                      | mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x55)[0x55a0a2b83705]
mysql_1                      | mysqld(+0xbf6c26)[0x55a0a302bc26]
mysql_1                      | mysqld(_Z40plugin_register_builtin_and_init_core_sePiPPc+0x1dc)[0x55a0a302dafc]
mysql_1                      | mysqld(+0x6f738e)[0x55a0a2b2c38e]
mysql_1                      | mysqld(_Z11mysqld_mainiPPc+0x758)[0x55a0a2b2d7f8]
mysql_1                      | /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x7fa91cc6b09b]
mysql_1                      | mysqld(_start+0x2a)[0x55a0a2b2283a]
mysql_1                      | The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
mysql_1                      | information that should help you find out what is causing the crash.
mysql_1                      | 2023-10-10 13:01:13+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.35-1debian10 started.

Is there something I can do to re-initialize and possibly get around this error?

[ABC]

I hit this error again (including crash in the log) on one of my servers I’m trying to upgrade (from a fresh snapshot), but not the other (where I successfully upgraded to Olive). This implies there’s something about the data on this server (which started at Nutmeg instead of Lilac) which is causing it to break. Perhaps this has to do with me following the earlier instructions on how to get utf8mb4 working? Do I need to somehow roll that back? (cc @dave for opinions)

Update: On a hunch I tried just upgrading Nutmeg 14.0.3 to 14.2.5, and it exhibits the same error: crash & subsequent failure to connect.

[dave (Dave Ormsbee)]

Are you running MySQL 8.0.34 or higher? @regis did some investigation that determined that 8.0.33 and earlier had issues that cause problems during the upgrade process, but I don’t remember the specifics.

[ABC]

ubuntu@ip-172-31-44-20:~$ tutor local run mysql bash
docker-compose -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.yml -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.prod.yml -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.tmp.yml -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.override.yml --project-name tutor_local run --rm mysql bash
Creating tutor_local_mysql_run ... done
bash-4.4# mysql --version
mysql  Ver 8.0.30 for Linux on x86_64 (MySQL Community Server - GPL)

Looks like your hunch is right, while I’m on tutor 14.0.3 I have 8.0.30 (which I guess is non-standard, according to reading other threads? Nutmeg is supposed to have MySQL 5.7x? In which case this does appear to be because I needed Emoji support in classes last year, and I ran through the steps I found on the forum to enable them.)

@regis How can I update just the MySQL to 8.0.34 before attempting upgrade?

[regis (Régis Behmo)]

Do you have a backup of your database? Because in my experience, upgrading to 8.0.33 and applying migrations without a server restart will corrupt your database. Here’s a more extensive description: fix: broken mysql after palm upgrade by regisb · Pull Request #890 · overhangio/tutor · GitHub

[ABC]

At some point I used the mysql8utf8mb4 plugin from here: Upgrading MySQL charset to utf8mb4 - #2 by Abdess in order to get around the Open edX emoji regression that occurred from Maple → Nutmeg. During upgrades I normally disable all my plugins, since 1) I don’t know if they will work with the new version and 2) I’ve had errors in the past due to incompatible plugins.

It turns out that the root cause of these mysql errors was not having this plugin enabled across upgrades! So if anyone ever knows they changed over to the MySQL 4-byte support, you need to make sure the mysql8utf8mb4 plugin is enabled across upgrades!. After doing that, I could upgrade from Nutmeg → Olive → Palm → Quince with no problems!

cc @dave , @Abdess from original mysql fix thread as an FYI