Containers keep restarting and cant connect to mysql

Development
1

Hello everyone, I’m trying to run “tutor local launch” but I’m having problems to connect at mysql.

Running init task in mysql
docker-compose -f /home/julio/.local/share/tutor/env/local/docker-compose.yml -f /home/julio/.local/share/tutor/env/local/docker-compose.prod.yml -f /home/julio/.local/share/tutor/env/local/docker-compose.tmp.yml --project-name tutor_local -f /home/julio/.local/share/tutor/env/local/docker-compose.jobs.yml -f /home/julio/.local/share/tutor/env/local/docker-compose.jobs.tmp.yml run --rm mysql-job sh -e -c 'echo "Initialising MySQL..."
mysql_connection_max_attempts=10
mysql_connection_attempt=0
until mysql -u root --password="aUjlT7h4" --host "mysql" --port 3306 -e '"'"'exit'"'"'
do
    mysql_connection_attempt=$(expr $mysql_connection_attempt + 1)
    echo "    [$mysql_connection_attempt/$mysql_connection_max_attempts] Waiting for MySQL service (this may take a while)..."
    if [ $mysql_connection_attempt -eq $mysql_connection_max_attempts ]
    then
      echo "MySQL initialisation error" 1>&2
      exit 1
    fi
    sleep 10
done
echo "MySQL is up and running"

# edx-platform database
mysql -u root --password="aUjlT7h4" --host "mysql" --port 3306 -e "CREATE DATABASE IF NOT EXISTS openedx;"
mysql -u root --password="aUjlT7h4" --host "mysql" --port 3306 -e "CREATE USER IF NOT EXISTS '"'"'openedx'"'"';"
mysql -u root --password="aUjlT7h4" --host "mysql" --port 3306 -e "ALTER USER '"'"'openedx'"'"'@'"'"'%'"'"' IDENTIFIED BY '"'"'QMPj9KKI'"'"';"
mysql -u root --password="aUjlT7h4" --host "mysql" --port 3306 -e "GRANT ALL ON openedx.* TO '"'"'openedx'"'"'@'"'"'%'"'"';"'
Creating tutor_local_mysql-job_run ... done
Initialising MySQL...
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2005 (HY000): Unknown MySQL server host 'mysql' (11)
    [1/10] Waiting for MySQL service (this may take a while)...
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2005 (HY000): Unknown MySQL server host 'mysql' (11)
    [2/10] Waiting for MySQL service (this may take a while)...
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2005 (HY000): Unknown MySQL server host 'mysql' (11)
    [3/10] Waiting for MySQL service (this may take a while)...
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2005 (HY000): Unknown MySQL server host 'mysql' (11)
    [4/10] Waiting for MySQL service (this may take a while)...
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2005 (HY000): Unknown MySQL server host 'mysql' (11)
    [5/10] Waiting for MySQL service (this may take a while)...
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2005 (HY000): Unknown MySQL server host 'mysql' (11)
    [6/10] Waiting for MySQL service (this may take a while)...
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2005 (HY000): Unknown MySQL server host 'mysql' (11)
    [7/10] Waiting for MySQL service (this may take a while)...
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2005 (HY000): Unknown MySQL server host 'mysql' (11)
    [8/10] Waiting for MySQL service (this may take a while)...
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2005 (HY000): Unknown MySQL server host 'mysql' (11)
    [9/10] Waiting for MySQL service (this may take a while)...
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2005 (HY000): Unknown MySQL server host 'mysql' (11)
    [10/10] Waiting for MySQL service (this may take a while)...
MySQL initialisation error

and i realized that some container is always restarting…

CONTAINER ID   IMAGE                           COMMAND                  CREATED          STATUS                           PORTS                                                                                         NAMES
3a3f2535b43d   overhangio/openedx:15.3.6       "celery --app=cms.ce…"   16 minutes ago   Up 3 minutes                     8000/tcp                                                                                      tutor_local_cms-worker_1
aec10e94fbf1   overhangio/openedx-mfe:15.0.7   "caddy run --config …"   16 minutes ago   Up 3 minutes                     80/tcp, 443/tcp, 2019/tcp, 443/udp                                                            tutor_local_mfe_1
bb38cdc0bb6c   overhangio/openedx:15.3.6       "celery --app=lms.ce…"   16 minutes ago   Up 3 minutes                     8000/tcp                                                                                      tutor_local_lms-worker_1
368578b49d41   overhangio/openedx:15.3.6       "/bin/sh -c 'uwsgi u…"   16 minutes ago   Up 3 minutes                     8000/tcp                                                                                      tutor_local_cms_1
f1e7355141c4   overhangio/openedx:15.3.6       "/bin/sh -c 'uwsgi u…"   16 minutes ago   Up 3 minutes                     8000/tcp                                                                                      tutor_local_lms_1
b5cb2fcef9fd   mongo:4.2.17                    "docker-entrypoint.s…"   16 minutes ago   Restarting (62) 47 seconds ago                                                                                                 tutor_local_mongodb_1
970b66a80a67   redis:6.2.6                     "docker-entrypoint.s…"   16 minutes ago   Up 3 minutes                     6379/tcp                                                                                      tutor_local_redis_1
433a57f8f716   elasticsearch:7.10.1            "/tini -- /usr/local…"   16 minutes ago   Restarting (1) 11 seconds ago                                                                                                  tutor_local_elasticsearch_1
6db8bbf595a5   devture/exim-relay:4.95-r0-2    "/sbin/tini -- exim …"   17 minutes ago   Up 4 minutes                     8025/tcp                                                                                      tutor_local_smtp_1
b8c9e60acd84   caddy:2.6.3                     "caddy run --config …"   17 minutes ago   Up 4 minutes                     0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 443/udp, 2019/tcp   tutor_local_caddy_1
b772c817345c   mysql:5.7.35                    "docker-entrypoint.s…"   17 minutes ago   Restarting (2) 59 seconds ago

I runned the mysql container solo, using docker run and it didnt restarting and i’ve got connect in mysql using DBeaver. So, I think i cant connect because the containers keeps restarting.

I have the same project in a VM and there i have no problems , i dont know why in my pc it doesnt work.

Tutor logs mysql:

e.tmp.yml --project-name tutor_local logs --tail 50 mysql
Attaching to tutor_local_mysql_1
mysql_1                      | InnoDB: about forcing recovery.
mysql_1                      | 14:26:44 UTC - mysqld got signal 6 ;
mysql_1                      | This could be because you hit a bug. It is also possible that this binary
mysql_1                      | or one of the libraries it was linked against is corrupt, improperly built,
mysql_1                      | or misconfigured. This error can also be caused by malfunctioning hardware.
mysql_1                      | Attempting to collect some information that could help diagnose the problem.
mysql_1                      | As this is a crash and something is definitely wrong, the information
mysql_1                      | collection process might fail.
mysql_1                      | 
mysql_1                      | key_buffer_size=8388608
mysql_1                      | read_buffer_size=131072
mysql_1                      | max_used_connections=0
mysql_1                      | max_threads=151
mysql_1                      | thread_count=0
mysql_1                      | connection_count=0
mysql_1                      | It is possible that mysqld could use up to 
mysql_1                      | key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68197 K  bytes of memory
mysql_1                      | Hope that's ok; if not, decrease some variables in the equation.
mysql_1                      | 
mysql_1                      | Thread pointer: 0x0
mysql_1                      | Attempting backtrace. You can use the following information to find out
mysql_1                      | where mysqld died. If you see no messages after this, something went
mysql_1                      | terribly wrong...
mysql_1                      | stack_bottom = 0 thread_stack 0x40000
mysql_1                      | mysqld(my_print_stacktrace+0x2c)[0x5646d59e71bc]
mysql_1                      | mysqld(handle_fatal_signal+0x501)[0x5646d52e7bc1]
mysql_1                      | /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f897aaef730]
mysql_1                      | /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7f897a5ca7bb]
mysql_1                      | /lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7f897a5b5535]
mysql_1                      | mysqld(+0x6c3a8b)[0x5646d52aea8b]
mysql_1                      | mysqld(+0x6c3e69)[0x5646d52aee69]
mysql_1                      | mysqld(+0x11ca23c)[0x5646d5db523c]
mysql_1                      | mysqld(+0x11ca86e)[0x5646d5db586e]
mysql_1                      | mysqld(_Z6fil_ioRK9IORequestbRK9page_id_tRK11page_size_tmmPvS8_+0x312)[0x5646d5dbd372]
mysql_1                      | mysqld(+0x118ac98)[0x5646d5d75c98]
mysql_1                      | mysqld(_Z13buf_read_pageRK9page_id_tRK11page_size_t+0x31)[0x5646d5d762c1]
mysql_1                      | mysqld(_Z16buf_page_get_genRK9page_id_tRK11page_size_tmP11buf_block_tmPKcmP5mtr_tb+0x409)[0x5646d5d4a249]
mysql_1                      | mysqld(_Z31trx_rseg_get_n_undo_tablespacesPm+0x13e)[0x5646d5ce6e9e]
mysql_1                      | mysqld(+0x10cdb47)[0x5646d5cb8b47]
mysql_1                      | mysqld(_Z34innobase_start_or_create_for_mysqlv+0x2f67)[0x5646d5cbc707]
mysql_1                      | mysqld(+0xfa3dd0)[0x5646d5b8edd0]
mysql_1                      | mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x55)[0x5646d5339705]
mysql_1                      | mysqld(+0xbf6c26)[0x5646d57e1c26]
mysql_1                      | mysqld(_Z40plugin_register_builtin_and_init_core_sePiPPc+0x1dc)[0x5646d57e3afc]
mysql_1                      | mysqld(+0x6f738e)[0x5646d52e238e]
mysql_1                      | mysqld(_Z11mysqld_mainiPPc+0x758)[0x5646d52e37f8]
mysql_1                      | /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x7f897a5b709b]
mysql_1                      | mysqld(_start+0x2a)[0x5646d52d883a]
mysql_1                      | The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
mysql_1                      | information that should help you find out what is causing the crash.
1 Like
2

Did you upgrade an existing installation or launch a new platform from scratch?

3

I have the same problem after upgrading to palm

1 Like
4

I am facing the same issue in a fresh install of tutor 15 (olive):

pip freeze | grep tutor
tutor==15.3.8
tutor-android==15.0.0
tutor-cairn==15.0.6
tutor-contrib-codejail @ git+https://github.com/edunext/tutor-contrib-codejail@0de72c65d6032cdb536cf13e757cea95f3f3874c
tutor-discovery==15.0.0
tutor-ecommerce==15.0.2
tutor-forum==15.0.0
tutor-license==15.0.0
tutor-mfe==15.0.4
tutor-minio==15.0.0
tutor-notes==15.0.0
tutor-webui==15.0.0
tutor-xqueue==15.0.0

I have followed the steps at Troubleshooting — Tutor documentation, and also pruned all docker containers and volumes and restarted everything but I always encounter this issue.

1 Like
5

@JulioLeal @martibosch
It does feel like the real issue is the amount of free memory dedicated to the running instances. I guess that it’s the original “scalable” version of the system which is assumed to run with containers orchestration (kubernetes), so it’s supposed to run each docker image in separate virtual machine. I have the same problem and it looks like I just need to increase the allocated memory.

Or someone solved this in another way?

6

Hello. I have the same issue.

After running docker ps | grep mysql and tutor local logs --tail=100 mysql , I have these:

mysql-1  | 2024-05-30 20:33:11+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.1.0-1.el8 started.
mysql-1  | '/var/lib/mysql/mysql.sock' -> '/var/run/mysqld/mysqld.sock'
mysql-1  | 2024-05-30T20:33:11.757960Z 0 [System] [MY-015015] [Server] MySQL Server - start.
mysql-1  | 2024-05-30T20:33:12.053581Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead.
mysql-1  | 2024-05-30T20:33:12.055794Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.1.0) starting as process 1
mysql-1  | 2024-05-30T20:33:12.057645Z 0 [Warning] [MY-013243] [Server] --character-set-server: The character set UTF8MB3 is deprecated and will be removed in a future release. Please consider using UTF8MB4 instead.
mysql-1  | 2024-05-30T20:33:12.057654Z 0 [Warning] [MY-013244] [Server] --collation-server: 'utf8mb3_general_ci' is a collation of the deprecated character set UTF8MB3. Please consider using UTF8MB4 with an appropriate collation instead.
mysql-1  | 2024-05-30T20:33:12.063397Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
mysql-1  | 2024-05-30T20:33:12.897324Z 1 [ERROR] [MY-012526] [InnoDB] Upgrade is not supported after a crash or shutdown with innodb_fast_shutdown = 2. This redo log was created with MySQL 5.7.35, and it appears logically non empty. Please follow the instructions at http://dev.mysql.com/doc/refman/8.1/en/upgrading.html
mysql-1  | 2024-05-30T20:33:12.897541Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Generic error.
mysql-1  | 2024-05-30T20:33:13.328843Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
mysql-1  | 2024-05-30T20:33:13.329206Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
mysql-1  | 2024-05-30T20:33:13.329291Z 0 [ERROR] [MY-010119] [Server] Aborting
mysql-1  | 2024-05-30T20:33:13.329810Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.1.0)  MySQL Community Server - GPL.
mysql-1  | 2024-05-30T20:33:13.330769Z 0 [System] [MY-015016] [Server] MySQL Server - end.

Please help.

7

I am also facing same issue on my Ubuntu machine.
tutor, version 12.2.0