Lost my caddy dir and my backup

Site Operators Site Operations Help
1

I’m using the standard docker-compose version of OpenEdX and this morning when installing the backup plugin for OpenEdX, which the installation seemed to go fine and I admit that at that point I didn’t check to see if the site was working.

My maintenance window came and went while I worked on an error that was being generated by the backup plugin:

2061, 'RSA Encryption not supported - caching_sha2_password plugin was built with GnuTLS support'.

Once I got past that error and made my first backup, I noticed that the Caddy directory was very small and immediate checked the site and got a 502 error.

I thought “No problem, I still have my /data/caddy backup here I’ll just replace it” but that directory was also empty.

The databases, however, are intact.

Now it comes down to my question:

Assuming we have the media that was deleted, videos, photos, etc-- is there a way we can “reconstitute” everything then re-embed or re-upload our media?

Edit: For example, I have a complete mongodb dump, mysql dump, and… an empty caddy directory.

I’m wondering if the caddy directory was gone before I ever got my first backup.

2

What’s the backup plugin you are using? Is it this one? GitHub - hastexo/tutor-contrib-backup: This is an experimental Tutor plugin. You should not consider it ready for production use at this point.

What commands did you run?

3

Yes— it was that plugin.

I ran:

pip install git+https://github.com/hastexo/tutor-contrib-backup@v2.1.0
tutor plugins enable backup
tutor config save
tutor images build backup
tutor local backup

At which point I got this:

(.venv) ubuntu@ip-172-31-86-54:~/tutor$ tutor local backup
docker compose -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.yml -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.jobs.yml run --rm backup-job sh -e -c 'python backup_services.py'
[+] Creating 4/0
 ✔ Container tutor_local-mysql-1        Running                                                                                                    0.0s
 ✔ Container tutor_local-permissions-1  Created                                                                                                    0.0s
 ✔ Container tutor_local-caddy-1        Running                                                                                                    0.0s
 ✔ Container tutor_local-mongodb-1      Running                                                                                                    0.0s
[+] Running 1/1
 ✔ Container tutor_local-permissions-1  Started                                                                                                    0.3s
2023-10-06 15:48:20,364 INFO Dumping all MySQL databases on mysql:3306 to /data/mysql_dump.sql
2023-10-06 15:48:26,890 INFO Complete. /data/mysql_dump.sql is 6217802 bytes.
2023-10-06 15:48:26,895 INFO Dumping all MongoDB databases on mongodb:27017 to /data/mongodb_dump.
2023-10-06T15:48:27.033+0000    writing admin.system.version to /data/mongodb_dump/admin/system.version.bson
2023-10-06T15:48:27.034+0000    done dumping admin.system.version (1 document)
2023-10-06T15:48:27.037+0000    writing openedx.fs.files to /data/mongodb_dump/openedx/fs.files.bson
2023-10-06T15:48:27.040+0000    writing openedx.fs.chunks to /data/mongodb_dump/openedx/fs.chunks.bson
2023-10-06T15:48:27.040+0000    writing openedx.modulestore.structures to /data/mongodb_dump/openedx/modulestore.structures.bson
2023-10-06T15:48:27.044+0000    writing openedx.modulestore.definitions to /data/mongodb_dump/openedx/modulestore.definitions.bson
2023-10-06T15:48:27.056+0000    done dumping openedx.modulestore.definitions (358 documents)
2023-10-06T15:48:27.057+0000    writing openedx.modulestore.active_versions to /data/mongodb_dump/openedx/modulestore.active_versions.bson
2023-10-06T15:48:27.060+0000    done dumping openedx.modulestore.active_versions (4 documents)
2023-10-06T15:48:27.076+0000    done dumping openedx.fs.files (1182 documents)
2023-10-06T15:48:27.199+0000    done dumping openedx.modulestore.structures (542 documents)
2023-10-06T15:48:27.289+0000    done dumping openedx.fs.chunks (1226 documents)
2023-10-06 15:48:27,298 INFO Complete. /data/mongodb_dump total size 38204620 bytes.
2023-10-06 15:48:27,298 INFO Copying Caddy data to /data/caddy
2023-10-06 15:48:27,332 INFO Complete. /data/caddy total size 132435 bytes.
2023-10-06 15:48:27,332 INFO Creating archive /data/backup/backup.2023-10-06.tar.xz
2023-10-06 15:48:46,411 INFO Complete. /data/backup/backup.2023-10-06.tar.xz is 10897380 bytes.
(.venv) ubuntu@ip-172-31-86-54:~/tutor$ df -h
Filesystem       Size  Used Avail Use% Mounted on
/dev/root         58G   21G   38G  36% /
tmpfs            3.9G     0  3.9G   0% /dev/shm
tmpfs            1.6G  1.8M  1.6G   1% /run
tmpfs            5.0M     0  5.0M   0% /run/lock
/dev/nvme0n1p15  105M  6.1M   99M   6% /boot/efi
tmpfs            784M  4.0K  784M   1% /run/user/1000
(.venv) ubuntu@ip-172-31-86-54:~/tutor$ ls
CHANGELOG.md      Dockerfile   MANIFEST.in  README.rst  changelog.d  my.cnf          requirements  setup.py  test_smtp.sh  tutor           tutor.spec
CONTRIBUTING.rst  LICENSE.txt  Makefile     bin         docs         pyproject.toml  set_smtp.sh   start.sh  tests         tutor.egg-info
(.venv) ubuntu@ip-172-31-86-54:~/tutor$ du -h
12K     ./tutor/templates/config
8.0K    ./tutor/templates/apps/redis

Which is when I knew something was amiss immediately because the size of the caddy directory was suddenly so small.

I did learn my lesson about this plugin.

-Spuds

4

It seems that there is a misunderstanding. Caddy is not used for storing videos, images, etc. Caddy is the web proxy of the Open edX platform. The Caddy data directory stores only certificates information. And that data is probably automatically generated because Caddy generates certificates with certbot.

I do not understand how you came to the conclusion that the supposedly missing Caddy data was causing the 502 errors. That conclusion is probably mistaken. What are the logs coming from the caddy container? tutor local logs --tail=100 caddy

5

Looks like the database connections are failing because I changed the user to connect with native mysql passwords. I can probably just fix that by regenerating the database user.

I recreated – but probably still did it wrong because I get the error:

django.db.utils.OperationalError: (2061, “Authentication plugin ‘caching_sha2_password’ reported error: Authentication requires secure connection.”)

Even after restart-

I used the following to recreate the user:

drop user 'root'@'%';
drop user 'root'@'localhost';
 create user 'root'@'%' identified with caching_sha2_password by 'password';
create user 'root'@'localhost' identified with caching_sha2_password by 'password';
rm /etc/my.cnf  -- (As I had created a custom my.cnf)
tutor local restart

Things seem to be back in order again.

6

I’m glad things seem to be working for you again. You should be aware that it is very likely that your database is going to be failing again next time you run tutor local launch or tutor local do init. That’s because the root user password is going to be re-defined.

7

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.