I’m working on a platform upgrade from Koa to Nutmeg. Both the MySQL and MongoDB data imported to the new platform without any unexpected problems. However, the courses do not appear in Studio nor anywhere in LMS except the unauthenticated landing page; in which case the course cards render perfectly but i get 404 errors if i click on any of them along with a Python error in the lms log, “openedx.core.djangoapps.content.course_overviews.models.CourseOverview.DoesNotExist”. On a slight sidebar, i did this same project a week ago for another client, albeit upgrading from Maple rather than Koa, and everything went smoothly.
Additional details about this Koa to Nutmeg upgrade:
I altered every table in every MySQL database to change the character sets to utf8mb4
Django database migrations ran successfully
the LMS and Studio both seem to work perfectly other than the courses themselves not appearing
the imported Koa courses are version 11, whereas new courses created from Studio in Nutmeg are version 17 (see screen shot). However, i don’t know what “version” means nor how to upgrade a “version 11” course to version 17.
@lpm0073 My current Production system which is still running Koa Native has most courses (except 3 which are still at version 6) at version 11.
My current Test system which is based on the same Production system but running Nutmeg Tutor has most courses at version 17 (except 2 that stayed at version 6).
I did do a migration from Koa Native to Koa Tutor to Lilac Tutor to Maple Tutor and then to Nutmeg Tutor without skipping a step and looking at potential Feature Changes for each release as mentioned in the previous post.
thank you @sambapete, this clearly is part of the solution, however, i still haven’t figured out the correct sequence, or if perhaps there are additional steps. what i’ve done so far:
on the Koa platform, i ran ./manage.py cms simulate_publish --settings=production. Just a note for anyone reading this post, that this operation did find and process every course. Afterwards however, all of the courses are still version 11.
i repeated the MongDB migration to the Nutmeg platform
on the Nutmeg platform i ran ./manage.py cms backfill_course_tabs --settings=tutor.production. However, this has absolutely no affect whatsoever. Nutmeg does not see any courses and thus, it does nothing. this command is benign.
There seems to be some other interim step. Any ideas? Thanks again for your help!!
I ran the sumulate_publish command right after migrating from Lilac to Maple.
Remember that I did go from Native Koa to Tutor Koa then Tutor Lilac then Tutor Maple and then Tutor Nutmeg in that order, not skipping any versions. I did migrate MySQL and Mongo directly from Koa Native to Koa Tutor.
Could this be the difference? Are you going straight from Koa Native to Nutmeg Tutor?
Seems like that might be the case. I’ll look for documentation on going from Koa native to Koa Tutor, to investigate whether something is happening in this step that bumps the course versions.
I remember seeing somewhere that it was recommended to first migrate from Native to Tutor on the same release. Maybe it was during a discussion I had with @regis or @BbrSofiane.
what documentation did you use for this step, from Koa native to Koa tutor?
Well, actually – just thinking out loud – i guess i could just build a generic installation of Koa on tutor and then import my course content. The MySQL data for this particular project appears to already be working as expected on Nutmeg and so i guess i could mostly ignore this at this stage.
i’m grateful that @BbrSofiane took the time to publish his notes to the Wiki. But at the same time i have to ask out loud: why are these instructions buried 5 levels deep in an atlassian wiki, and apparently beyond the field of vision of google’s searchbots?
Always bear in mind that I upgraded Tutor from Koa and then to Lilac to Maple to Nutmeg.
You will need to install the proper Tutor version for Lilac.
And then run these commands,
tutor local upgrade --from=koa
tutor local quickstart
You will have to do something similar when moving to Maple and Nutmeg. You could be missing some of the steps taken by the tutor local upgrade commands.
@sambapete , confirming that this solved my problem. Thank you!!!
Note however that the actual technical steps are convoluted. Additionally, I’m unclear about the necessity of running this procedure version-by-version. The earliest usable tutor version that i’ve found is 12.2.0 which installs Lilac.3.
For the benefit of others, these are the steps that I took to re-produce the steps that @sambapete describes above:
ANTECEDENTS.
#
# Note that in my case I had to run this procedure a half dozen times
# until it actually worked. If you're as unfortunate as me then you'll
# potentially need to get your ubuntu instance back to a pristine state
# depending on what went wrong on your most recent failed attempt.
#
# This is how to "reset" your Ubuntu environment to pristine in the
# event that you run this procedure more than once.
#
# - shut down and remove any running Docker containers
# - delete any existing Docker volumes
# - if tutor is currently installed then uninstall it and all of its modules
# -------------------------------
docker stop $(docker ps -a -q)
docker rm $(docker ps -a -q)
docker rmi $(docker images -a -q)
docker volume prune
sudo rm -rf "$(tutor config printroot)"
pip uninstall tutor-openedx
sudo rm "$(which tutor)"
pip uninstall tutor -y
pip uninstall tutor-xqueue -y
pip uninstall tutor-webui -y
pip uninstall tutor-richie -y
pip uninstall tutor-notes -y
pip uninstall tutor-minio -y
pip uninstall tutor-mfe -y
pip uninstall tutor-license -y
pip uninstall tutor-forum -y
pip uninstall tutor-ecommerce -y
pip uninstall tutor-discovery -y
pip uninstall tutor-android -y
# I. Setup
# -------------------------------
pip install "tutor[full]"
tutor local quickstart
# II. modify the db name inside your Koa mysql dump file
# https://openedx.atlassian.net/wiki/spaces/COMM/pages/3249438723/How+to+migrate+from+a+native+deployment+to+a+tutor+deployment+of+the+Open+edX+platform
# -------------------------------
# III. Import your Koa MySQL data
# -------------------------------
MYSQL_ROOT_PASSWORD=$(tutor config printvalue MYSQL_ROOT_PASSWORD)
MYSQL_ROOT_USERNAME=root
docker exec -i tutor_local_mysql_1 sh -c 'exec mysql -uroot -p$MYSQL_ROOT_PASSWORD -e "DROP DATABASE openedx;"'
docker exec -i tutor_local_mysql_1 sh -c 'exec mysql -uroot -p$MYSQL_ROOT_PASSWORD' < /home/ubuntu/backups/mysql/mysql-data-koa.sql # EDIT THIS
# IV. take care of any Tutor-specific conflicts that might exist in your MySQL data
# in my case there was a username name conflict with a service account user that tutor creates
# -------------------------------
# V. Import your Koa MongoDB
# -------------------------------
sudo mkdir /home/ubuntu/.local/share/tutor/data/mongodb/backup
sudo mv /home/ubuntu/backups/mongodb/mongo-dump-koa/edxapp /home/ubuntu/.local/share/tutor/data/mongodb/backup/. # EDIT THIS
sudo chown -R systemd-coredump /home/ubuntu/.local/share/tutor/data/mongodb/backup
sudo chgrp -R systemd-coredump /home/ubuntu/.local/share/tutor/data/mongodb/backup
tutor local exec mongodb bash
mongorestore -d openedx /data/db/backup/edxapp/
sudo rm -r /home/ubuntu/.local/share/tutor/data/mongodb/backup
# VI. Upgrades
# NOTE the following in my case:
# - the upgrade --from=koa step only partially worked. The Django db migrations worked, but the MongoDB upgrade broke.
# - "--from=lilac" seems to have picked up where the other left off.
# - "--from=maple" was benign. it didn't perform any additional operations.
# -------------------------------
tutor local upgrade --from=koa
tutor local quickstart
tutor local upgrade --from=lilac
tutor local quickstart
tutor local upgrade --from=maple
tutor local quickstart
# VII. Verify your upgrade: connect to MySQL
# -------------------------------
MYSQL_ROOT_PASSWORD=$(tutor config printvalue MYSQL_ROOT_PASSWORD)
MYSQL_ROOT_USERNAME=root
docker exec -it tutor_local_mysql_1 bash -l
MYSQL_ROOT_PASSWORD=ADD-YOUR-PASSWORD-HERE
MYSQL_ROOT_USERNAME=root
mysql -u$MYSQL_ROOT_USERNAME -p$MYSQL_ROOT_PASSWORD
use openedx;
SELECT id, version FROM openedx.course_overviews_courseoverview;
# VIII. Dump the Nutmeg Mysql data and save to AWS S3
# https://docs.tutor.overhang.io/tutorials/datamigration.html
# -------------------------------
tutor local exec \
-e USERNAME="$(tutor config printvalue MYSQL_ROOT_USERNAME)" \
-e PASSWORD="$(tutor config printvalue MYSQL_ROOT_PASSWORD)" \
mysql sh -c 'mysqldump --databases openedx --user=$USERNAME --password=$PASSWORD > /var/lib/mysql/dump.sql'
sudo mv /home/ubuntu/.local/share/tutor/data/mysql/dump.sql ~/backups/mysql/
sudo chown -R ubuntu ~/backups/mysql/dump.sql
sudo chgrp -R ubuntu ~/backups/mysql/dump.sql
tar -czf ~/backups/mysql/dump.tgz ~/backups/mysql/dump.sql
aws s3 cp ~/backups/mysql/dump.sql s3://academiacentral-global-staging-backup/
# IX. Dump the Nutmeg MongoDB data and save to AWS S3
# -------------------------------
tutor local exec mongodb mongodump --out=/data/db/dump.mongodb
sudo mv /home/ubuntu/.local/share/tutor/data/mongodb/dump.mongodb ~/backups/mongodb/
sudo chown -R ubuntu ~/backups/mongodb/dump.mongodb
sudo chgrp -R ubuntu ~/backups/mongodb/dump.mongodb
aws s3 cp ~/backups/mongodb/ s3://academiacentral-global-staging-backup/ --recursive
thanks @misilot . I’ll work on pros for a more generalized upgrade. Your link above for example includes additional steps that are important (and unintuitive) for performing a complete upgrade. It’ll be good to publish a comprehensive future-proof guide with the complete procedure.
In all fairness @lpm0073, and to give him credits where credits are due, the instructions I gave you from @BbrSofiane were originally taken from @misilot post.
@lpm0073 if you are able to take the initiative to do the documentation / writing up, I’m sure I can help you find an appropriate (top-level and searchable!) place for these instructions.
