ADR for removing MongoDB from edx-platform

dave · November 12, 2021, 12:38am

PRs related to this effort:

github.com/edx/edx-platform

Store split modulestore's course indexes in Django/MySQL (Take 2)

edx:master ← open-craft:split-modulestore-mysql-take2

opened 12:21AM - 19 Oct 21 UTC

bradenmacdonald

+691 -127

## Description This is a revised version of #27565 with some minor changes, a…s the prior attempt had to be reverted (#28979). ## What has changed There are two changes since the first version of this PR: 1. The data migration has been removed for now. Instead, this PR will immediately start writing to MySQL+MongoDB but continue to read from MongoDB only. This way no writes will be lost. A future PR will migrate the data for any courses that don't yet appear in MySQL, and a third PR will switch reads to MySQL. [See discussion](https://github.com/edx/edx-platform/pull/28979#issuecomment-942429562) See a876bde5e729e0868f16a34c0c1a63f8bf3a2db9 for this change. 2. The `course_id` column is now case-sensitive, for compatibility with MongoDB. Although the platform code generally [tries to prevent having two courses whose IDs differ only by case](https://github.com/edx/edx-platform/blob/ac8b4f5a6dfdc4ccc6433aeae60d42f6aa341207/common/lib/xmodule/xmodule/modulestore/split_mongo/split.py#L1833-L1838) (note `ignore_case=True`), we found at least one pair of courses on stage that differs only by case in its `org` ID (`edx` vs `edX`). So for backwards compatibility with MongoDB and to avoid issues for anyone else with similar course IDs that differ only by case, we've made the new version case sensitive too. The system still tries to prevent creation of courses that differ only by case (that hasn't changed), but now the MySQL version won't break if that has somehow happened. [See discussion](https://github.com/edx/edx-platform/pull/28979#issuecomment-938001847) See 9e03d0f5179af7ce409cefe14983154feeecda43 for this change. ## Testing instructions Check out this PR, run the migrations, then verify that courses are still listed and accessible in Studio (despite no data migration). To test the case sensitivity issue, run this PR's migration then open an LMS django shell (`./manage.py lms shell`) and run the commands in [this gist](https://gist.github.com/bradenmacdonald/d280ae7124257f06f70dd9ce08178fd8). If you do that on the old version, it will give errors, but on the new version (with the modified migration), it works. A unit test for this has been added which checks the same thing.

github.com/edx/edx-platform

Add a data migration to copy all course index data into MySQL (Take 2)

edx:master ← open-craft:braden/course-indexes-mysql-4-data-migration-take2

opened 03:25AM - 10 Nov 21 UTC

bradenmacdonald

+80 -0

## Description This PR is a repeat of https://github.com/edx/edx-platform/pul…l/29144 which [had to be reverted](https://github.com/edx/edx-platform/pull/29212). This version has similar code but when encountering unexpected data, it will just log it and proceed rather than throwing an error. [See this discussion](https://github.com/edx/edx-platform/pull/29212#issuecomment-959780731). If the error occurs again, the log output will look like this: ``` 2021-11-10 03:19:06,653 ERROR 449 [common.djangoapps.split_modulestore_django.migrations.0002_data_migration] [user None] [ip None] 0002_data_migration.py:36 - Possible data issue found during data migration of course indexes from MongoDB to MySQL: Course course-v1:edX+DemoX+Demo_Course already exists in MySQL but the MongoDB version is newer. That's unexpected because since the course index table was added to MySQL, there has never been a time when we would write course_indexes updates only to MongoDB without also writing to MySQL. Mongo data: edited_on: 2021-11-03 22:23:48.989000+00:00, last_update: 2021-11-03 22:24:14.439000+00:00, published_version: 61830c0dca80abc00d1b233f MySQL data: edited_on: 2021-10-01 11:38:42.219000+00:00, last_update: 2021-10-01 11:38:42.219000+00:00, published_version: bc00d1b233f61830c0dca80a The MySQL version will be overwritten and the MongoDB version used. ``` ## Supporting information See previous PRs. ## Testing instructions 1. Check out a master devstack. Make some changes to a course or two (maybe a library too). 1. At http://localhost:18000/admin/split_modulestore_django/splitmodulestorecourseindex/ verify that any courses that you've modified are already copied into MySQL. Keep this tab open. 1. Check out this PR and run LMS migrations 1. Open http://localhost:18000/admin/split_modulestore_django/splitmodulestorecourseindex/ in a new tab, and compare to the prior version. Courses that were already present in MySQL should be unchanged, and all remaining courses from your devstack should now be listed there. ## Deadline None

I’ll add more PRs to this thread as they come, for folks who are interested in following this. Thank you.

Topic		Replies	Views
Old Mongo Removal Work Deprecation	2	345	September 14, 2022
Atlas MongoDB Performance Issues (un-indexed queries) Site Operators	17	397	April 24, 2024
500 errors accessing an older course in our LMS, modulestore values out of sync? Development mongodb , edx-platform	1	91	March 3, 2025
Deprecation/Removal: DraftModuleStore (Old Mongo Modulestore) Deprecation	5	1005	September 19, 2023
Mongo: how to prune/delete orphaned course data? Site Operators	2	789	April 11, 2020

ADR for removing MongoDB from edx-platform

Related topics