Hi all!
We are running into some glitches with database migration from Ironwood to Juniper and would appreciate some thoughts on whether it’s something that we’re doing wrong, or perhaps there is something actually broken in the upgrade path.
So, looking at https://github.com/edx/configuration/blob/master/util/install/native.sh, https://github.com/edx/configuration/blob/master/playbooks/openedx_native.yml and the edxapp role, the “migrate”-tagged task lives in https://github.com/edx/configuration/blob/master/playbooks/roles/edxapp/tasks/service_variant_config.yml#L268.
This is the task that we see fail, when we run the playbook after importing our Ironwood database as described in https://edx.readthedocs.io/projects/edx-installing-configuring-and-running/en/latest/platform_releases/juniper.html#upgrading-from-the-ironwood-release.
We’ve tried to duplicate what that task does, by logging into an app server, becoming root, setting the DB_MIGRATION_USER and DB_MIGRATION_PASSWORD environment variables to the appropriate values, and running /edx/bin/edxapp-migrate-lms. This failed with the following error:
Apply all migrations: admin, announcements, api_admin, assessment, auth, block_structure, bookmarks, catalog, celery_utils, completion, consent, content_libraries, content_type_gating, con
tentserver, contentstore, contenttypes, cornerstone, course_action_state, course_creators, course_date_signals, course_duration_limits, course_groups, course_modes, course_overviews, coursew
are, coursewarehistoryextended, crawlers, credit, dark_lang, database_fixups, degreed, discounts, django_comment_common, django_notify, djcelery, edx_proctoring, edx_when, edxval, embargo, e
nterprise, entitlements, experiments, external_user_ids, hastexo, integrated_channel, milestones, oauth2_provider, oauth_dispatch, organizations, redirects, sap_success_factors, schedules, s
elf_paced, sessions, site_configuration, sites, static_replace, student, submissions, survey, system_wide_roles, tagging, theming, track, user_api, user_authn, user_tasks, verified_track_con
tent, verify_student, video_config, video_pipeline, waffle, waffle_utils, wiki, workflow, xapi, xblock_config, xblock_django
Running migrations:
Applying admin.0003_logentry_add_action_flag_choices... OK
Applying announcements.0001_initial... OK
Applying assessment.0004_historicalsharedfileupload_sharedfileupload... OK
Applying assessment.0005_add_filename_to_sharedupload... OK
Applying assessment.0006_TeamWorkflows... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying auth.0010_alter_group_name_max_length... OK
Applying auth.0011_update_proxy_permissions... OK
Applying completion.0003_learning_context... OK
Applying organizations.0007_historicalorganization... OK
Applying content_libraries.0001_initial... OK
Applying content_libraries.0002_group_permissions... OK
Applying content_type_gating.0005_auto_20190306_1547... OK
Applying content_type_gating.0006_auto_20190308_1447... OK
Applying content_type_gating.0007_auto_20190311_1919... OK
Applying content_type_gating.0008_auto_20190313_1634... OK
Applying contentstore.0004_remove_push_notification_configmodel_table... OK
Applying enterprise.0061_systemwideenterpriserole_systemwideenterpriseuserroleassignment... OK
Applying enterprise.0062_add_system_wide_enterprise_roles... OK
Applying enterprise.0063_systemwideenterpriserole_description... OK
Applying enterprise.0064_enterprisefeaturerole_enterprisefeatureuserroleassignment... OK
Applying enterprise.0065_add_enterprise_feature_roles... OK
Applying enterprise.0066_add_system_wide_enterprise_operator_role... OK
Applying enterprise.0067_add_role_based_access_control_switch... OK
Applying cornerstone.0001_initial...Traceback (most recent call last):
File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 71, in execute
return self.cursor.execute(query, args)
File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/MySQLdb/cursors.py", line 209, in execute
res = self._query(query)
File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/MySQLdb/cursors.py", line 315, in _query
db.query(q)
File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/MySQLdb/connections.py", line 239, in query
_mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (1005, 'Can\'t create table `edxapp`.`#sql-4806_3e` (errno: 150 "Foreign key constraint is incorrectly formed")')
After rerunning it a couple more times, it progressively got further, until it eventually completed altogether. Running /edx/bin/edxapp-migrate-cms, the task completed on the first run. Once the manual migration has completed, the “migrate” task in the playbook essentially becomes a no-op, and the complete Ansible playbook runs successfully. So basically, we have a workaround here.
We’ve seen this on two separate environments now, so probably an issue related to the content of one specific database can be ruled out. We’d be curious to see if other people run into similar issues.
Has anyone else run into this?