Automated database migration from Ironwood to Juniper

Hi all!

We are running into some glitches with database migration from Ironwood to Juniper and would appreciate some thoughts on whether it’s something that we’re doing wrong, or perhaps there is something actually broken in the upgrade path.

So, looking at https://github.com/edx/configuration/blob/master/util/install/native.sh, https://github.com/edx/configuration/blob/master/playbooks/openedx_native.yml and the edxapp role, the “migrate”-tagged task lives in https://github.com/edx/configuration/blob/master/playbooks/roles/edxapp/tasks/service_variant_config.yml#L268.
This is the task that we see fail, when we run the playbook after importing our Ironwood database as described in https://edx.readthedocs.io/projects/edx-installing-configuring-and-running/en/latest/platform_releases/juniper.html#upgrading-from-the-ironwood-release.

We’ve tried to duplicate what that task does, by logging into an app server, becoming root, setting the DB_MIGRATION_USER and DB_MIGRATION_PASSWORD environment variables to the appropriate values, and running /edx/bin/edxapp-migrate-lms. This failed with the following error:

Apply all migrations: admin, announcements, api_admin, assessment, auth, block_structure, bookmarks, catalog, celery_utils, completion, consent, content_libraries, content_type_gating, con
tentserver, contentstore, contenttypes, cornerstone, course_action_state, course_creators, course_date_signals, course_duration_limits, course_groups, course_modes, course_overviews, coursew
are, coursewarehistoryextended, crawlers, credit, dark_lang, database_fixups, degreed, discounts, django_comment_common, django_notify, djcelery, edx_proctoring, edx_when, edxval, embargo, e
nterprise, entitlements, experiments, external_user_ids, hastexo, integrated_channel, milestones, oauth2_provider, oauth_dispatch, organizations, redirects, sap_success_factors, schedules, s
elf_paced, sessions, site_configuration, sites, static_replace, student, submissions, survey, system_wide_roles, tagging, theming, track, user_api, user_authn, user_tasks, verified_track_con
tent, verify_student, video_config, video_pipeline, waffle, waffle_utils, wiki, workflow, xapi, xblock_config, xblock_django
Running migrations:
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying announcements.0001_initial... OK
  Applying assessment.0004_historicalsharedfileupload_sharedfileupload... OK
  Applying assessment.0005_add_filename_to_sharedupload... OK
  Applying assessment.0006_TeamWorkflows... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying auth.0010_alter_group_name_max_length... OK
  Applying auth.0011_update_proxy_permissions... OK
  Applying completion.0003_learning_context... OK
  Applying organizations.0007_historicalorganization... OK
  Applying content_libraries.0001_initial... OK
  Applying content_libraries.0002_group_permissions... OK
  Applying content_type_gating.0005_auto_20190306_1547... OK
  Applying content_type_gating.0006_auto_20190308_1447... OK
  Applying content_type_gating.0007_auto_20190311_1919... OK
  Applying content_type_gating.0008_auto_20190313_1634... OK
  Applying contentstore.0004_remove_push_notification_configmodel_table... OK
  Applying enterprise.0061_systemwideenterpriserole_systemwideenterpriseuserroleassignment... OK
  Applying enterprise.0062_add_system_wide_enterprise_roles... OK
  Applying enterprise.0063_systemwideenterpriserole_description... OK
  Applying enterprise.0064_enterprisefeaturerole_enterprisefeatureuserroleassignment... OK
  Applying enterprise.0065_add_enterprise_feature_roles... OK
  Applying enterprise.0066_add_system_wide_enterprise_operator_role... OK
  Applying enterprise.0067_add_role_based_access_control_switch... OK
  Applying cornerstone.0001_initial...Traceback (most recent call last):
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 71, in execute
    return self.cursor.execute(query, args)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/MySQLdb/cursors.py", line 209, in execute
    res = self._query(query)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/MySQLdb/cursors.py", line 315, in _query
    db.query(q)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/MySQLdb/connections.py", line 239, in query
    _mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (1005, 'Can\'t create table `edxapp`.`#sql-4806_3e` (errno: 150 "Foreign key constraint is incorrectly formed")')

After rerunning it a couple more times, it progressively got further, until it eventually completed altogether. Running /edx/bin/edxapp-migrate-cms, the task completed on the first run. Once the manual migration has completed, the “migrate” task in the playbook essentially becomes a no-op, and the complete Ansible playbook runs successfully. So basically, we have a workaround here.

We’ve seen this on two separate environments now, so probably an issue related to the content of one specific database can be ruled out. We’d be curious to see if other people run into similar issues.

Has anyone else run into this?

Hi @mrtmm! Unfortunately I can’t help you with your problem. But if you do manage to upgrade your server from Ironwood to Juniper with the native installation scripts, I’d love to collect your notes and aggregate them here: https://openedx.atlassian.net/wiki/spaces/COMM/pages/1488977937/Juniper+Upgrade+instructions+WIP

Also, see my related comment on another topic: Draft Juniper release notes

Hi @regis, thanks for the links! Unfortunately, I can’t be of use for upgrading an existing server in-place. We are also going with the recommended approach, making a fresh installation of Juniper on a new machine and we face the above explained issue at database migration.