Migrate from Eucalyptus to Juniper

I’ve got some old course content from I think as old as Eucalyptus, that I want to migrate to Juniper. - I think I’ve also got some newer ficus and gingko.

Really I want to aggregate all the data into a clean install of Juniper.

Current plan: install; Eucalyptus, load backup, exporting course content;

  • then upgrade deployment & migrate to Ficus;
  • then upgrade deployment & migrate to Ginkgo;
  • then upgrade deployment & migrate to Hawkthorn;
  • then upgrade deployment & migrate to Ironwood;
  • then upgrade deployment & migrate to Juniper, exporting, then deploying a vanilla Juniper and importing

But I’d really prefer to avoid all this heartache and just go straight to Juniper… or at least skip some steps.

Thanks for your suggestions

Are you trying to migrate course content, or are you migrating user and student data as well?

If it’s just course content, exporting courses to OLX and importing them again works fairly well — there are a couple of gotchas, particularly if you are using unusual xBlocks. But it might be relatively quicker to test an export/import flow rather than going through multiple upgrade processes.

1 Like

I need to migrate everything. I have the latest version of one of the courses in that nice export format, but missed one so need to restore from my SQL backups.

Then I’ve got all the user and student data which needs to be migrated all the way up to Juniper.

No fancy xBlocks, just the builtin (including, IIRC, the now-deprecated drag-and-drop one).

Any ideas for a cleaner workflow than I proposed would be appreciated.


I don’t know of a way to skip versions. Database migrations in particular are designed assuming that each successive version will be installed.

Hi @SamuelMarks, glad to hear you’re putting the effort in to upgrade to Juniper! We’ve done migrations like this before, so can provide some advice.

It’s also worth checking the versions of your OS and supporting database software, to see if you need to upgrade any of these. Below are the minimum versions supported.

Release Ubuntu MySQL MongoDB RabbitMQ Elasticsearch
Dogwood 12.04 5.6 2.6.5 0.90
Eucalyptus 12.04 5.6 2.6.5 0.90
Ficus 16.04 5.6 3.0 3.2.1 0.90
Ginkgo 16.04 5.6 3.0 is ok 3.6.9 OC run 3.6.14 (erlang 20.3) 1.5
Hawthorn 16.04 5.6 3.2 3.6.14 (erlang 20.3) 1.5
Ironwood 16.04 5.6 3.2 3.6.14 (erlang 20.3) 1.5
Juniper 16.04 5.6 3.6 3.6.14 (erlang 20.3) 1.5

I have some bad news for you – there’s no skipping steps. Also, some of the releases have minor releases that you have to run database migrations for as well. For instance, there’s 3 steps needed to migrate through Ficus successfully, because of breaking migrations caused by external libraries like django_social.

So for each step, you’ll need to checkout the named version tag, update the dependencies, and then run the LMS + Studio migrations.

You may also encounter issues with the python libraries, since python 2.7 is now obsolete, and so some dependencies may no longer exist. So I recommend creating a new virtualenv for each step, in case you need to roll back and forth.

We’ve used the following migration pathway with success.

  1. Eucalyptus.1
  2. Eucalyptus.3
    Eucalyptus links are to OpenCraft’s branches which have some dependency changes that might be helpful.
  3. Ficus.1
  4. Ficus.3
  5. Ficus.4
  6. Ginkgo.1
    Run drop_djcelery_tables prior to running migrations for ginkgo.1.
  7. Hawthorn.1
  8. Hawthorn.2
  9. Ironwood.1
  10. Ironwood.2
  11. Juniper.2
    No need to go via juniper.1 (small win!).

Best of luck to you on this epic journey :mountain:


Thanks for the feedback and suggestions.

Finally found the time to begin the process, started playing around with the old Vagrant boxes https://edx.readthedocs.io/projects/edx-developer-docs/en/latest/named_releases.html

Looks promising at first, using this helper Vagrantfile https://gist.github.com/johnbaldwin/edb5b21d62cc4db40709dae3211a1f8a

But now getting errors as it’s putting together the virtualenv, as the dependency is no longer available:

    default: Traceback (most recent call last):
    default:   File "/edx/app/edx_ansible/venvs/edx_ansible/bin/ansible-playbook", line 324, in <module>
    default:     sys.exit(main(sys.argv[1:]))
    default:   File "/edx/app/edx_ansible/venvs/edx_ansible/bin/ansible-playbook", line 210, in main
    default:     force_handlers=options.force_handlers,
    default:   File "/edx/app/edx_ansible/venvs/edx_ansible/local/lib/python2.7/site-packages/ansible/playbook/__init__.py", line 181, in __init__
    default:     ansible.callbacks.load_callback_plugins()
    default:   File "/edx/app/edx_ansible/venvs/edx_ansible/local/lib/python2.7/site-packages/ansible/callbacks.py", line 51, in load_callback_plugins
    default:     callback_plugins = [x for x in utils.plugins.callback_loader.all()]
    default:   File "/edx/app/edx_ansible/venvs/edx_ansible/local/lib/python2.7/site-packages/ansible/utils/plugins.py", line 232, in all
    default:     self._module_cache[path] = imp.load_source('.'.join([self.package, name]), path)
    default:   File "/edx/app/edx_ansible/edx_ansible/playbooks/callback_plugins/hipchat_plugin.py", line 12, in <module>
    default:     from ansible.plugins.callback import CallbackBase
    default: ImportError: No module named plugins.callback

I’m hoping this doesn’t prove to be a stopper. At worst, I suppose I can ask people here for a tarball of their virtualenv (I’m very trusting!)

I have some old Eucalyptus backups to restore, and bring up to Juniper.

Really I just want to DB migrate MongoDB and MySQL all the way up from Eucalyptus to Juniper.

Attempts thus far, and their success:

  • Official .box files for Vagrant use: deploy of eucalyptus-devstack-2016-09-01.box failed
  • openfun/openedx-docker: deploy worked, database restore started to show promise, then realised there aren’t that many versions of the images available; and upgrading each within the currently-running container would be quite annoying and error prone
  • Native Ubuntu 16.04 install failed due to this sanity check; removing sanity-check left other errors
  • Native Ubuntu 12.04 install failed due to PGP errors from pgp.mit.edu and id 69464050 not being verifiable (see util/install/ansible-bootstrap.sh)

What should I try next? - I’m thinking FUN’s version on 16.04, because at least it will install…

FYI: Latest attempt derived from openedx-docker and written in Python (Fabric) https://github.com/offscale/offregister-openedx/blob/master/offregister_openedx/ubuntu/fun_bare.py ; currently debugging ImportError: No module named proj.settings (I’m reverse-engineering the Dockerfiles to figure out what the settings files should contain, and whence)

Also tried following:

with devstack from https://edx.readthedocs.io/projects/edx-developer-docs/en/latest/named_releases.html#eucalyptus and Vagrantfile from https://raw.github.com/edx/configuration/open-release/eucalyptus.3/vagrant/release/devstack/Vagrantfile, but was left with a ‘default: ERROR: the playbook: vagrant-devstack.yml could not be found’. Which, IIRC is in configuration so I’ll need to figure out what package layout this expects also.

Didn’t expect this process to be so circuitous…

Looks like the Vagrantfile won’t work, at least not without messing with caches or editing the implementation itself (like FUN does; in their Docker port):

configuration/util/install$ ./install_stack.sh 'devstack' 'open-release/eucalyptus.3'
    default: Collecting git+https://github.com/hmarr/django-debug-toolbar-mongo.git@b0686a76f1ce3532088c4aee6e76b9abe61cc808 (from -r /edx/app/edxapp/edx-platform/requirements/edx/github.txt (line 13))
    default:   Cloning https://github.com/hmarr/django-debug-toolbar-mongo.git (to b0686a76f1ce3532088c4aee6e76b9abe61cc808) to /tmp/pip-Z2PIk5-build
    default:     Complete output from command python setup.py egg_info:
    default:     Download error on https://pypi.python.org/simple/versiontools/: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590) -- Some packages may not be found!
    default:     Couldn't find index page for 'versiontools' (maybe misspelled?)
    default:     Download error on https://pypi.python.org/simple/: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590) -- Some packages may not be found!
    default:     No local packages or download links found for versiontools>=1.6
    default:     Traceback (most recent call last):
    default:       File "<string>", line 20, in <module>
    default:       File "/tmp/pip-Z2PIk5-build/setup.py", line 26, in <module>
    default:         'Topic :: Software Development :: Libraries :: Python Modules',
    default:       File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
    default:         _setup_distribution = dist = klass(attrs)
    default:       File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/setuptools/dist.py", line 268, in __init__
    default:         self.fetch_build_eggs(attrs['setup_requires'])
    default:       File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/setuptools/dist.py", line 313, in fetch_build_eggs
    default:         replace_conflicting=True,
    default:       File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 836, in resolve
    default:         dist = best[req.key] = env.best_match(req, ws, installer)
    default:       File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1081, in best_match
    default:         return self.obtain(req, installer)
    default:       File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1093, in obtain
    default:         return installer(requirement)
    default:       File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/setuptools/dist.py", line 380, in fetch_build_egg
    default:         return cmd.easy_install(req)
    default:       File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 623, in easy_install
    default:         raise DistutilsError(msg)
    default:     distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('versiontools>=1.6')
    default:     ----------------------------------------