Upgrade from Native to Tutor

Site Operators Tutor Help
, ,
1

Good day fellow developers! I am quite in need of assistance in my case. My Open edX is native installed (koa version) and it has hundreds of thousand users with tremendous amount of data. Since the native installation maintained by edX relies on Ansible scripts to deploy Open edX on one or multiple servers, these scripts suffer from a couple of issues that Tutor tries to address.

Therefore I want to upgrade my edx to the latest version of Tutor (currently nutmeg).

How can I upgrade my Open edX from native to tutor and include its data, settings and configurations safely? It would be a huge assistance if I could have a step by step guidance or even where to start so I could know if what I’m implementing is on the right track.

1 Like
2

@Engr_James_Lusuegro, hi!

First off, a minor correction: the latest version of Tutor, v15, actually installs Olive, not Nutmeg.

As for how exactly to upgrade, you might ask @sambapete, as he went (or is going) through a similar procedure recently. I have an idea how to go about it but I’ve never actually done it; it’s better if somebody with practical experience chips in.

3

Thanks @arbrandes I had already seen @Engr_James_Lusuegro post, but I was planning to answer later. Here’s a Reader’s Digest version.

We did migrate from Native Koa to Tutor Nutmeg last weekend after months of preparation and hurdles.

There was a lot of preparation on our part because we had been running a fork of Open edX since Aspen. We had sone data spread on S3 and in some cases on the local filesystem. We had to repatriate all that data locally in orde to keep the data centralized for use after the migration.

We also had to switch our Payment Processor from Paysafe to Cybersource because we didn’t have any ressource to help us develop a Paysafe connector for the Payment MFE.

If you plan to use ecommerce and the Payment MFE, please make sure you can use Cybersource or PayPal. I am not sure if Stripe made it into Nutmeg. You will have to check yourself.

You will need to migrate Native Koa to Tutor Koa first. This is where you need to recover your MySQL and MongoDB data.

Since we also had ecommerce installed, we found out early in our tests that there would be a conflict with the site_id and partner_id used by Tutor that were different than the ones used by Native. It took us a while to figure out where we needed to make changes in the different tables in ecommerce after recovering the database.

In the end, it is mostly a data integrity issue since we want to be able to reconnect previous records with the proper site_id or partner_id after restoring the data. What was site_id = 1 previously under Native is not necessarily the same as site_id = 1 under Tutor before restoring the data. It probably doesn’t affect the operations at a major level, but I would prefer not to find out while the system is in production. Just modifying the values requires a good understanding of the underlying databases and tables.

You will also need to make use to then upgrade to Lilac, then to Maple and then to Nutmeg. There are operations in each step that are necessary for the integrity of the data and to make sure it is properly modified between each release.

Suggested readings:

https://openedx.atlassian.net/wiki/spaces/COMM/pages/3249438723/How+to+migrate+from+a+native+deployment+to+a+tutor+deployment+of+the+Open+edX+platform?focusedCommentId=3337453635#comment-3337453635

You edxapp database dump might be too big to edit. You may need to split it with split, make the modifications required and then concatenate the chunks again before importing it.

https://openedx.atlassian.net/wiki/spaces/COMM/pages/2023915819/Lilac

https://openedx.atlassian.net/wiki/spaces/COMM/pages/2603816700/Maple

https://openedx.atlassian.net/wiki/spaces/COMM/pages/3205201949/Nutmeg

If you have any questions, please do not hesitate to post them here. I may not be able to answer right away but I will let you know if this a problem we encountered and how we managed it or worked around it.

4 Likes
4

This is an awesome write-up, @sambapete! Thanks! I know I’ll be pointing to it often in the future.

5

There is definitely more to it given what we needed to go through because of our infamous problem with the Payment MFE :grin:

There are also tables we needed to remove from our dumps of other databases (ecommerce, notes, xqueue, discovery) other than edxapp because they were not used by Tutor.

Mostly empty tables that crept up from Aspen to Koa and were not used by a virgin Tutor Koa version of ecommerce or discovery. But your mileage may vary depending when you first started installing Open edX. We had a lot of history since some of our data dates backs from Aspen. A lot of backward engineering to figure out a lot of things or decide to recover or not Discovery for example. We ended up not recovering Discovery and letting it recreate the data after the migration.

That’s why it might be better if someone encounters an issue and if I can determine if I faced it and was able to fix it or get around it.

Tutor is a great tool. In my mind, it’s wonderful for anyone installing a Fresh Version of Open edX from scratch. Upgrading from A to K in Native, switching from to K to K, and then upgrading from K to N in Tutor is a completely different story.

There was also a lot of preparation to modify our internal management scripts in order to work with containers instead of the native filesystem and / or MySQL and MongoDB.

We only upgraded when we were convinced it would not impact our users and our operations. Lots of “test” migrations involved in order to reduce our downtime. It took us about 12 hours to dump the data, recover the data, and migrate from Tutor Koa through Tutor Lilac through Tutor Maple and finally through Tutor Nutmeg. Good thing I tried to kept our fork simple over the last 8 years.

I’ll probably say more if my talk proposal for the conference is accepted :wink:

But 10 days later, things are running smoothly. A few glitches we had to fix, but they were not technically related to Tutor.

We also do have Olive.1 installed on one of our test systems with our fork. We might upgrade our Production system to Olive before Palm comes out, but the decision is out of my hands at this point. I am ready, my management is not.

1 Like
6

Honestly, I would have thought it would take much longer! That’s very impressive. I’m looking forward to your talk.

7

That 12 hours is only for the day of the migration.

There were countless hours before that day.

Lots of “preparation” beforehand.

Lots of “digging around” to make sure data stored in multiple places like S3 or the local filesystem would end up in the right directories under data.

Lots of “adaptation” to convert our own tools to manage the platform on day 1 of the migration.

Lots of “dress rehearsal” beforehand.

Lots of “help” from you @regis and the community to resolve some of the issues we had during the planning of that migration.

I am a perfectionist and I wanted to reduce to a minimum the downtime for our users. That 12 hours was “acceptable”. And it was the first downtime in more than 2 years since we had installed Koa Native.

2 Likes
8

Thank you for this response! I will consider performing the said instructions step by step and keep track of my progress starting with what you’ve said here. It seems that not all the references I’ve found online works on my end. But I will try your suggested reading first.

Will mark this as the solution once it works. For the meantime, I will post for questions here.

9

Thank you for your assistance. Currently now, we have migrated our mysql and mongodb data into tutor koa. It turns out to be successful. But now here’s my question.

  1. We have django apps installed in our native installed open edx. Is there a way we can also migrate it into tutor koa? It seems that django-apps has different configurations in tutor.

  2. Can we not just upgrade directly from tutor koa to tutor nutmeg by running upgrade commands? Why should we do it gradually like from koa to lilac, lilac to maple, maple to nutmeg?

  3. If so, can I also have some references on how to upgrade it gradually?

It would be a huge help if at least any of these questions are answered.:slight_smile: Thank you fellow developers! I will be waiting for your reply.

10
  1. We have django apps installed in our native installed open edx. Is there a way we can also migrate it into tutor koa? It seems that django-apps has different configurations in tutor.

We did not have specific owned djangoapps installed in our native fork of Open edX for Koa.3. I can’t help you with that. Let’s hope someone will be able to help you.

  1. Can we not just upgrade directly from tutor koa to tutor nutmeg by running upgrade commands? Why should we do it gradually like from koa to lilac, lilac to maple, maple to nutmeg?

It is how it was suggested to me. It is how we also did it when we were in Native. It’s a step by step process to make sure you do not miss a specific migration between releases. Going directly from Koa to Nutmeg wasn’t recommended.

  1. If so, can I also have some references on how to upgrade it gradually?

It will look something similar to this:

tutor local stop

install the latest release of tutor for lilac following the instructions on Releases · overhangio/tutor · GitHub

tutor config save

tutor local upgrade --from=koa

tutor local quickstart

Repeat for maple and nutmeg.
If you go to olive, it will be tutor local launch instead.

At some point, if you have your own fork, you may need to rebuild images with your own fork.

For example:

tutor images build openedx --build-arg EDX_PLATFORM_REPOSITORY=https://github.com/MyORG/edx-platform.git --build-arg EDX_PLATFORM_VERSION=MyBRANCH --no-cache

tutor local start -d

where you would substitute MyORG with your organization and MyBRANCH with the branch from your fork

You can look at the following for more instructions:

1 Like