How do openedx operators upgrade Mongo? Do you use the upgrade playbooks in configuration?

Hi Everyone,

How do you do your Mongo upgrades? Do you use ansible playbooks in the configuration repo? There are some playbooks in particular that seem to me like they might be unmaintained cruft that I’d like to get rid of, but I’m curious is anyone’s using them. In particular mongo_rolling_upgrade.yml and mongo_upgrade_hidden_secondaries.yml. As they reference the Mongo 3.0 playbook, I suspect they haven’t been useful for a long time without local modification.

We’re currently halfway through upgrading edx.org from Mongo 3.2 to 3.6. Right now we’re on 3.4 on the way to 3.6, as Mongo requires upgrading one major release at a time. This means that Juniper will likely involve upgrading Mongo from 3.2 to 3.6 in two steps. I don’t know if there’s any precedence for upgrading Mongo by more than one release at a time, per named release. Is there any automated process from openedx that upgrades mongo for openedx operators or do you usually upgrade Mongo manually or using your own custom process? Our process is to spin up new nodes on the new version using the playbooks, joining them to the cluster, migrating to the new nodes and then shutting down the old nodes.

1 Like

The last time I had to do a Mongo upgrade was in between Hawthorn and Ironwood when we went from 2.6 to 3.2.

I guess we will upgrade Mongo before migrating to Juniper so that the database will already be at the right version when we install Juniper.

Thanks @sambapete. For Juniper I might suggest upgrading the apps first and then upgrading Mongo. At the moment we’re not using anything 3.4 or 3.6 specific. The pymongo upgrade happened weeks before the Mongo clusters were upgraded to 3.4. We’ll need to figure out and document the recommended upgrade order.

Hi @jdmulloy,

juniper.alpha1 works well with 3.2.16 on my test server thus far.

To be honest, I was surprised to see your message about 3.4 and 3.6 today because the ansible scripts still install 3.2.16. I understand that in the end this is all about the End of Life support for 3.2 and 3.4 that we have already reached.

I’ll do as suggested when the time comes for the final release of juniper. Thanks.

Hi @jdmulloy,

Not sure if this helps, but we followed Open edX’s update guide, and from dogwood to eucalyptus to ficus to ginkgo to hawthorn mostly what I’ve done is take mongodump of existing server make a new server of required version and put the dump there, for example if I wanted to go from ficus to ginkgo I took a dump from ficus version, install ginkgo in new server and put mongodump there.

That is as far as I remember, but in between if edX’s update page had any commands to update mongo through playbooks, we probably did go that way because any time I needed to update I always go to the version update guide and execute the steps there, though I don’t remember running the playbooks that you’ve mentioned.

Did Hawthorn to Ironwood update the same way once in devstack docker, haven’t tried in production, also haven’t tried Ironwood to juniper as well.

@jdmulloy We actually have 2 different setups for our MongoDBs:

  • Comercial SaaS MongoDB: we’ve done the upgrade from 3.2 -> 3.4 -> 3.6 using the tools offered by the provider and monitoring the migration. Since we always use replicasets we had almost no downtime during the upgrades.
  • Self hosted MongoDB: we’re planning the migration, but for the most part we’ll follow the official MongoDB upgrade documentation, which basically goes like this:
    1. Ensure all replicas are using the same mongod version.
    2. Upgrade binaries from replicas.
    3. Wait for sync.
    4. Perform a failover to one of the replicas.
    5. Upgrade old master and perform failover so that the original master takes over the replicaset again.

I’m not sure if we used the playbooks from the configuration repo to do the previous upgrades, but currently we maintain our own deployment playbooks for self-hosted servers.

One extra note for everyone that is not using the master branch of Open edX: you need to cherry-pick the changes from the Ironwood PR linked below before doing these upgrades (otherwise course exports and other MongoDB related features will stop working).

Some references that I’ve used in the preliminary discovery:
https://docs.mongodb.com/manual/release-notes/3.6-upgrade-replica-set/
https://docs.mongodb.com/manual/release-notes/3.4-upgrade-replica-set/
Master PR: https://github.com/edx/edx-platform/pull/21197
Ironwood PR: https://github.com/open-craft/edx-platform/pull/196

1 Like