How do you do your Mongo upgrades? Do you use ansible playbooks in the configuration repo? There are some playbooks in particular that seem to me like they might be unmaintained cruft that I’d like to get rid of, but I’m curious is anyone’s using them. In particular mongo_rolling_upgrade.yml and mongo_upgrade_hidden_secondaries.yml. As they reference the Mongo 3.0 playbook, I suspect they haven’t been useful for a long time without local modification.
We’re currently halfway through upgrading edx.org from Mongo 3.2 to 3.6. Right now we’re on 3.4 on the way to 3.6, as Mongo requires upgrading one major release at a time. This means that Juniper will likely involve upgrading Mongo from 3.2 to 3.6 in two steps. I don’t know if there’s any precedence for upgrading Mongo by more than one release at a time, per named release. Is there any automated process from openedx that upgrades mongo for openedx operators or do you usually upgrade Mongo manually or using your own custom process? Our process is to spin up new nodes on the new version using the playbooks, joining them to the cluster, migrating to the new nodes and then shutting down the old nodes.
Thanks @sambapete. For Juniper I might suggest upgrading the apps first and then upgrading Mongo. At the moment we’re not using anything 3.4 or 3.6 specific. The pymongo upgrade happened weeks before the Mongo clusters were upgraded to 3.4. We’ll need to figure out and document the recommended upgrade order.
juniper.alpha1 works well with 3.2.16 on my test server thus far.
To be honest, I was surprised to see your message about 3.4 and 3.6 today because the ansible scripts still install 3.2.16. I understand that in the end this is all about the End of Life support for 3.2 and 3.4 that we have already reached.
I’ll do as suggested when the time comes for the final release of juniper. Thanks.
Not sure if this helps, but we followed Open edX’s update guide, and from dogwood to eucalyptus to ficus to ginkgo to hawthorn mostly what I’ve done is take mongodump of existing server make a new server of required version and put the dump there, for example if I wanted to go from ficus to ginkgo I took a dump from ficus version, install ginkgo in new server and put mongodump there.
That is as far as I remember, but in between if edX’s update page had any commands to update mongo through playbooks, we probably did go that way because any time I needed to update I always go to the version update guide and execute the steps there, though I don’t remember running the playbooks that you’ve mentioned.
Did Hawthorn to Ironwood update the same way once in devstack docker, haven’t tried in production, also haven’t tried Ironwood to juniper as well.
@jdmulloy We actually have 2 different setups for our MongoDBs:
Comercial SaaS MongoDB: we’ve done the upgrade from 3.2 -> 3.4 -> 3.6 using the tools offered by the provider and monitoring the migration. Since we always use replicasets we had almost no downtime during the upgrades.
Self hosted MongoDB: we’re planning the migration, but for the most part we’ll follow the official MongoDB upgrade documentation, which basically goes like this:
Ensure all replicas are using the same mongod version.
Upgrade binaries from replicas.
Wait for sync.
Perform a failover to one of the replicas.
Upgrade old master and perform failover so that the original master takes over the replicaset again.
I’m not sure if we used the playbooks from the configuration repo to do the previous upgrades, but currently we maintain our own deployment playbooks for self-hosted servers.
One extra note for everyone that is not using the master branch of Open edX: you need to cherry-pick the changes from the Ironwood PR linked below before doing these upgrades (otherwise course exports and other MongoDB related features will stop working).
Hi @jdmulloy! I discovered this thread when my devstack started complaining about outdated 3.2 MongoDb data.
As a maintainer of Tutor, I do not use the Ansible playbooks. I would very much appreciate an instructions manual, in the spirit of OEP-45, where instead of an opaque 1-click Ansible playbook you provide upgrade instructions. That is, if they differ from the official instructions (linked above by @giovannicimolin):
Based on your upgrade_mongo_3_6.sh script from the devstack, I do not see any difference with the official instructions. Correct? If there is no difference then I believe Tutor users will upgrade their databases by simply running in succession the 3.4 and 3.6 versions of the Mongodb docker image.
Thanks for your work! It’s always great to see old software being upgraded in Open edX.