This evening (on the old continent), we had a meeting with @nedbat and @sambapete to discuss the creation of the “open-release/juniper.master” branch. There has been a lot of testing from the community in the past couple weeks, all blocking issues have been resolved, and we were confident that we would be able to create this branch today.
This branch is a requirement, but it’s not the actual release of Juniper. Technically, Juniper is only released when the “open-release/juniper.1” branch is created. However, the juniper.1 branch is expected to follow up very quickly after the creation of the juniper.master branch.
Unfortunately, we could not create the juniper.master branch because we discovered an important issue in the current Juniper release candidate (RC2). The deployment playbooks run RC2 with an outdated version of MongoDB: they install MongoDB v3.2.16 instead of v3.6.17. This is a serious issue for a couple reasons:
The upgrade from 3.2 and 3.6 is non-trivial and we cannot ask users to upgrade between the moment we release juniper.master and juniper.1.
edX.org already runs v3.6, so Juniper would become outdated on the day it was released if it came with v3.2.
Thus, we decided to postpone the release of the juniper.master branch by at least a week. This is a 4-day weekend in the US, so there won’t be much time to get a PR reviewed to upgrade the deployment playbooks. A juniper.rc3 branch will be created, and we hope that many users will have the opportunity to test this third and (hopefully) final release candidate.
It remains an open question why the MongoDB upgrade (or lack thereof) went unnoticed by almost everyone in the community. I expect that the post-mortem investigation of this issue will give us great insights about the way we can improve the release process. In the meantime, thank you all for your hard work!
I know we actually installed this all the time using mongo 3.6, so all our tests actually happend in that version. I’ll ping someone from our team to please comment here.
Interesting. I also did my testing with Mongodb 3.6, but we realized during Friday’s meeting that:
the juniper release notes did not include any mention of the mongodb upgrade.
the ansible playbooks were installing mongodb 3.2.
Thus we assumed that most people had tested Juniper.rc2 with v3.2. It will be interesting to find out how we came to the conclusion that we had to run 3.6, despite the lack of information in the release notes, and what changes you made to the playbooks to make sure you were running 3.6.
I agree with releasing Juniper on Mongo 3.6. However, we still need to document how operators can do the upgrade. Or, at the very least, give a 3.2 deprecation warning in the release notes and point to some existing upgrade documentation.
@regishttps://docs.mongodb.com provides details about how to do this upgrade, and they recommend doing it in two stages (3.2 → 3.4 → 3.6). OpenCraft did an internal investigation into this process, which is tailored to our clustered setup used by the Ocim hosted service, but it goes through the steps we’ll be taking for our upgrade, as well as the authentication mechanism (SCRAM) update.
