About a month before the release, create the “open-release/release-name.master” branch from the master branch. Let’s call this the release-master branch.
Testing and fixes happen on the release-master branch
Once everything is working and stable, we tag the release-master branch with “open-release/release-name.1” to mark the official supported release.
Fixes continue to accumulate on the release-master branch.
Occasionally, at poorly-defined times, we tag the release-master branch again, with “open-release/release-name.2”, “open-release/release-name.3”, and so on.
I’m wondering whether that last step makes sense. We tagged “open-release/juniper.3” after an important security fix, but then immediately added more commits to the release-master branch for more fixes. So almost as soon as .3 existed, it was out of date.
Is there a reason to create the .2, .3, etc, tags? If we don’t create them should we still make the .1 tag? Is there a better way to communicate what’s happening on the release-master branch?
In our particular case I just periodically rebase our fork on top of the release-name.master branch rather than focusing on the periodically tagged point versions. In terms of communicating important changes, perhaps just maintain a thread here in Discourse where notable changes are appended to the thread so that any interested parties can read through that thread to know what the current state is.
1 Like
toxinu
(Geoffrey L. (OpenCraft) - opencraft.com/help)
3
@nedbat Since no feature are supposed to land on juniper.master I don’t think it really makes sense to have these juniper.x tags as long are we are well informed that security or bug fixes are being pushed to juniper.master. So we can update our internal fork and in the meanwhile regularly updating it any way.
I am in favor of dropping the minor branches but having a mailing list or a forum thread dedicated to the announce of important fixes being pushed on {juniper.master} could be really nice.
We’ve relied on the tagged named releases to help old instances through breaking database migrations, see Migrate from Eucalyptus to Juniper, but I believe that’s the only reason.
Interesting. If we hadn’t tagged .2, .3, etc, what would that page look like? The tags weren’t made with regard to migrations, so you would be jumping in larger increments? Without the tags, you wouldn’t have a name for the point to migrate to, other than “tip of Ironwood master”.
Maybe we should make a tag when the release is official, and then one more when the next release is ready? “Ironwood.last” and “Juniper.1” at the same time?
It’s totally possible that some steps can be skipped. I can only report the steps we know we went through successfully as part of our usual release update process.
The ficus.3 and ficus.4 tags were created to help with the django social and djcelery migration issues. So if we have anything similar to manage in future, tags are a nice way to do it. Otherwise, I think it’s fine to just use the master branches.
I don’t think that’s necessary either. Any changes made to ironwood.master since juniper.1 is cut are likely to be things like fixing breaking dependencies, and so the ironwood.last won’t really be very useful.
One of the reason why tags are useful for us is because we have your own fork and we need to rebase our fork on top of the current “release” or “soon to be released” version of Open edX.
For exemple, whenever I start testing a new release, I will rebase our work on top of juniper.rc1 for example. I then test and test and test. Until I do a rebase with juniper.1 before we go into Production.
...
RUN curl https://github.com/edx/edx-platform/commit/<git sha1>.patch | git apply -
...
On the other hand, I agree that tags are inconvenient, as they can quickly become obsolete.
So I’d like to propose a mixed approach. Since we agreed upon a 6-month release frequency, maybe we could also switch to periodical minor tags? It would work like this:
On the day of the release (D+0), create the open-release/xxx.1 tag.
At D+2 months, tag open-release/xxx.2
At D+4 months, tag open-release/xxx.3
Minor tags would thus give us an indication of how close we are to the next major release. Also, if we need to tag open-release/xxx.4, then that’s a clear indication that we are behind on schedule for the next release.
Will we still have a tag or branch in order to do a rebase of our fork before testing anything ? Shall we assume it will always be open-release/xxx.master even before it is released as open-release/xxx.1 ? Rebasing directly on master doesn’t give me a warm feeling. Especially if we don’t know at what point Koa starts.
Time-based tagging is an interesting compromise. Is there a logic to the two-month cadence? Why not one month? (I’m not saying one month would be better, just trying to understand.)
If it helps decide, here are all of the commits to Juniper since the juniper.1 tag:
---- ./ecommerce ----
d032c882 2020-09-01 Ned Batchelder: Upgrade Django to 2.2.16
ca9cddb4 2020-08-05 Ned Batchelder: Upgrade Django to 2.2.15
8aec1df4 2020-06-17 Dillon Dumesnil: DISCO-1643: Making some improvements around supporting Honor (#2992)
---- ./devstack ----
10f02ca 2020-08-17 Zachary Trabookis: Remove `xqueue` as `DEFAULT_SERVICES` for Juniper release.
8ff8dd0 2020-08-17 Zachary Trabookis: Make additional adjustments to the documentation for multiple releases one machine (regisb review).
57455fe 2020-08-10 Zachary Trabookis: Add `xqueue` to default services to provision initially.
3ca4c9d 2020-07-29 Zachary Trabookis: Make sure to pass in `DOCKER_COMPOSE_FILES` to `docker-compose exec` commands for services outside of `docker-compose.yml`.
cef4aa2 2020-07-28 Zachary Trabookis: Updated `README` to include necessary information when running multiple isolated devstacks for same host.
9415683 2020-07-27 Zachary Trabookis: Update `docker` commands to be `docker-compose` to handle unnamed containers for multi-version devstack support
67c7c9b 2020-08-16 morenol: Do not use openedx release for registrar and edx-mktg docker images. (#594)
56312bc 2020-08-04 Guruprasad Lakshmi Narayanan: Remove duplicate section
34a46a3 2020-07-24 Guruprasad Lakshmi Narayanan: Remove the non-release services from the default devstack services
---- ./xqueue ----
5945e7f 2020-09-01 Ned Batchelder: Upgrade Django to 2.2.16
f004caa 2020-08-05 Ned Batchelder: Upgrade Django to 2.2.15
---- ./edx-e2e-tests ----
---- ./edx-platform ----
6a0cdf46fc 2020-09-02 Awais Jibran: Fix video handouts uploads.
c32b3becdd 2020-08-18 SaadYousaf: fix issue with transcript dropdown.
bb7025eb21 2020-09-07 Ali-D-Akbar: Sustaining security fixes 4 & 5
cffd2373b1 2020-09-02 uzairr: fix xss in front-end templates
c635372e0a 2020-09-01 Ned Batchelder: Security patch from aug 26
543cc040aa 2020-09-01 Ned Batchelder: Upgrade Django to 2.2.16
28a3c167d2 2020-09-01 Ned Batchelder: Use juniper-worker for Juniper
234050b199 2020-07-30 Ali-D-Akbar: sustaining xss security fixes 3
63ff8fe07f 2020-02-12 uzairr: PROD-1236: Do not expose user id with certificate URL.
d9e0ca5e70 2020-08-12 Ali-D-Akbar: This commit contains security fixes for the following JIRA tickets: 1. PROD-1603 2. PROD-1605 3. PROD-1612 4. PROD-1619 5. PROD-1289 6. PROD-153 0 7. PROD-1525 8. PROD-1534
c8421f66fc 2020-08-07 uzairr: Fix xss vulnerabilities in templates
47ab6af637 2020-08-06 Attiya Ishaque: [YONK-1759] Version bump of studio-frontend. (#24677)
8dd78619c9 2020-08-05 Ned Batchelder: Upgrade Django to 2.2.15
b295389e96 2020-07-23 Zachary Trabookis: Set `SESSION_COOKIE_SAMESITE=Lax` for `devstack_docker` environment by default to allow login to LMS service.
91af099933 2020-07-23 uzairr: Fix xss in templates
0e45ecb743 2020-07-22 Ali-D-Akbar: Sustaining xss fixes This commit contains xsslint fixes for the following Jira Tickets:
3757f0d11e 2020-07-06 Florian Haas: Fix profile image URLs for image storage on non-public S3 buckets
bf9e8c260b 2020-06-12 stvn: Add Studio warning for deprecated course keys
226fb100c8 2020-06-29 Michael Terry: Downgrade pycontracts to 1.8.12
1ab907b2b0 2020-06-19 Ali-D-Akbar: Fix XSS issues
c611b8600a 2019-10-15 Feanil Patel: Upgrade and constrain things for python 3.
---- ./edx-analytics-pipeline ----
---- ./repo-tools/repo-tools ----
---- ./edx-notes-api ----
2570395 2020-09-01 Ned Batchelder: Upgrade Django to 2.2.16
ad53edd 2020-08-05 Ned Batchelder: Upgrade Django to 2.2.15
---- ./cs_comments_service ----
3079804 2020-08-19 Samuel Walladge: Bump codecov to latest version
---- ./course-discovery ----
6cb7eda3 2020-09-01 Ned Batchelder: Upgrade Django to 2.2.16
e984f273 2020-08-05 Ned Batchelder: Upgrade Django to 2.2.15
a84bca93 2020-05-26 Rebecca Graber: WS-1025 make default search results configurable in admin (#2621)
97f75930 2020-06-11 Dillon Dumesnil: DISCO-1643: Add in Honor Course Types to be created by default
---- ./credentials ----
ff4dd86c 2020-09-01 Ned Batchelder: Upgrade Django to 2.2.16
7a7aab55 2020-08-05 Ned Batchelder: Upgrade Django to 2.2.15
---- ./src/edx-analytics-configuration ----
---- ./src/edx-documentation ----
c42df602 2020-09-15 Gábor Boros: docs(lms): Replace the remaining lms.*.json references
27649e5a 2020-09-15 Gábor Boros: docs(studio): Replace all studio.yaml to studio.yml
cee627df 2020-09-15 Gábor Boros: docs(lms): Replace all lms.yaml to lms.yml
701913ce 2020-07-09 Ned Batchelder: Style: micro-frontend
93cfdb17 2020-09-11 Cory Lee: ISRE-508 Updating documentation to include json to yaml instructions. (#1885)
---- ./src/configuration ----
787b628dd 2020-09-18 Thomas Misilo: Add EDXAPP_MEDIA_URL to cms nginx configuration.
5c4ac73bf 2020-07-23 Florian Haas: Make the special exams toggle configurable
a2ba46c3c 2020-08-31 Cory Lee: Fix for setuptools 50.0.0 breakage.
05bb4edcf 2020-08-24 Feanil Patel: Improve sandboxing. (#5953) (#5960)
860994c0d 2020-08-21 Feanil Patel: Timmc/codejail improvements (#5956)
1ee048001 2020-07-02 Florian Haas: Add "| default" filter on AWS_GATHER_FACTS conditional in edx_service
d81faf673 2020-06-15 Feanil Patel: Update changelog message to be more descriptive.
9ed159446 2020-06-02 Feanil Patel: Make sandbox python version configurable.
27817dead 2019-11-01 Feanil Patel: Update code sandboxes to also run Python3
---- ./src/enterprise-catalog ----
cdeb172 2020-09-01 Ned Batchelder: Upgrade Django to 2.2.16
f886da6 2020-08-05 Ned Batchelder: Upgrade Django to 2.2.15
---- ./src/blockstore ----
---- ./src/edx-analytics-data-api ----
6d70283 2020-09-01 Ned Batchelder: Upgrade Django to 2.2.16
64b4c7f 2020-08-05 Ned Batchelder: Upgrade Django to 2.2.15
---- ./src/frontend-app-publisher ----
---- ./src/edx-app-android ----
---- ./src/notifier ----
---- ./src/edx-analytics-dashboard ----
d370bfad 2020-09-01 Ned Batchelder: Upgrade Django to 2.2.16
b8dfa559 2020-08-05 Ned Batchelder: Upgrade Django to 2.2.15
---- ./src/frontend-app-support-tools ----
---- ./src/edx-app-ios ----
---- ./src/edx-demo-course ----
---- ./src/ecommerce-worker ----
---- ./src/frontend-app-learning ----
---- ./src/edx-certificates ----
---- ./src/frontend-app-profile ----
---- ./src/license-manager ----
de7b75c 2020-09-01 Ned Batchelder: Upgrade Django to 2.2.16
85003a6 2020-08-05 Ned Batchelder: Upgrade to Django 2.2.15
---- ./src/testeng-ci ----
---- ./src/frontend-app-gradebook ----
---- ./src/edx-developer-docs ----
---- ./src/frontend-app-account ----
Before the creation of open-release/xxx.1, we will have to run our tests on the master branch, and we will no longer create alpha tags. This is what we agreed on in the BTR meeting, right?
Yes, me neither, which is why I’d like to keep having tags, and manually apply security patches on top of the tag that I work on.
Well, sort of. A two-month cadence would give us three tags, which would be consistent with what we have right now. Also, I’d be willing to bet that sometimes one month goes by without modification to the release master branch, whereas this does not happen over two month periods.
I’m not saying either that two months is the best cadence… Less frequent tags would be too infrequent to my taste, but I don’t have anything against more frequent tags.
Security fixes would immediately go to the release master branch, which is the one that is installed with the native installation, but they would only get tagged every two months. People working with tags would have to manually pull the security patches.
Because I do have local modifications in our fork, testing against master (or anything that isn’t a fixed point in time if you prefer) does not give me anything worth testing against until there is a release candidate, tag or whatever you call it. So, I might just skip my testing until there is an open-release/xxx.1 tag.
This is back to the same issue we had before open-releases were created, people will be testing against multiple versions of master. And we won’t be able to agree that a fix is needed by someone because someone else tested it with a different version of master.
Since we are getting into the details of tagging, keep in mind that funky git syntax can get you a fixed point in time whenever you want: open-release/juniper.master@{2020-08-01} is Juniper master as it was on August 1st.
(Updated: that syntax won’t work in GitHub urls for fetching code during install, so it won’t work for our purposes…)
Maybe I don’t understand how people use the releases. “Working with tags” seems like a good thing so that you are using a known, named thing. But if you are going to apply security patches manually, then you no longer have a known named thing. Wouldn’t it be better just to install from the master branch when the security fix lands there?
So, let’s assume I want to test Koa, or what will become Koa. I still need to rebase the multiples repositories in my fork. What would I rebase against? I suppose the “current” answer will be master.
Are we all going to test on different fixed point in time of master then?
We should split this discussion: I was originally asking about tagging the master branches after release. You are asking about pre-release testing, and I think they can have different answers.
I understand your point. Yes they can definitely have different answers. In my mind, tag remains a tag. Be it before a release or after a release. I was all encompassing and you were very specific. My bad.