@xitij2000 This sounds great! What are the next steps, and what can I do to help?
I would like to mention that there is now a Long Term Support (LTS) version of Tutor: https://overhang.io/tutor/lts/
Among other things, Tutor LTS includes 2 years of support after the end of life of every release.
I realise this is a little off-topic, as Tutor does not depend on the native install. Still, it’s a “community-driven” solution to the problem.
I think for breakages to the master or named release branch we’d like to have a process in place where we can announce these breakages, perhaps in this forum so that community members can help fix those issues. @itsjeyd can speak more on this.
For reviewing PRs to old release branches, you can ping @jill (pomegranited on GitHub) and @giovannicimolin (giovannicimolin on GitHub) on the PR once a reviewer is assigned, and they will initiate the process of setting up the sandbox, and linking those sandboxes to the PR.
Yes, we’d like to establish a process that would make it easy for both edX and the wider community to learn about and fix breakages of master and as well as named release branches.
For master in particular, it would be great if we could get to a point where each individual breakage would be fixed by the author(s) of the changes that introduced the breakage.
For breakages of older release branches (which would be caused by external changes rather than specific PRs in most cases, like you mentioned), our hope is that by announcing those in a way that is easy to follow will enable the community to pull together better and fix them faster and more efficiently.
In terms of how to approach this, we were thinking of creating a thread in the Announcements category on this forum and posting our findings from monitoring periodic builds there. We’d start with a thread for master, and would follow up with additional threads for specific releases once we’re done setting up periodic builds for those as well. Would that work for you?
I ilke the idea of getting automated notifications of breakage. I’m not a fan of posting them as announcements here. Is tracking them in Jira off the table?
@nedbat Tracking breakages in JIRA is not off the table What do you think that process should look like in detail?
We have a CRI (Community Reported Issues) project in JIRA now. They could be reported there. I’m not sure what other parts of the process need to be define? (I’m not saying there isn’t more to it, I’m just not sure what detail you are looking for.)
The installations are broken now. Seems like we’ve never properly set the version of xqueue to install, but xqueue rarely changes, so we didn’t notice.
But https://github.com/edx/xqueue/commit/352d102a7e3badcb987e59b76d704eb41228d543 changed xqueue, and now installations are failing.
This pull request identifies the root case, but isn’t the right fix for the future. Anyone want to take a stab at the right fix?
A few details that would be worth clarifying are:
- How can people get notified of issues being added to this board? And would it be possible to only subscribe to updates for issues that are breaking builds for specific releases and/or master?
- On the board itself, how can we clearly distinguish between issues that are breaking builds and other types of issues?
- How does prioritization for the CRI board work today? Is there anything we could consider doing and/or changing that would help expedite the process of getting individual breakages fixed?
To be honest, we do not have a strong process for triaging and fixing issues in CRI today. And I don’t know the ins and outs of Jira well enough to know what tools are available for us to finely slice and notify issues.
Yes, definitely – this is something we essentially get for free in the scenario where we set up a sandbox for a PR against an older release branch (to allow reviewers to test that PR and to verify that the changes aren’t breaking the provisioning process for the target release, as described by @xitij2000 here).
For the other scenario (where we detect a breakage while reviewing the status of individual instances belonging to our periodic build system) this is going to be a bit trickier. We currently don’t have a way to automatically detect whether we’re dealing with an “external” breakage (as defined by @nedbat here) or a breakage coming from changes to an edX repository, nor a way to automatically pinpoint the PR that introduced a specific breakage. So we’d be relying on some amount of manual investigation here, at least in the short term.
OK, in that case let’s just start small. How about this:
- For the question of how we could clearly identify issues that are breaking builds, using a specific label for the corresponding tickets would probably be sufficient. (This post explains how to make labels show up on the backlog page.)
- Regarding prioritization, just knowing who to ping initially on the CRI tickets would already be helpful. Who would be the right person on your team for this?
Lastly, I don’t have any suggestions on how to make it easier for people to get notified of issues that are breaking builds (other than the original idea of posting on this forum).
Just to say: KEEP OLD RELEASES RUNNING
We need old release to keep installable!