@xitij2000 This sounds great! What are the next steps, and what can I do to help?
I would like to mention that there is now a Long Term Support (LTS) version of Tutor: https://overhang.io/tutor/lts/
Among other things, Tutor LTS includes 2 years of support after the end of life of every release.
I realise this is a little off-topic, as Tutor does not depend on the native install. Still, itās a ācommunity-drivenā solution to the problem.
1 Like
I think for breakages to the master or named release branch weād like to have a process in place where we can announce these breakages, perhaps in this forum so that community members can help fix those issues. @itsjeyd can speak more on this.
For reviewing PRs to old release branches, you can ping @jill (pomegranited on GitHub) and @giovannicimolin (giovannicimolin on GitHub) on the PR once a reviewer is assigned, and they will initiate the process of setting up the sandbox, and linking those sandboxes to the PR.
Yes, weād like to establish a process that would make it easy for both edX and the wider community to learn about and fix breakages of master and as well as named release branches.
For master in particular, it would be great if we could get to a point where each individual breakage would be fixed by the author(s) of the changes that introduced the breakage.
For breakages of older release branches (which would be caused by external changes rather than specific PRs in most cases, like you mentioned), our hope is that by announcing those in a way that is easy to follow will enable the community to pull together better and fix them faster and more efficiently.
In terms of how to approach this, we were thinking of creating a thread in the Announcements category on this forum and posting our findings from monitoring periodic builds there. Weād start with a thread for master, and would follow up with additional threads for specific releases once weāre done setting up periodic builds for those as well. Would that work for you?
I ilke the idea of getting automated notifications of breakage. Iām not a fan of posting them as announcements here. Is tracking them in Jira off the table?
@nedbat Tracking breakages in JIRA is not off the table 
What do you think that process should look like in detail?
We have a CRI (Community Reported Issues) project in JIRA now. They could be reported there. Iām not sure what other parts of the process need to be define? (Iām not saying there isnāt more to it, Iām just not sure what detail you are looking for.)
The installations are broken now. Seems like weāve never properly set the version of xqueue to install, but xqueue rarely changes, so we didnāt notice.
But https://github.com/edx/xqueue/commit/352d102a7e3badcb987e59b76d704eb41228d543 changed xqueue, and now installations are failing.
This pull request identifies the root case, but isnāt the right fix for the future. Anyone want to take a stab at the right fix?
@nedbat Reporting breakages via the CRI project sounds good, we can definitely try that and see how it goes 
A few details that would be worth clarifying are:
- How can people get notified of issues being added to this board? And would it be possible to only subscribe to updates for issues that are breaking builds for specific releases and/or master?
- On the board itself, how can we clearly distinguish between issues that are breaking builds and other types of issues?
- How does prioritization for the CRI board work today? Is there anything we could consider doing and/or changing that would help expedite the process of getting individual breakages fixed?
@itsjeyd @nedbat When the breakage comes from a specific PR, in addition to the CRI tickets which will take time to process, it could also speed things up to ping the author on the PR?
To be honest, we do not have a strong process for triaging and fixing issues in CRI today. And I donāt know the ins and outs of Jira well enough to know what tools are available for us to finely slice and notify issues.
Yes, definitely ā this is something we essentially get for free in the scenario where we set up a sandbox for a PR against an older release branch (to allow reviewers to test that PR and to verify that the changes arenāt breaking the provisioning process for the target release, as described by @xitij2000 here).
For the other scenario (where we detect a breakage while reviewing the status of individual instances belonging to our periodic build system) this is going to be a bit trickier. We currently donāt have a way to automatically detect whether weāre dealing with an āexternalā breakage (as defined by @nedbat here) or a breakage coming from changes to an edX repository, nor a way to automatically pinpoint the PR that introduced a specific breakage. So weād be relying on some amount of manual investigation here, at least in the short term.
OK, in that case letās just start small. How about this:
- For the question of how we could clearly identify issues that are breaking builds, using a specific label for the corresponding tickets would probably be sufficient. (This post explains how to make labels show up on the backlog page.)
- Regarding prioritization, just knowing who to ping initially on the CRI tickets would already be helpful. Who would be the right person on your team for this?
Lastly, I donāt have any suggestions on how to make it easier for people to get notified of issues that are breaking builds (other than the original idea of posting on this forum).
1 Like
Just to say: KEEP OLD RELEASES RUNNING 
We need old release to keep installable!