Rather than moving to the next supported version, as a part of this I’m proposing that we simply drop the idea that we only support one version of Ubuntu. I believe the software is agnostic of Ubuntu versions so long as they are sufficiently modern (16.04+) and as such constantly bumping ubuntu versions in our testing is a waste of time and resources. As a corollary to accepting this DEPR we can move all testing to use GitHub’s ubuntu-latest pointer for test workers one last time and then not worry about this specific maintenance in the future.
My main concern is that a lot of people deploy on the most recent Ubuntu LTS rather than ubuntu-latest. If the software ends up depending on a newer-than-LTS feature, that potentially leaves deployers in a bad situation. I think I could pretty readily be convinced that we don’t need overlapping test/support windows the way we do for Python and Django, but I’d still advocate for explicitly bumping versions from one LTS to the next.
So ubuntu-latest is a bit misleading, it is not in-fact the latest version of ubuntu but the latest LTS version. Currently it’s 22.04 but it will likely bump to 24.04 in the next year. GitHub only provides LTS images as runners.
Simply copying your call-out of codejail as an exception from elsewhere. So the rare repo that isn’t agnostic to Ubuntu version can continue to use a test matrix as-needed.
Maybe this discussion belongs elsewhere, but we need to discuss edx-platform testing. The cost of a matrix in edx-platform is high. However, regarding codejail, if it started to break due to an Ubuntu change, I think we’d learn about it too late because it doesn’t get tested often.
@robrap the edx-platform tests do not test codejail in any meaningful way. That is done in the codejail repo tests which is one of the few repos that I’ve called out as having version specific testing still in place.
codejail should test explicitly on ubuntu versions since it needs to validate the apparmor setup for all of them. However, edx-platform doesn’t setup apparmor as a part of its unit testing and so it doesn’t make sense to worry about codejail testing in the edx-platform tests.
edit: While I think I’ve said this outloud a bunch of times and covered it in the codejail ticket, it was not actually in the DEPR ticket. I’ve added it there to make it clear that codejail should test on specific ubuntu vesions.
The problem is that the codejail repo tests will not protect our production environment.
Adding a codejail-related test to edx-platform is one solution for this gap in testing, and would re-open the question of what ubuntu versions we need to use in edx-platform.
@robrap I’m confused about what the goal is here. I think if in the future we added testing to edx-platform that made it a requirement to test on specific versions of Ubuntu, we could change the testing at that point. Maybe even just running those new tests on multiple versions.
But I’m saying that today, the edx-platform tests don’t have any major dependencies on the underlying OS (except for one minor thing noted in this PR which goes away in 2 months) and so they can just be tested on the latest version of ubuntu for now.
The codejail repo currently actively requires the AppArmor subsystem to exist on the underlying OS and be configurable in a specific way. While that is a super important dependency of edx-platform, and could dictate what OS versions edx-platform claims to support, it doesn’t influence how we run the rest of the edx-platform tests which I believe are OS agnostic.
I agree that if edx-platform is not testing anything OS-dependent, then your PR is pretty much unblocked (see below).
Possibly, the simplest OS-dependent AppArmor testing might be as a part of [C]D, and not a part of CI. It might be useful to note this to operators, and maybe add a code comment or doc as appropriate in your edx-platform PR? Note: This is in place of adding AppArmor unit tests and matrices to edx-platform.
If the above seems reasonable, I’m not sure if Tutor would like any sort of test around this. Something to consider.
Can you be more specific here, I’m not sure what you’re imagining. Something like “This PR assumes that you are validating your codejail/apparmor setup independently?” If so I’m not sure why this PR changes anything, this has always been true.
I’m happy to add more docs about this to edx-platform but I think it’s independent of this PR.
Agreed. It has always been true. This discussion and the switch to ubuntu-latest has simply highlighted what has always been true. It seems like a reasonable time to leave a note about this. That said, I agree that this isn’t blocking. Worst case, we leave a note in the PR description if we can’t think of a better place, and maybe open a separate github issue? These are just ideas to keep things moving.
