At 2U, I’m preparing for us to upgrade our Codejail sandbox environment from Python 3.8 to something more recent (let’s say 3.14). We want to test existing XBlocks against the new environment, so my overall plan is to repurpose[1] the local/remote codejail darklaunch feature that was originally built for migration from local codejail to a remote codejail-service.
The darklaunch side of things is already pretty solid, but I’d be interested to hear the community’s thoughts on how to operate the server side of things. Fundamentally: Should codejail-service provide first-class support for darklaunching?
Broadly, there are two ways to go about migration:
Option A: Duplicate service
Stand up a second deployment of codejail-service that provides Python 3.14 (and an updated selection of packages). For each XBlock execution, edxapp routes duplicate requests to both deployments.
On the deployer’s side, this involves temporarily setting up a second internal domain name and a new build/deploy pipeline with different parameters for the 3.14 version. These can be torn down again after the migration.
codejail-service itself would not require any supporting changes.
The darklaunch settings would accept two base URLs for two remote codejail-services deployments.
Option B: Edition parameter
Change codejail-service to accept an edition parameter that selects between multiple sandbox environments. For each XBlock execution, edxapp routes two requests to the same deployment, but with differing edition requests.
On the deployer’s side side, this means changing the image to have two sandboxes, probably duplicating a good deal of Dockerfile code. This would be a permanent aspect of the Dockerfile. The apparmor profile would also need to target two sandboxes, although probably less duplication is involved here. Standardized deployment frameworks such as Tutor might reduce some of the deployer burden here.
codejail-service would accept a new edition parameter for exec requests, and route to the correct sandbox. Unknown editions would result in an HTTP 400. If no edition is selected, the first sandbox is used.
The darklaunch settings would still just require one base URL for the remote service, but now have two edition parameters.
My thoughts
I’m currently partial to Option A because it provides maximum flexibility, including the possibility of switching to entirely new implementations of remote codejail, or on new deployment technologies (e.g. switching from EC2 to Kubernetes). Option B introduces a lot of ways for the deployer to make mistakes and involves long-lasting duplication of code. The main advantage would be not having to duplicate an entire service, but I just think that advantage doesn’t stack up to the downsides.
I’m curious to hear what others think, though! Have I missed an option, or pros and cons?
[1] We wouldn’t be able to contribute the repurposed version upstream until the Local Codejail DEPR has completed. I expect that might land in Willow. Having local/remote and remote1/remote2 darklaunch at the same time would probably make the configuration space too complicated.
Route A also makes more sense to me for the deployments that we manage. Its easier to add a second codejail during migration time than fiddle with the dual sandbox in codejail forever.
I agree that option A is the preferable approach. It is a bit of increased operational overhead, but is much simpler from a conceptual standpoint, and as you pointed out it offers a bigger potential upside.
I think for people using Tutor option B doesn’t actually introduce that much friction. Most of the repetition on the Dockerfile and apparmor profile would be handled by jinja templating and globbing. And most of those changes would be performed by the codejail Tutor plugin, but the same applies to option A; The tutor user would probably only need to define the additional Python versions used by the sandbox and their respective requirements and the plugin will configure accordingly.
I wonder if it could go beyond a single Dark Launch and be a canary of sorts with a per-course switch, similar to how the Forum migration from Mongo to MySQL was done.
I think regardless of the approach this is a desirable feature.
I’m generally with Option A here, but it partly depends on whether going from 3.8 to 3.14 breaks a significant amount of content. It’s possible that we’ll need to consider more sophisticated routing between multiple sandboxed environments as a more permanent feature, rather than just a roll out mechanism.
Ah, yes, thank you for bringing that up – at one point I had been wondering whether we should make sandbox editions a user-visible feature. Then instructors would be able to migrate content at their own pace, and maybe even see dark-launch results for their own courses. I was thinking of how an xblock could indicate whether it takes the default edition, early access, legacy, a specific edition, etc.
I think that’s compatible with either option, but would tend to align better with Option A (because we’d be potentially looking at simultaneous support of 3+ editions, not just 2.)
And yeah, 3.14 is definitely going to break some things. I suspect we’re just going to have to try the darklaunch and see how bad the mismatch is before deciding how to address it.
We could definitely use a CourseWaffleFlag for rollout! The current dark launch code doesn’t make use of that because local/remote should have very, very similar behavior, mostly differing in memory/cpu constraints but there’s no reason not to switch to CourseWaffleFlag, I suppose.
I’m in support of that generally, since I do think there are other script backends that make sense. Ultimately, I’d really like to do something with WASM so that the sandbox execution environment becomes data that we can bundle and keep around forever. I think we should be able to add it as an optional attribute to the <script> tag we use to wrap instructor code in ProblemBlock.
Agreed. We might also have options that don’t really use codejail at all, like WASM or CEL.
So is the idea that the LMS would be making requests to both, using the 3.8 sandbox for the real result, and then comparing with 3.14 results in order to error log any potential behavioral differences?
Yeah, WASM is a good example of how we probably don’t want to constrain ourselves to just one backend.
So is the idea that the LMS would be making requests to both, using the 3.8 sandbox for the real result, and then comparing with 3.14 results in order to error log any potential behavioral differences?
Yep. That’s what the existing dark launch code does, just with local vs. remote codejail. Although that code is written with local vs. remote differences kept strongly in mind – the expectation was that we largely shouldn’t see differences, but with different Python versions (and package versions) we should expect differences, and so the dark launch code will likely need to change.
We may even want to use an off-the-shelf dark launch utility to help with that comparison. Haven’t looked into options recently.