I have seen various references to codejail and see it purports to be required for a few things. In the context of a reasonably locked down kubernetes, is there some link/document with an explanation of the security benefits it provides for system security in addition to that you can set up in kubernetes?
There are a couple of paragraphs in the README that should shed some light on the design.
Otherwise, the question you need to ask yourself is: as an operator, how much do I want to isolate the running of completely arbitrary Python code (as in, code that literally any learner can submit from a course page) from the rest of my system, including isolating it from sister processes running similarly arbitrary code (so that one learner’s efforts can’t affect another’s)?
People seem to have been running it successfully with Tutor for a while.
Thanks very much for the reply, and I can certainly understand the use case when running on bare metal. I’m sure there have been many happy users for a while too! But it’s also true that Amazon, Google, Microsoft, Oracle, Alibaba… let you execute completely arbitrary code in any language on their hardware, potentially next to even “state-level” bad actors. It seems to have taken off though…
However, code that literally any learner can submit from a course page - this is an interesting part, and certainly requires further attention. I definitely missed that part in the docs - can you point me to the docs/discussion? If that were open on all instances and there were even a small number of operators not using the codejail, that would presumably be pretty disruptive…
Having done a bit (a while ago now but the principles stay the same) of security work - poorly understood/badly implemented security practices can end up being worse if it lulls you into a false sense of security…
Thanks very much for your help and work on this, we all need to take the security of our learners’ data seriously!
This was developed way-back-when in the edx.org days. Not sure there’s any public discussion or even design documents available. @dave?
Anyway, what I am aware of is documentation on the classic supported case, which doesn’t actually run learner-submitted code, but does run course-author-submitted-code-that-is-potentially-learner-hackable: the Python-evaluated Input Problem. Basically, a course author writes some Python to evaluate and grade arbitrary learner input. As you can imagine, this poses all sorts of security conundrums, so that as an operator, you’re faced with either the prospect of launching one VM or container per grading process (theoretically possible with the external grader), or finding some less overhead-heavy alternative. This is where AppArmor - and thus, codejail - comes in.
You can say that again. 
In the original design, codejail was run on the same host as the LMS, which meant it had to be very locked down. Additionally, running student code was indeed a desirable feature – some instructor Python explicitly executes student-submitted code as part of grading the code.
Moving to Kubernetes just by itself doesn’t bring much benefit, except that Kubernetes has ways to lock down egress networking. You still need AppArmor and rlimit to limit resource consumption, prevent persistence of malicious code, add another layer of egress prevention, etc.
I’d strongly recommend running codejail via https://github.com/openedx/codejail-service which provides a number of additional safeguards, including start-time checks that probe the execution environment. (AppArmor is powerful but can be a bit fragile in the face of changes to the image, so these checks are important.)
You can find a bit more context here: https://github.com/openedx/codejail-service/blob/main/docs/decisions/0001-purpose-of-this-repo.rst
1 Like
Another way to think about it is that yes, AWS lets you run arbitrary code on their servers, but they have to deal with any legal and reputation consequences when a bad actor abuses their service. You probably don’t want to give your Open edX installation that risk exposure. 
Thanks for the info. In 2025 I would definitely try and do Python in the browser rather than execute it on my app server… GitHub - pyscript/pyscript: PyScript is an open source platform for Python in the browser. Try PyScript: https://pyscript.com Examples: https://tinyurl.com/pyscript-examples Community: https://discord.gg/HxvBtukrg2 . It’s not perfect but you can do most of what a teacher might want (like using pandas… https://examples.pyscriptapps.com/pandas/latest/). Pretty sure most of what a course creator might want in terms of calculations could be done that way with well designed data in/outflows.
And I was definitely under the impression that Kubernetes does a LOT more that just lock down egress (cpu, memory limits, very tight control of network, strict user control, strict container lifetimes, zero file persistence, etc.)… Yes, I’m a kube fanboi, sorry! But these things are best discussed over a coffee/beer at a conference :-).
1 Like
Yep, in-browser is certainly a thing you could do, but it would allow the student to see the instructor’s code, which sometimes contains answers.
Kubernetes does allow for locking things down more than I described, but not within a container—it only allows constraining the container as a whole. What you’re describing would only work if you spun up a new k8s job for each codejail execution. I think that might be a cleaner greenfield solution, but I’m unsure how much latency that would add to each execution. (And an execution is required to even display certain xblocks, not just evaluate a submission.)
I do agree it would take some thought, notably “well designed data in/outflows”. My basic point is that the only reason I see for a codejail over a browser-based solution is if you want the injected code to be able to access some raw resources directly on the server, rather than via the normal APIs. And I can’t say I’d want to be doing that - I definitely might want to allow them to hit the APIs though. Some thing like both student’s and instructor’s code gets executed via wasm, the student’s in their browser, hitting an API that would be able to load/execute a defined set of WASMs - the instructor’s python code - the instructor’s code would then call an API to update the marks and reply to the student (or some other realtime update mechanism). I’m almost tempted to give it a try… :-).
There were some quite promising initiatives that looked at running WASM directly on kube (amongst other places) and it was apparently basically free to spin up. Looking at the projects again, they don’t really seem to have taken off unfortunately…
The WasmEdge Runtime provides a well-defined execution sandbox for its contained WebAssembly bytecode program. The runtime offers isolation and protection for operating system resources (e.g., file system, sockets, environment variables, processes) and memory space. The most important use case for WasmEdge is to safely execute user-defined or community-contributed code as plug-ins in a software product (e.g., SaaS, software-defined vehicles, edge nodes, or even blockchain nodes). It enables third-party developers, vendors, suppliers, and community members to extend and customize the software product.
As I see it, there is also the question of client-side versus server-side validation. No matter how well obfuscated/compiled/encrypted your validation code is, if it runs in the client it’ll be easier for the end user to overcome it. This is a perennial problem with multiplayer gaming, for instance. For performance reasons, a lot of validation happens in the client, which results in leaving the door open for determined cheaters. To counteract this, game companies have often resorted to some sort of “proctoring”, which in turn raises privacy concerns.
That said 1: proctoring - including some very invasive examples - is already very much a thing in online learning. I don’t personally like it, and have doubts whether an open source project should include support for it, but it does exist and people do use it.
That said 2: WASM has been brought up in the past as something we could leverage, yes. Not only for grading, but it could definitely help there too. If a WASM grader were built as a pluggable, alternative backend to codejail, I’m sure many operators would take advantage of it!
1 Like
It’s a server-side POC, though. But it doesn’t require AppArmor, so it should be easier to deploy.
1 Like