Problem loading the codejail settings in async tasks

Development
1

Hello community, I have a question.

Have you had a problem loading the codejail settings in async tasks?

For some tasks it is required to run code in edxapp-sandbox through codejail and the settings are read with the middleware, but in edxapp async the middlewares are not run and it uses the default settings and in my case it is very little, this happens because example in the rescore of a loncapa problem.

I got this PR with an idea: wip: Load codejail configuration on-demand instead of on-initializtaion by kdmccormick · Pull Request #98 · openedx/codejail · GitHub
However I am not quite sure how to continue that work or if there is another way or a better approach to solve this problem.

How to replicate the problem:

It took me a while to figure out how to run celery workers on the docker devstack - am I missing some documentation?

First, use make lms-shell then edit /edx/etc/lms.yml . Make sure CELERY_QUEUES is as follows:

CELERY_QUEUES:
- edx.lms.core.default
- edx.lms.core.high
- edx.lms.core.high_mem

(for some reason, on my devstack, these all had wrong values like edx.lms.core.defaultedx.lms.core. , while CELERY_DEFAULT_QUEUE had a correct value, resulting in the worker never seeing the tasks)

Second, add this to lms/envs/private.py :

CELERY_ALWAYS_EAGER = False
BROKER_URL = 'redis://:password@edx.devstack.redis:6379/'

Third, start Redis with make redis-up

Fourth, restart the LMS.

Fifth: from make lms-shell , start a celery worker with

DJANGO_SETTINGS_MODULE=lms.envs.devstack_with_worker celery worker --app=lms.celery:APP

print("#"*30)
print(effective_limits)
  • Do a rescore in Instructor > Student Admin and see that changes in devstack.py or common.py don’t change the codejail settings
2 Likes
2

Hi @mafermazu !

I wrote that PR back when I was trying to figure out why codejail tasks were failing on courses.edx.org. I had a hypothesis that our Celery workers weren’t loading ConfigureCodeJailMiddleware and therefore were running codejail with the default configuration settings (as you are seeing in devstack now), so I wrote that PR to get rid of the middleware strategy so that codejail would read directly from Django settings instead. It turned out, though, that I was wrong:

kdmccormick
Looks like this won’t be necessary. The additional logging in version 3.1.3 shows that the Celery workers load up codejail config just fine. How? I’m not sure.

…I really wish I had left us more detail here :man_facepalming: All I know is that codejail was getting configured fine on our Celery workers. (The real fix ended up being to increase the resource limits for several of our courses).

Based on your debugging, I agree that (unlike my situation) something is in fact wrong with the Celery configuration in Devstack, such the workers are not running ConfigCodeJailMiddleware on startup. I’m not sure why, and I haven’t tried Devstack using async workers myself.

courses.edx.org is deployed using the old Ansible native installation; I’m not sure what’s different about that deployment that is allowing Celery to run correctly.

Where to go from here? I’d recommend one of these paths:

  • See if any other middlewares are being executed when Celery is loaded. That could help you pinpoint whether this is something wrong with codejail’s middleware specifically vs. Devstack’s async edx-platform setup in general.
  • Assuming Devstack is causing the problem and it’s not easy to fix, could you avoid it by testing async codejail in a different production-like environment? We used async codejail heavily on courses.edx.org but did not test that interaction on Devstack. We used Ansible-provisioned sandboxes or a staging environment for that.
  • If you’re willing to make core changes: I honestly do think that the ConfigCodeJailMiddleware system is quite convoluted. The PR that I abandoned attempted to get rid of it in favor of using Django settings directly, which should work with Celery more seamlessly. If you wanted to revive and rebase that change I would not be opposed.
1 Like