Celery tasks are broken

Site Operators Site Operations Help
28

Let’s try to remove all data from Redis:

tutor local stop redis
sudo mv $(tutor config printroot)/data/redis $(tutor config printroot)/data/redis.backup
tutor local restart redis cms-worker

After you’ve run these commands, is the course export feature working?

EDIT: I’ve attempted to reproduce the issue with a large Redis dataset, without success. Here’s what I did to fill the Redis cache:

tutor local run cms ./manage.py cms shell -c 'import string, random; from django.core.cache import cache
for _ in range(10**7):
    cache.set("dummy_" + "".join(random.choices(string.ascii_letters, k=256)), "".join(random.choices(string.ascii_letters, k=256)), timeout=3600); print(f"{_/10**5}%")'

I’ve also moved the Redis volume to a slower disk than my SSD.

Then, restart the redis and cms-worker containers:

tutor local restart redis cms-worker

I can see that the cms-worker is having trouble connecting to Redis:

tutor_local-cms-worker-1  | [2022-10-17 13:39:50,213: ERROR/MainProcess] consumer: Cannot connect to redis://redis:6379/0: Redis is loading the dataset in memory.
tutor_local-cms-worker-1  | Trying again in 2.00 seconds... (1/100)
tutor_local-cms-worker-1  | 
tutor_local-cms-worker-1  | [2022-10-17 13:39:52,221: ERROR/MainProcess] consumer: Cannot connect to redis://redis:6379/0: Redis is loading the dataset in memory.
tutor_local-cms-worker-1  | Trying again in 4.00 seconds... (2/100)

Eventually connection is successful:

tutor_local-cms-worker-1  | [2022-10-17 13:39:56,229: INFO/MainProcess] Connected to redis://redis:6379/0
tutor_local-cms-worker-1  | [2022-10-17 13:39:56,232: INFO/MainProcess] mingle: searching for neighbors
tutor_local-cms-worker-1  | [2022-10-17 13:39:57,241: INFO/MainProcess] mingle: all alone

I am then able to successfuly trigger a course export from the Studio. Alternatively, we can trigger an artificial asynchronous task by running:

tutor local run cms ./manage.py cms shell -c 'from cms.djangoapps.contentstore.tasks import update_search_index; update_search_index.delay("course-v1:edX+DemoX+Demo_Course", "2022-10-17T15:00:00.00")'

We can see that the task is correctly picked up by the cms-worker.

At this point I really don’t know what to do to reproduce the error…