Since RabbitMQ 3.8.16, the configuration consumer_timeout is introduced and defaults to 30 mins.
compute_all_grades_for_course ETA/countdown is set to an hour in code (why is it an hour anyway?)
celery would not ACK the task until countdown is over and task starts processing. RMQ waits for 30 minutes and not gets ACK, so it deems the consumer dead.
Here’s official doc (it happens to : Calling Tasks — Celery 5.3.1 documentation
Here’s the same issue describe on celery: celery/celery#6760
I cannot change the default consumer_timeout as I don’t control the rmq. What’s the easiest way to deal with this in edx code?
EDUCATOR-1088 for this change: Aed/educator 1088 by sanfordstudent · Pull Request #16165 · openedx/edx-platform · GitHub