Celery tasks stop working after redis files become too large

Site Operators Site Operations Help
1

Hello,

our platform (tutor version 15.1.0, local deployment on a 16 GB Memory / 320 GB Disk, ~7k users with 10k unique enrollments over 10 courses) is periodically running into the issues described here and here. In short, as the platform runs, the reddit files at $(tutor config printroot)/data/redis become large. After a while, they get to about ~10GB at which point celery tasks stop working. I can manually delete the files and restart reddit and the cms-worker images (as described in the topics linked above) and then everything works again. The issue is that at first it took almost a year to get to that point, but as our users grew it now it barely takes a month.

As suggested here, there should be a way to configure redis to “trigger a rewrite at the right time”, however I have not found any documentation on how to do that within the tutor setup.

Does anyone have a working solution? Thank you!

2

I believe this issue was fixed in 17.0.3: tutor/CHANGELOG.md at master · overhangio/tutor · GitHub

Since you are running an older version of Tutor, you could get the benefits by adding a couple of lines to your redis.conf file: fix: infinite cache growth by regisb · Pull Request #1001 · overhangio/tutor · GitHub

3

Hello!

I suppose that upgrading should fix this. I will close it once I manage to upgrade.

Thank you. Best,
Martí

4

I am encountering many issues with the upgrade (notably Mysql initialisation error when upgrading tutor), so I think I will try adding the lines to the redis.conf file.

I understand that directly editing $(tutor config printroot)/env/apps/reddit/reddit.conf is not the way to go since the env folder will be regenerated every time I do tutor config save.

However, I also understand that I cannot use a plugin via openedx-lms-common-settings with the redis-conf template patch unless I upgrade tutor to a version post your linked commit. I have instead tried a custom plugin, i.e.:

from tutor import hooks

hooks.Filters.ENV_PATCHES.add_items(
    [
        (
            "redis",
            """
            echo "maxmemory {{ REDIS_MAX_MEMORY }}" >> $(tutor config printroot)/apps/redis/redis.conf
            echo "maxmemory-policy {{ REDIS_MAX_MEMORY_POLICY }}" >> $(tutor config printroot)/apps/redis/redis.conf
            """,
        ),
    ]
)

However, without luck. Is this the rigth way to go? What am I doing wrong?

Thank you. Best,
Martí

5

otherwise, it just occured to me, may I use tutor >= 17.0.3 with openedX olive? this way I could use the redis config fix without having to upgrade openedX.

6

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.