Currently running on Maple in production.
After performing several course updates and re-runs we’re receiving the following error from ElastiCache Redis. Any advice on how to mitigate this max memory from occuring? This also happened when we updated several courses CMS > Course > Settings > Advanced Settings values all at once.
CMS Import Indicates the following message.
cms-worker error logs
tutor_local-cms-worker-1 | [2023-07-20 13:44:57,757: CRITICAL/MainProcess] Unrecoverable error: ResponseError("Command # 1 (ZADD unacked_index 1689860697.7538862 65aa0263-7560-458c-8121-fb7a9fcdfafc) of pipeline caused error: OOM command not allowed when used memory > 'maxmemory'.")
tutor_local-cms-worker-1 | Traceback (most recent call last):
tutor_local-cms-worker-1 | File "/openedx/venv/lib/python3.8/site-packages/redis/client.py", line 3911, in _execute_transaction
tutor_local-cms-worker-1 | response = self.parse_response(connection, '_')
tutor_local-cms-worker-1 | File "/openedx/venv/lib/python3.8/site-packages/redis/client.py", line 3977, in parse_response
tutor_local-cms-worker-1 | result = Redis.parse_response(
tutor_local-cms-worker-1 | File "/openedx/venv/lib/python3.8/site-packages/redis/client.py", line 915, in parse_response
tutor_local-cms-worker-1 | response = connection.read_response()
tutor_local-cms-worker-1 | File "/openedx/venv/lib/python3.8/site-packages/redis/connection.py", line 756, in read_response
tutor_local-cms-worker-1 | raise response
tutor_local-cms-worker-1 | redis.exceptions.ExecAbortError: Transaction discarded because of previous errors.
tutor_local-cms-worker-1 |
tutor_local-cms-worker-1 | During handling of the above exception, another exception occurred:
tutor_local-cms-worker-1 |
tutor_local-cms-worker-1 | Traceback (most recent call last):
tutor_local-cms-worker-1 | File "/openedx/venv/lib/python3.8/site-packages/celery/worker/worker.py", line 208, in start
tutor_local-cms-worker-1 | self.blueprint.start(self)
tutor_local-cms-worker-1 | File "/openedx/venv/lib/python3.8/site-packages/celery/bootsteps.py", line 119, in start
tutor_local-cms-worker-1 | step.start(parent)
tutor_local-cms-worker-1 | File "/openedx/venv/lib/python3.8/site-packages/celery/bootsteps.py", line 369, in start
tutor_local-cms-worker-1 | return self.obj.start()
tutor_local-cms-worker-1 | File "/openedx/venv/lib/python3.8/site-packages/celery/worker/consumer/consumer.py", line 318, in start
tutor_local-cms-worker-1 | blueprint.start(self)
tutor_local-cms-worker-1 | File "/openedx/venv/lib/python3.8/site-packages/celery/bootsteps.py", line 119, in start
tutor_local-cms-worker-1 | step.start(parent)
tutor_local-cms-worker-1 | File "/openedx/venv/lib/python3.8/site-packages/celery/worker/consumer/consumer.py", line 599, in start
tutor_local-cms-worker-1 | c.loop(*c.loop_args())
tutor_local-cms-worker-1 | File "/openedx/venv/lib/python3.8/site-packages/celery/worker/loops.py", line 83, in asynloop
tutor_local-cms-worker-1 | next(loop)
tutor_local-cms-worker-1 | File "/openedx/venv/lib/python3.8/site-packages/kombu/asynchronous/hub.py", line 364, in create_loop
tutor_local-cms-worker-1 | cb(*cbargs)
tutor_local-cms-worker-1 | File "/openedx/venv/lib/python3.8/site-packages/kombu/transport/redis.py", line 1083, in on_readable
tutor_local-cms-worker-1 | self.cycle.on_readable(fileno)
tutor_local-cms-worker-1 | File "/openedx/venv/lib/python3.8/site-packages/kombu/transport/redis.py", line 354, in on_readable
tutor_local-cms-worker-1 | chan.handlers[type]()
tutor_local-cms-worker-1 | File "/openedx/venv/lib/python3.8/site-packages/kombu/transport/redis.py", line 744, in _brpop_read
tutor_local-cms-worker-1 | self.connection._deliver(loads(bytes_to_str(item)), dest)
tutor_local-cms-worker-1 | File "/openedx/venv/lib/python3.8/site-packages/kombu/transport/virtual/base.py", line 983, in _deliver
tutor_local-cms-worker-1 | callback(message)
tutor_local-cms-worker-1 | File "/openedx/venv/lib/python3.8/site-packages/kombu/transport/virtual/base.py", line 632, in _callback
tutor_local-cms-worker-1 | self.qos.append(message, message.delivery_tag)
tutor_local-cms-worker-1 | File "/openedx/venv/lib/python3.8/site-packages/kombu/transport/redis.py", line 155, in append
tutor_local-cms-worker-1 | pipe.zadd(self.unacked_index_key, *zadd_args) \
tutor_local-cms-worker-1 | File "/openedx/venv/lib/python3.8/site-packages/redis/client.py", line 4019, in execute
tutor_local-cms-worker-1 | return execute(conn, stack, raise_on_error)
tutor_local-cms-worker-1 | File "/openedx/venv/lib/python3.8/site-packages/redis/client.py", line 3914, in _execute_transaction
tutor_local-cms-worker-1 | raise errors[0][1]
tutor_local-cms-worker-1 | File "/openedx/venv/lib/python3.8/site-packages/redis/client.py", line 3904, in _execute_transaction
tutor_local-cms-worker-1 | self.parse_response(connection, '_')
tutor_local-cms-worker-1 | File "/openedx/venv/lib/python3.8/site-packages/redis/client.py", line 3977, in parse_response
tutor_local-cms-worker-1 | result = Redis.parse_response(
tutor_local-cms-worker-1 | File "/openedx/venv/lib/python3.8/site-packages/redis/client.py", line 915, in parse_response
tutor_local-cms-worker-1 | response = connection.read_response()
tutor_local-cms-worker-1 | File "/openedx/venv/lib/python3.8/site-packages/redis/connection.py", line 756, in read_response
tutor_local-cms-worker-1 | raise response
tutor_local-cms-worker-1 | redis.exceptions.ResponseError: Command # 1 (ZADD unacked_index 1689860697.7538862 65aa0263-7560-458c-8121-fb7a9fcdfafc) of pipeline caused error: OOM command not allowed when used memory > 'maxmemory'.