CPU usage high for `cms-worker` - elasticsearch error - maxClauseCount is set to 1024

Zachary_Trabookis · March 4, 2025, 7:36pm

We’re getting hit hard with CPU usage with this process. Is there a way to limit the number of cms.celery processes?

I see a lot of these processes running and we have about 4 content uploaders publishing content.

/openedx/venv/bin/python /openedx/venv/bin/celery worker --loglevel=info --hostname=edx.cms.core.default.%%h --maxtasksperchild 100 --exclude-queues=edx.lms.core.default

Confirming with the tutor service configuring to see if there is a setting that we can use to limit the number of these processes running.

Would the --maxtasksperchild 100 need to be altered? Any other settings to scale these cms.celery processes from consuming all the CPU usage?

We currently don’t have horizontal scaling implemented an all services (e.g. cms, cms-worker, lms, lms-worker) reside on one machine.

cc @dave

Zachary_Trabookis · March 4, 2025, 7:47pm

We’re using this version of celery. If you can recommend anything to limit the number of processes that would be helpful or reduce the CPU usage.
/openedx/venv/bin/celery is version 4.4.7

–max-tasks-per-child
I don’t think changing this setting --max-tasks-per-child will help out with our CPU usage.

–autoscale
Maybe this setting will help us out.

braden · March 4, 2025, 7:48pm

I’m really curious what task(s) these workers are running?

Zachary_Trabookis · March 4, 2025, 8:00pm

@braden
How would I go about finding that out?

Zachary_Trabookis · March 4, 2025, 9:15pm

@braden
Here are some of the current items in the celery queue.

I was able to connect using this command.

app@a156df8b1cb0:~/edx-platform$ /openedx/venv/bin/python /openedx/venv/bin/celery --broker redis://@[elasicache-redis-endpoint]:6379 --app=cms.celery inspect active

It appears there are a lot of calls to cms.djangoapps.contentstore.tasks.update_search_index.

-> celery@edx.lms.core.default.%88ea897c7545: OK
    - empty -
-> celery@edx.cms.core.default.%a156df8b1cb0: OK
    * {'id': '45b111cb-94d5-47aa-938e-ae56ad41fbbc', 'name': 'cms.djangoapps.contentstore.tasks.update_search_index', 'args': ['course-v1:REVVED+EV-MT-QLT+DEVELOPMENT', '2025-03-04T20:47:46.779854+00:00'], 'kwargs': {}, 'type': 'cms.djangoapps.contentstore.tasks.update_search_index', 'hostname': 'celery@edx.cms.core.default.%a156df8b1cb0', 'time_start': 1741122444.1033635, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'edx.cms.core.default', 'priority': 0, 'redelivered': None}, 'worker_pid': 14112}
    * {'id': '657479ea-f7a2-49ac-a8eb-51b935928fca', 'name': 'cms.djangoapps.contentstore.tasks.update_search_index', 'args': ['course-v1:REVVED+EV-MT-QLT+DEVELOPMENT', '2025-03-04T20:51:16.609744+00:00'], 'kwargs': {}, 'type': 'cms.djangoapps.contentstore.tasks.update_search_index', 'hostname': 'celery@edx.cms.core.default.%a156df8b1cb0', 'time_start': 1741122518.6042817, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'edx.cms.core.default', 'priority': 0, 'redelivered': None}, 'worker_pid': 11719}
    * {'id': '230d7ff8-08d1-4b97-8add-b49da40d44da', 'name': 'cms.djangoapps.contentstore.tasks.update_search_index', 'args': ['course-v1:REVVED+EV-MT-QLT+DEVELOPMENT', '2025-03-04T20:50:13.223560+00:00'], 'kwargs': {}, 'type': 'cms.djangoapps.contentstore.tasks.update_search_index', 'hostname': 'celery@edx.cms.core.default.%a156df8b1cb0', 'time_start': 1741122459.9061623, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'edx.cms.core.default', 'priority': 0, 'redelivered': None}, 'worker_pid': 13280}
    * {'id': 'dc56dd50-3087-4947-8073-043dd1e86d59', 'name': 'cms.djangoapps.contentstore.tasks.update_search_index', 'args': ['course-v1:REVVED+EV-MT-QLT+DEVELOPMENT', '2025-03-04T20:50:22.285569+00:00'], 'kwargs': {}, 'type': 'cms.djangoapps.contentstore.tasks.update_search_index', 'hostname': 'celery@edx.cms.core.default.%a156df8b1cb0', 'time_start': 1741122474.121779, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'edx.cms.core.default', 'priority': 0, 'redelivered': None}, 'worker_pid': 10066}
    * {'id': '2575aa02-a55d-4080-affe-7d0f24155980', 'name': 'openedx.core.djangoapps.bookmarks.tasks.update_xblocks_cache', 'args': ['course-v1:REVVED+EV-MT-QLT+DEVELOPMENT'], 'kwargs': {}, 'type': 'openedx.core.djangoapps.bookmarks.tasks.update_xblocks_cache', 'hostname': 'celery@edx.cms.core.default.%a156df8b1cb0', 'time_start': 1741122743.8964186, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'edx.cms.core.default', 'priority': 0, 'redelivered': None}, 'worker_pid': 14978}
    * {'id': '3501b7e7-6df5-4227-afbf-87820de5d1b3', 'name': 'cms.djangoapps.contentstore.tasks.update_search_index', 'args': ['course-v1:REVVED+EV-MT-QLT+DEVELOPMENT', '2025-03-04T20:53:21.722445+00:00'], 'kwargs': {}, 'type': 'cms.djangoapps.contentstore.tasks.update_search_index', 'hostname': 'celery@edx.cms.core.default.%a156df8b1cb0', 'time_start': 1741122741.9840996, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'edx.cms.core.default', 'priority': 0, 'redelivered': None}, 'worker_pid': 17439}
    * {'id': '7deba14f-9b43-44ff-8534-0ca971c70579', 'name': 'cms.djangoapps.contentstore.tasks.update_search_index', 'args': ['course-v1:REVVED+EV-MT-QLT+DEVELOPMENT', '2025-03-04T20:51:05.570712+00:00'], 'kwargs': {}, 'type': 'cms.djangoapps.contentstore.tasks.update_search_index', 'hostname': 'celery@edx.cms.core.default.%a156df8b1cb0', 'time_start': 1741122509.0563273, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'edx.cms.core.default', 'priority': 0, 'redelivered': None}, 'worker_pid': 12613}
    * {'id': '094df18b-f468-45bc-aa20-8819f60e39e0', 'name': 'cms.djangoapps.contentstore.tasks.update_search_index', 'args': ['course-v1:REVVED+EV-MT-QLT+DEVELOPMENT', '2025-03-04T20:47:49.028729+00:00'], 'kwargs': {}, 'type': 'cms.djangoapps.contentstore.tasks.update_search_index', 'hostname': 'celery@edx.cms.core.default.%a156df8b1cb0', 'time_start': 1741122449.6721253, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'edx.cms.core.default', 'priority': 0, 'redelivered': None}, 'worker_pid': 14964}
    * {'id': '66dc22c3-c100-40b6-9dd7-21cc236a3b3d', 'name': 'cms.djangoapps.contentstore.tasks.update_search_index', 'args': ['course-v1:REVVED+EV-MT-QLT+DEVELOPMENT', '2025-03-04T20:50:18.188201+00:00'], 'kwargs': {}, 'type': 'cms.djangoapps.contentstore.tasks.update_search_index', 'hostname': 'celery@edx.cms.core.default.%a156df8b1cb0', 'time_start': 1741122469.5452387, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'edx.cms.core.default', 'priority': 0, 'redelivered': None}, 'worker_pid': 14950}
    * {'id': '388b32da-c69a-4c98-8c6b-ec2eaa013fc0', 'name': 'cms.djangoapps.contentstore.tasks.update_search_index', 'args': ['course-v1:REVVED+EV-MT-QLT+DEVELOPMENT', '2025-03-04T20:47:34.567905+00:00'], 'kwargs': {}, 'type': 'cms.djangoapps.contentstore.tasks.update_search_index', 'hostname': 'celery@edx.cms.core.default.%a156df8b1cb0', 'time_start': 1741122428.514781, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'edx.cms.core.default', 'priority': 0, 'redelivered': None}, 'worker_pid': 15795}
    * {'id': 'adf80b0a-a149-4a7d-b1a8-d66141eefa4e', 'name': 'cms.djangoapps.contentstore.tasks.update_search_index', 'args': ['course-v1:REVVED+EV-MT-QLT+DEVELOPMENT', '2025-03-04T20:47:30.230188+00:00'], 'kwargs': {}, 'type': 'cms.djangoapps.contentstore.tasks.update_search_index', 'hostname': 'celery@edx.cms.core.default.%a156df8b1cb0', 'time_start': 1741122424.8484986, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'edx.cms.core.default', 'priority': 0, 'redelivered': None}, 'worker_pid': 16624}
    * {'id': 'f3fdb22a-ca61-4077-a350-88278022901f', 'name': 'cms.djangoapps.contentstore.tasks.update_search_index', 'args': ['course-v1:REVVED+EV-MT-QLT+DEVELOPMENT', '2025-03-04T20:50:54.985116+00:00'], 'kwargs': {}, 'type': 'cms.djangoapps.contentstore.tasks.update_search_index', 'hostname': 'celery@edx.cms.core.default.%a156df8b1cb0', 'time_start': 1741122487.2290258, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'edx.cms.core.default', 'priority': 0, 'redelivered': None}, 'worker_pid': 11662}
    * {'id': 'e5127fb2-c52a-4c36-86c1-b7f7eebe9447', 'name': 'cms.djangoapps.contentstore.tasks.update_search_index', 'args': ['course-v1:REVVED+EV-MT-QLT+DEVELOPMENT', '2025-03-04T20:53:19.579171+00:00'], 'kwargs': {}, 'type': 'cms.djangoapps.contentstore.tasks.update_search_index', 'hostname': 'celery@edx.cms.core.default.%a156df8b1cb0', 'time_start': 1741122538.5196118, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'edx.cms.core.default', 'priority': 0, 'redelivered': None}, 'worker_pid': 13393}
    * {'id': 'e196b16b-e2a7-4fec-ad7f-43d5bb4b56a4', 'name': 'cms.djangoapps.contentstore.tasks.update_search_index', 'args': ['course-v1:REVVED+EV-MT-QLT+DEVELOPMENT', '2025-03-04T20:47:42.202583+00:00'], 'kwargs': {}, 'type': 'cms.djangoapps.contentstore.tasks.update_search_index', 'hostname': 'celery@edx.cms.core.default.%a156df8b1cb0', 'time_start': 1741122436.8843408, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'edx.cms.core.default', 'priority': 0, 'redelivered': None}, 'worker_pid': 12414}
    * {'id': '03f5dac4-3634-4b98-bb0e-d65b2446c30c', 'name': 'cms.djangoapps.contentstore.tasks.update_search_index', 'args': ['course-v1:REVVED+EV-MT-QLT+DEVELOPMENT', '2025-03-04T20:47:21.740890+00:00'], 'kwargs': {}, 'type': 'cms.djangoapps.contentstore.tasks.update_search_index', 'hostname': 'celery@edx.cms.core.default.%a156df8b1cb0', 'time_start': 1741122415.6067073, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'edx.cms.core.default', 'priority': 0, 'redelivered': None}, 'worker_pid': 13249}
    * {'id': '637e3087-91bc-41f6-9f7b-65ac5691410a', 'name': 'cms.djangoapps.contentstore.tasks.update_search_index', 'args': ['course-v1:REVVED+EV-MT-QLT+DEVELOPMENT', '2025-03-04T20:51:56.595380+00:00'], 'kwargs': {}, 'type': 'cms.djangoapps.contentstore.tasks.update_search_index', 'hostname': 'celery@edx.cms.core.default.%a156df8b1cb0', 'time_start': 1741122527.732504, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'edx.cms.core.default', 'priority': 0, 'redelivered': None}, 'worker_pid': 10829}

Zachary_Trabookis · March 4, 2025, 9:22pm

@braden @dave
When I output the cms-worker logs I do see this error which makes me believe is what is causing all these celery worker tasks to be queued.

Something with elasticsearch and maxClauseCount set to 1024.

cms-worker logs

tutor_local-cms-worker-1  | [2025-03-04 21:19:26,074: ERROR/ForkPoolWorker-19] Indexing error encountered, courseware index may be out of date course-v1:REVVED+EV-MT-QLT+DEVELOPMENT - RequestError(400, 'search_phase_execution_exception', {'error': {'root_cause': [{'type': 'too_many_clauses', 'reason': 'too_many_clauses: maxClauseCount is set to 1024'}], 'type': 'search_phase_execution_exception', 'reason': 'all shards failed', 'phase': 'query', 'grouped': True, 'failed_shards': [{'shard': 0, 'index': 'courseware_content', 'node': 'dkHtHp4LQo6dGqFWQeddbw', 'reason': {'type': 'query_shard_exception', 'reason': 'failed to create query: maxClauseCount is set to 1024', 'index_uuid': 'aB5NoZx0Qqiv38JlSw0VFA', 'index': 'courseware_content', 'caused_by': {'type': 'too_many_clauses', 'reason': 'too_many_clauses: maxClauseCount is set to 1024'}}}], 'caused_by': {'type': 'too_many_clauses', 'reason': 'too_many_clauses: maxClauseCount is set to 1024'}}, 'status': 400})
tutor_local-cms-worker-1  | Traceback (most recent call last):
tutor_local-cms-worker-1  |   File "/openedx/edx-platform/cms/djangoapps/contentstore/courseware_index.py", line 339, in index
tutor_local-cms-worker-1  |     cls.remove_deleted_items(searcher, structure_key, indexed_items)
tutor_local-cms-worker-1  |   File "/openedx/edx-platform/cms/djangoapps/contentstore/courseware_index.py", line 185, in remove_deleted_items
tutor_local-cms-worker-1  |     response = searcher.search(
tutor_local-cms-worker-1  |   File "/openedx/venv/lib/python3.8/site-packages/search/elastic.py", line 645, in search
tutor_local-cms-worker-1  |     es_response = self._es.search(index=self.index_name, body=body, **kwargs)
tutor_local-cms-worker-1  |   File "/openedx/venv/lib/python3.8/site-packages/newrelic/hooks/datastore_elasticsearch.py", line 136, in _nr_wrapper_Elasticsearch_method_
tutor_local-cms-worker-1  |     result = wrapped(*args, **kwargs)
tutor_local-cms-worker-1  |   File "/openedx/venv/lib/python3.8/site-packages/elasticsearch/client/utils.py", line 168, in _wrapped
tutor_local-cms-worker-1  |     return func(*args, params=params, headers=headers, **kwargs)
tutor_local-cms-worker-1  |   File "/openedx/venv/lib/python3.8/site-packages/elasticsearch/client/__init__.py", line 1670, in search
tutor_local-cms-worker-1  |     return self.transport.perform_request(
tutor_local-cms-worker-1  |   File "/openedx/venv/lib/python3.8/site-packages/elasticsearch/transport.py", line 415, in perform_request
tutor_local-cms-worker-1  |     raise e
tutor_local-cms-worker-1  |   File "/openedx/venv/lib/python3.8/site-packages/elasticsearch/transport.py", line 381, in perform_request
tutor_local-cms-worker-1  |     status, headers_response, data = connection.perform_request(
tutor_local-cms-worker-1  |   File "/openedx/venv/lib/python3.8/site-packages/elasticsearch/connection/http_urllib3.py", line 277, in perform_request
tutor_local-cms-worker-1  |     self._raise_error(response.status, raw_data)
tutor_local-cms-worker-1  |   File "/openedx/venv/lib/python3.8/site-packages/elasticsearch/connection/base.py", line 330, in _raise_error
tutor_local-cms-worker-1  |     raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
tutor_local-cms-worker-1  | elasticsearch.exceptions.RequestError: RequestError(400, 'search_phase_execution_exception', 'too_many_clauses: maxClauseCount is set to 1024')
tutor_local-cms-worker-1  | [2025-03-04 21:19:26,077: ERROR/ForkPoolWorker-19] cms.djangoapps.contentstore.tasks.update_search_index[56b299e0-afc3-4bf8-a867-d33bb4efb554]: Search indexing error for complete course course-v1:REVVED+EV-MT-QLT+DEVELOPMENT - Error(s) present during indexing - ['General indexing error occurred']

cms logs (after CMS > Course > Reindex)

tutor_local-cms-1  | 2025-03-04 21:30:51,648 INFO 44 [tracking] [user 5] [ip 96.37.60.77] logger.py:41 - {"name": "/course/course-v1:REVVED+EV-MT-QLT+DEVELOPMENT/search_reindex", "context": {"user_id": 5, "path": "/course/course-v1:REVVED+EV-MT-QLT+DEVELOPMENT/search_reindex", "course_id": "", "org_id": "", "enterprise_uuid": ""}, "username": "ZacharyTrabookis", "session": "b3a43ab4457338af3941aa49e27cf676", "ip": "96.37.60.77", "agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36 Edg/133.0.0.0", "host": "cms.educateworkforce.com", "referer": "https://cms.educateworkforce.com/course/course-v1:REVVED+EV-MT-QLT+DEVELOPMENT", "accept_language": "en-US,en;q=0.9", "event": "{\"GET\": {}, \"POST\": {}}", "time": "2025-03-04T21:30:51.648513+00:00", "event_type": "/course/course-v1:REVVED+EV-MT-QLT+DEVELOPMENT/search_reindex", "event_source": "server", "page": null}
tutor_local-cms-1  | 2025-03-04 21:30:51,701 INFO 44 [elasticsearch] [user 5] [ip 96.37.60.77] base.py:270 - HEAD https://vpc-maple3-staging-x3plptgxuz6tlzlj56wz2e5xde.us-east-1.es.amazonaws.com:443/courseware_content [status:200 request:0.018s]
tutor_local-cms-1  | 2025-03-04 21:30:51,834 INFO 44 [elasticsearch] [user 5] [ip 96.37.60.77] base.py:270 - HEAD https://vpc-maple3-staging-x3plptgxuz6tlzlj56wz2e5xde.us-east-1.es.amazonaws.com:443/course_info [status:200 request:0.023s]
tutor_local-cms-1  | 2025-03-04 21:30:51,895 INFO 44 [elasticsearch] [user 5] [ip 96.37.60.77] base.py:270 - POST https://vpc-maple3-staging-x3plptgxuz6tlzlj56wz2e5xde.us-east-1.es.amazonaws.com:443/_bulk [status:200 request:0.016s]
tutor_local-cms-1  | 2025-03-04 21:30:53,328 INFO 44 [botocore.vendored.requests.packages.urllib3.connectionpool] [user 5] [ip 96.37.60.77] connectionpool.py:238 - Resetting dropped connection: s3.amazonaws.com
tutor_local-cms-1  | 2025-03-04 21:31:17,515 INFO 44 [elasticsearch] [user 5] [ip 96.37.60.77] base.py:270 - POST https://vpc-maple3-staging-x3plptgxuz6tlzlj56wz2e5xde.us-east-1.es.amazonaws.com:443/_bulk [status:200 request:0.087s]
tutor_local-cms-1  | 2025-03-04 21:31:17,657 INFO 44 [elasticsearch] [user 5] [ip 96.37.60.77] base.py:270 - POST https://vpc-maple3-staging-x3plptgxuz6tlzlj56wz2e5xde.us-east-1.es.amazonaws.com:443/_bulk [status:200 request:0.098s]
tutor_local-cms-1  | 2025-03-04 21:31:17,842 INFO 44 [elasticsearch] [user 5] [ip 96.37.60.77] base.py:270 - POST https://vpc-maple3-staging-x3plptgxuz6tlzlj56wz2e5xde.us-east-1.es.amazonaws.com:443/_bulk [status:200 request:0.144s]
tutor_local-cms-1  | 2025-03-04 21:31:18,192 INFO 44 [elasticsearch] [user 5] [ip 96.37.60.77] base.py:270 - POST https://vpc-maple3-staging-x3plptgxuz6tlzlj56wz2e5xde.us-east-1.es.amazonaws.com:443/_bulk [status:200 request:0.312s]
tutor_local-cms-1  | 2025-03-04 21:31:18,554 INFO 44 [elasticsearch] [user 5] [ip 96.37.60.77] base.py:270 - POST https://vpc-maple3-staging-x3plptgxuz6tlzlj56wz2e5xde.us-east-1.es.amazonaws.com:443/_bulk [status:200 request:0.318s]
tutor_local-cms-1  | 2025-03-04 21:31:18,615 INFO 44 [elasticsearch] [user 5] [ip 96.37.60.77] base.py:270 - POST https://vpc-maple3-staging-x3plptgxuz6tlzlj56wz2e5xde.us-east-1.es.amazonaws.com:443/_bulk [status:200 request:0.054s]
tutor_local-cms-1  | 2025-03-04 21:31:18,713 WARNING 44 [elasticsearch] [user 5] [ip 96.37.60.77] base.py:293 - POST https://vpc-maple3-staging-x3plptgxuz6tlzlj56wz2e5xde.us-east-1.es.amazonaws.com:443/courseware_content/_search [status:400 request:0.082s]
tutor_local-cms-1  | 2025-03-04 21:31:18,714 ERROR 44 [search.elastic] [user 5] [ip 96.37.60.77] elastic.py:647 - error while searching index - RequestError(400, 'search_phase_execution_exception', {'error': {'root_cause': [{'type': 'too_many_clauses', 'reason': 'too_many_clauses: maxClauseCount is set to 1024'}], 'type': 'search_phase_execution_exception', 'reason': 'all shards failed', 'phase': 'query', 'grouped': True, 'failed_shards': [{'shard': 0, 'index': 'courseware_content', 'node': 'dkHtHp4LQo6dGqFWQeddbw', 'reason': {'type': 'query_shard_exception', 'reason': 'failed to create query: maxClauseCount is set to 1024', 'index_uuid': 'aB5NoZx0Qqiv38JlSw0VFA', 'index': 'courseware_content', 'caused_by': {'type': 'too_many_clauses', 'reason': 'too_many_clauses: maxClauseCount is set to 1024'}}}], 'caused_by': {'type': 'too_many_clauses', 'reason': 'too_many_clauses: maxClauseCount is set to 1024'}}, 'status': 400})
tutor_local-cms-1  | Traceback (most recent call last):
tutor_local-cms-1  |   File "/openedx/venv/lib/python3.8/site-packages/search/elastic.py", line 645, in search
tutor_local-cms-1  |     es_response = self._es.search(index=self.index_name, body=body, **kwargs)
tutor_local-cms-1  |   File "/openedx/venv/lib/python3.8/site-packages/newrelic/hooks/datastore_elasticsearch.py", line 136, in _nr_wrapper_Elasticsearch_method_
tutor_local-cms-1  |     result = wrapped(*args, **kwargs)
tutor_local-cms-1  |   File "/openedx/venv/lib/python3.8/site-packages/elasticsearch/client/utils.py", line 168, in _wrapped
tutor_local-cms-1  |     return func(*args, params=params, headers=headers, **kwargs)
tutor_local-cms-1  |   File "/openedx/venv/lib/python3.8/site-packages/elasticsearch/client/__init__.py", line 1670, in search
tutor_local-cms-1  |     return self.transport.perform_request(
tutor_local-cms-1  |   File "/openedx/venv/lib/python3.8/site-packages/elasticsearch/transport.py", line 415, in perform_request
tutor_local-cms-1  |     raise e
tutor_local-cms-1  |   File "/openedx/venv/lib/python3.8/site-packages/elasticsearch/transport.py", line 381, in perform_request
tutor_local-cms-1  |     status, headers_response, data = connection.perform_request(
tutor_local-cms-1  |   File "/openedx/venv/lib/python3.8/site-packages/elasticsearch/connection/http_urllib3.py", line 277, in perform_request
tutor_local-cms-1  |     self._raise_error(response.status, raw_data)
tutor_local-cms-1  |   File "/openedx/venv/lib/python3.8/site-packages/elasticsearch/connection/base.py", line 330, in _raise_error
tutor_local-cms-1  |     raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
tutor_local-cms-1  | elasticsearch.exceptions.RequestError: RequestError(400, 'search_phase_execution_exception', 'too_many_clauses: maxClauseCount is set to 1024')
tutor_local-cms-1  | 2025-03-04 21:31:18,715 ERROR 44 [edx.modulestore] [user 5] [ip 96.37.60.77] courseware_index.py:342 - Indexing error encountered, courseware index may be out of date course-v1:REVVED+EV-MT-QLT+DEVELOPMENT - RequestError(400, 'search_phase_execution_exception', {'error': {'root_cause': [{'type': 'too_many_clauses', 'reason': 'too_many_clauses: maxClauseCount is set to 1024'}], 'type': 'search_phase_execution_exception', 'reason': 'all shards failed', 'phase': 'query', 'grouped': True, 'failed_shards': [{'shard': 0, 'index': 'courseware_content', 'node': 'dkHtHp4LQo6dGqFWQeddbw', 'reason': {'type': 'query_shard_exception', 'reason': 'failed to create query: maxClauseCount is set to 1024', 'index_uuid': 'aB5NoZx0Qqiv38JlSw0VFA', 'index': 'courseware_content', 'caused_by': {'type': 'too_many_clauses', 'reason': 'too_many_clauses: maxClauseCount is set to 1024'}}}], 'caused_by': {'type': 'too_many_clauses', 'reason': 'too_many_clauses: maxClauseCount is set to 1024'}}, 'status': 400})
tutor_local-cms-1  | Traceback (most recent call last):
tutor_local-cms-1  |   File "/openedx/edx-platform/./cms/djangoapps/contentstore/courseware_index.py", line 339, in index
tutor_local-cms-1  |     cls.remove_deleted_items(searcher, structure_key, indexed_items)
tutor_local-cms-1  |   File "/openedx/edx-platform/./cms/djangoapps/contentstore/courseware_index.py", line 185, in remove_deleted_items
tutor_local-cms-1  |     response = searcher.search(
tutor_local-cms-1  |   File "/openedx/venv/lib/python3.8/site-packages/search/elastic.py", line 645, in search
tutor_local-cms-1  |     es_response = self._es.search(index=self.index_name, body=body, **kwargs)
tutor_local-cms-1  |   File "/openedx/venv/lib/python3.8/site-packages/newrelic/hooks/datastore_elasticsearch.py", line 136, in _nr_wrapper_Elasticsearch_method_
tutor_local-cms-1  |     result = wrapped(*args, **kwargs)
tutor_local-cms-1  |   File "/openedx/venv/lib/python3.8/site-packages/elasticsearch/client/utils.py", line 168, in _wrapped
tutor_local-cms-1  |     return func(*args, params=params, headers=headers, **kwargs)
tutor_local-cms-1  |   File "/openedx/venv/lib/python3.8/site-packages/elasticsearch/client/__init__.py", line 1670, in search
tutor_local-cms-1  |     return self.transport.perform_request(
tutor_local-cms-1  |   File "/openedx/venv/lib/python3.8/site-packages/elasticsearch/transport.py", line 415, in perform_request
tutor_local-cms-1  |     raise e
tutor_local-cms-1  |   File "/openedx/venv/lib/python3.8/site-packages/elasticsearch/transport.py", line 381, in perform_request
tutor_local-cms-1  |     status, headers_response, data = connection.perform_request(
tutor_local-cms-1  |   File "/openedx/venv/lib/python3.8/site-packages/elasticsearch/connection/http_urllib3.py", line 277, in perform_request
tutor_local-cms-1  |     self._raise_error(response.status, raw_data)
tutor_local-cms-1  |   File "/openedx/venv/lib/python3.8/site-packages/elasticsearch/connection/base.py", line 330, in _raise_error
tutor_local-cms-1  |     raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
tutor_local-cms-1  | elasticsearch.exceptions.RequestError: RequestError(400, 'search_phase_execution_exception', 'too_many_clauses: maxClauseCount is set to 1024')
tutor_local-cms-1  | 2025-03-04 21:31:18,736 ERROR 44 [django.request] [user None] [ip None] log.py:224 - Internal Server Error: /course/course-v1:REVVED+EV-MT-QLT+DEVELOPMENT/search_reindex
tutor_local-cms-1  | [pid: 44|app: 0|req: 1115/4456] 172.18.0.5 () {70 vars in 2318 bytes} [Tue Mar  4 21:30:51 2025] GET /course/course-v1:REVVED+EV-MT-QLT+DEVELOPMENT/search_reindex => generated 53 bytes in 29197 msecs (HTTP/1.1 500) 7 headers in 533 bytes (1 switches on core 0)

Zachary_Trabookis · March 4, 2025, 9:53pm

@braden @dave
It appears that someone else had this too_many_clauses: maxClauseCount is set to 1024 issue.

They ended up modifying the elasicsearch Docker service configuration to 4096 size.

version: "3.7"
services:

  elasticsearch:
    environment:
      - indices.query.bool.max_clause_count=4096

We’re using Amazon OpenSearch Service instead of elasticsearch and therefore we need to alter this using

ChatGPT (ChatGPT - Elasticsearch maxClauseCount config) mentions doing this.

Amazon OpenSearch Service (previously Amazon Elasticsearch Service) does not allow direct modification of elasticsearch.yml, but you can update maxClauseCount dynamically using the OpenSearch API.

Update `maxClauseCount` in Amazon OpenSearch Service

Since Amazon OpenSearch runs in a managed environment, you must use the _cluster/settings API to modify indices.query.bool.max_clause_count.

1. Increase `maxClauseCount` Dynamically

Run the following request using cURL, Kibana (Dev Tools), or AWS OpenSearch API:

PUT _cluster/settings
{
  "persistent": {
    "indices.query.bool.max_clause_count": 4096
  }
}

This change will persist until the cluster is restarted.

2. Verify the New Setting

After applying the change, confirm the update using:

GET _cluster/settings

3. Make It Permanent

Amazon OpenSearch does not allow modifying elasticsearch.yml, so you must set this value dynamically after every cluster restart. If frequent restarts occur, consider automating this API request using AWS Lambda or a similar approach.

Zachary_Trabookis · March 5, 2025, 5:14am

For anyone using AWS OpenSearch Elasticsearch engine 7.10 there is an advanced setting where you define the Max clause count. I set this to value 4096 and the cluster performed a blue/green deployment.

Once the cluster updated, I was able to perform a CMS > Course > Reindex successfully. We’ll see how this performs and notify her if we run into any more issues with this configuration.

cc @braden @dave

Zachary_Trabookis · March 5, 2025, 2:28pm

We’ve also add in the additional celery worker options to limit CPU core usage on the server with all services running. This is a temporary fix until we can setup horizontal scaling with separation of services.

The cms-worker Docker container now has the following options added.

/openedx/venv/bin/python /openedx/venv/bin/celery worker --loglevel=info --hostname=edx.cms.core.default.%%h --maxtasksperchild 100 --exclude-queues=edx.lms.core.default –autoscale=8,4 --concurrency=8

I see now that there are a limited number (8) processes running cms.celery tasks initially.

dave · March 5, 2025, 3:29pm

Are you actually using the course content search feature? Maybe it would be better to turn that off altogether?

Zachary_Trabookis · March 5, 2025, 4:38pm

@dave
That’s a good suggestion, thank you. It looks like the cms.celery queue was receiving a lot of update_search_index calls and after looking at the code within the listen_for_course_publish handler is when this was being triggered.

github.com/openedx/edx-platform

maple.master/cms/djangoapps/contentstore/signals/handlers.py

open-release


      
          # Finally call into the course search subsystem
          # to kick off an indexing action
          if CoursewareSearchIndexer.indexing_is_enabled() and CourseAboutSearchIndexer.indexing_is_enabled():
              update_search_index.delay(course_key_str, datetime.now(UTC).isoformat())

I see that we an disable this by settings.FEATURES['ENABLE_COURSEWARE_INDEX'] = False, however, I think that we’ll leave it enable to ensure search cabalities for future releases of Open edX. I realize that the LMS frontend-app-learning for this release removed the course search interface so it’s not useful for that purpose but we do have the search enabled on the LMS Dashboard page.

Additionally, we have the following configuration set.

FEATURES[‘ENABLE_COURSEWARE_SEARCH’] = True
FEATURES[‘ENABLE_DASHBOARD_SEARCH’] = True

joel.edwards · March 6, 2025, 5:47am

You might be interested in updating to the latest version where Elasticsearch is no longer used, refer changelogs for v19.0.0:

[Feature] Replace Elasticsearch by Meilisearch. Elasticsearch was both a source of complexity and high resource usage. With this change, we no longer run Elasticsearch to perform common search queries across Open edX. This includes: course discovery, courseware search and studio search. Instead, we index all these documents in a Meilisearch instance, which is much more lightweight in terms of memory consumption.

Remember to update your DNS (if you’re not using a wildcard to resolve your server’s subdomains) to include meilisearch (eg meilisearch.lms.yourdomain.tld)

Topic		Replies	Views
Please help me, Tutor Help tutor	4	233	October 17, 2024
Horizontal Scaling - CMS/LMS and Workers Site Operators	9	136	August 8, 2024
LMS and CMS workers Site Operations Help how-to , tutor	1	375	April 6, 2023
CMS worker processing time for update_outline task Tutor Help tutor , cms	0	250	July 21, 2022
Open edx consuming huge amount of memory Site Operators	4	1463	November 21, 2019