Cannot edit course blocks in studio - CMS worker not receving jobs

martibosch · January 16, 2023, 12:51pm

Hello,

I recently successfully upgraded our tutor platform to olive (from nutmeg). Everything seemed to work fine, but one day after we noticed that we are unable to edit course blocks, e.g., I click on “EDIT” and change some text or images, and then click “Save”, but nothing happens and the “Publish” button remains disabled as if no changes were made whatsoever.

I have restarted tutor and tested it numerous times and managed to save the edits only twice, and I have not detected any pattern on why it worked when it worked.

Digging into the logs, the issue seems clear: the CMS worker is not receiving any task. It only shows logs for the two times that the edits worked.

I can also provide the full CMS logs for an edit block attempt if needed.

Why is this happening?

Thank you. Best,
Martí

PD: FYI:

we build a custom openedx image to include our custom theme (DOCKER_IMAGE_OPENEDX is set to Package openedx · GitHub - here is the action that built the image: build openedx container · martibosch/acl-tutor-do@2564f81 · GitHub). This worked properly in nutmeg.
pip freeze | grep tutor in the server gives:

tutor==15.1.0
tutor-android==15.0.0
tutor-discovery==15.0.0
tutor-ecommerce==15.0.2
tutor-forum==15.0.0
tutor-license==15.0.0
tutor-mfe==15.0.4
tutor-minio==15.0.0
tutor-notes==15.0.0
tutor-webui==15.0.0
tutor-xqueue==15.0.0

tutor plugins list in the server gives:

android            (disabled)  15.0.0
coursediscovery                /home/ubuntu/.local/share/tutor-plugins/coursediscovery.py
customcert                     /home/ubuntu/.local/share/tutor-plugins/customcert.py
discovery                      15.0.0
ecommerce          (disabled)  15.0.2
forum                          15.0.0
license            (disabled)  15.0.0
mfe                            15.0.4
minio              (disabled)  15.0.0
notes              (disabled)  15.0.0
webui              (disabled)  15.0.0
xqueue                         15.0.0

the two custom plugins are:

from tutor import hooks

hooks.Filters.ENV_PATCHES.add_item(
    (
        "openedx-lms-common-settings",
        """
FEATURES["ENABLE_COURSE_DISCOVERY"] = False
FEATURES["ENABLE_COURSEWARE_SEARCH"] = False
FEATURES["ENABLE_DASHBOARD_SEARCH"] = False
"""
    )
)

and

from tutor import hooks

hooks.Filters.ENV_PATCHES.add_item(
    (
        "openedx-lms-common-settings",
        "FEATURES['CUSTOM_CERTIFICATE_TEMPLATES_ENABLED'] = True",
    )
)

regis · January 17, 2023, 3:31pm

Hi @martibosch,

When you make changes to a course from the studio, it is not supposed to trigger an asynchronous task. A call to the cms-worker is only made once you click “publish” – but the problem occurs sooner than that, right?

Your problem occurs whenever you make a change to a course unit and hit “save”, right?

Let’s start in the browser. When you hit save, it should trigger the following requests:

You should not have any error in the javascript console.

Do you observe the same behaviour?

Can you also post the logs from the cms container? The first line after you hit “save” should be similar to this one:

cms_1                        | 2023-01-17 15:29:44,980 INFO 25 [tracking] [user 5] [ip 77.195.23.18] logger.py:41 - {"name": "/xblock/block-v1:edX+DemoX+Demo_Course+type@html+block@9a9703d932d14c28a2a9eff53c86327b", "context": {"user_id": 5, "path": "/xblock/block-v1:edX+DemoX+Demo_Course+type@html+block@9a9703d932d14c28a2a9eff53c86327b", "course_id": "", "org_id": "", "enterprise_uuid": ""}, "username": "admin", "session": "e8c635b4e265349a9f846c6c7eae9ca2", "ip": "77.195.23.18", "agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0", "host": "studio.demo.openedx.overhang.io", "referer": "https://studio.demo.openedx.overhang.io/container/block-v1:edX+DemoX+Demo_Course+type@vertical+block@46f3f1d9ddd04c0da624c91dfcd6fb08?action=new", "accept_language": "fr,en;q=0.5", "event": "{\"GET\": {}, \"POST\": {}}", "time": "2023-01-17T15:29:44.979645+00:00", "event_type": "/xblock/block-v1:edX+DemoX+Demo_Course+type@html+block@9a9703d932d14c28a2a9eff53c86327b", "event_source": "server", "page": null}

You’ll probably get many worrying errors concerning CORS errors, such as this one:

cms_1                        | 2023-01-17 15:30:11,359 INFO 7 [openedx.core.djangoapps.cors_csrf.helpers] [user 7] [ip 31.223.46.44] helpers.py:64 - Origin 'https://studio.demo.openedx.overhang.io' was not in `CORS_ORIGIN_WHITELIST`; full referer was 'https://studio.demo.openedx.overhang.io/learning/course/course-v1:edX+DemoX+Demo_Course/home' and requested host was 'studio.demo.openedx.overhang.io'; CORS_ORIGIN_ALLOW_ALL=False

You can safely ignore those errors.

It would also help if you managed to replicate the issue on a vanilla Open edX instance. Can you try to replicate the error on the demo Open edX platform?

martibosch · January 17, 2023, 4:38pm

Hello,

thank you for your thorough answer. It basically seems that one of the POST requests is not triggered:

The CMS logs seem quite equivalent:

tutor_local-cms-1  | 2023-01-17 16:05:46,293 INFO 7 [tracking] [user 6] [ip 194.230.158.119] logger.py:41 - {"name": "/xblock/container/block-v1:EPFL+VA_MTU+2022+type@vertical+block@179320df1fe1464b8f0dded76714e0c5", "context": {"user_id": 6, "path": "/xblock/container/block-v1:EPFL+VA_MTU+2022+type@vertical+block@179320df1fe1464b8f0dded76714e0c5", "course_id": "", "org_id": "", "enterprise_uuid": ""}, "username": "martibosch", "session": "def6beb60d6f5969296176e4dea92346", "ip": "194.230.158.119", "agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0", "host": "studio.courses.africancitieslab.org", "referer": "https://studio.courses.africancitieslab.org/container/block-v1:EPFL+VA_MTU+2022+type@vertical+block@179320df1fe1464b8f0dded76714e0c5", "accept_language": "en-US,en;q=0.5", "event": "{\"GET\": {}, \"POST\": {}}", "time": "2023-01-17T16:05:46.293179+00:00", "event_type": "/xblock/container/block-v1:EPFL+VA_MTU+2022+type@vertical+block@179320df1fe1464b8f0dded76714e0c5", "event_source": "server", "page": null}

I observe the same requests when editing the demo course in the demo platform:

However, the block is properly updated.

In a few cases, the save button works properly in our platform, however the triggered requests seem to be the same:

and the corresponding logs (I am attaching all the corresponding requests in case they are useful):

tutor_local-cms-1  | 2023-01-17 16:27:43,208 INFO 25 [tracking] [user 6] [ip 194.230.158.119] logger.py:41 - {"name": "/xblock/block-v1:EPFL+VA_MTU+2022+type@html+block@aab222e4d92c4eb28e53c90171f89681", "context": {"user_id": 6, "path": "/xblock/block-v1:EPFL+VA_MTU+2022+type@html+block@aab222e4d92c4eb28e53c90171f89681", "course_id": "", "org_id": "", "enterprise_uuid": ""}, "username": "martibosch", "session": "def6beb60d6f5969296176e4dea92346", "ip": "194.230.158.119", "agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0", "host": "studio.courses.africancitieslab.org", "referer": "https://studio.courses.africancitieslab.org/container/block-v1:EPFL+VA_MTU+2022+type@vertical+block@179320df1fe1464b8f0dded76714e0c5", "accept_language": "en-US,en;q=0.5", "event": "{\"GET\": {}, \"POST\": {}}", "time": "2023-01-17T16:27:43.208446+00:00", "event_type": "/xblock/block-v1:EPFL+VA_MTU+2022+type@html+block@aab222e4d92c4eb28e53c90171f89681", "event_source": "server", "page": null}
tutor_local-cms-1  | 2023-01-17 16:27:43,227 INFO 25 [openedx.core.djangoapps.cors_csrf.helpers] [user 6] [ip 194.230.158.119] helpers.py:64 - Origin 'https://studio.courses.africancitieslab.org' was not in `CORS_ORIGIN_WHITELIST`; full referer was 'https://studio.courses.africancitieslab.org/container/block-v1:EPFL+VA_MTU+2022+type@vertical+block@179320df1fe1464b8f0dded76714e0c5' and requested host was 'studio.courses.africancitieslab.org'; CORS_ORIGIN_ALLOW_ALL=False
tutor_local-cms-1  | [pid: 25|app: 0|req: 768/3672] 172.18.0.8 () {62 vars in 3930 bytes} [Tue Jan 17 16:27:43 2023] POST /xblock/block-v1%3AEPFL%2BVA_MTU%2B2022%2Btype%40html%2Bblock%40aab222e4d92c4eb28e53c90171f89681 => generated 4949 bytes in 223 msecs (HTTP/1.1 200) 7 headers in 379 bytes (1 switches on core 0)
tutor_local-cms-1  | 2023-01-17 16:27:43,520 INFO 27 [tracking] [user 6] [ip 194.230.158.119] logger.py:41 - {"name": "/xblock/block-v1:EPFL+VA_MTU+2022+type@html+block@aab222e4d92c4eb28e53c90171f89681/reorderable_container_child_preview", "context": {"user_id": 6, "path": "/xblock/block-v1:EPFL+VA_MTU+2022+type@html+block@aab222e4d92c4eb28e53c90171f89681/reorderable_container_child_preview", "course_id": "", "org_id": "", "enterprise_uuid": ""}, "username": "martibosch", "session": "def6beb60d6f5969296176e4dea92346", "ip": "194.230.158.119", "agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0", "host": "studio.courses.africancitieslab.org", "referer": "https://studio.courses.africancitieslab.org/container/block-v1:EPFL+VA_MTU+2022+type@vertical+block@179320df1fe1464b8f0dded76714e0c5", "accept_language": "en-US,en;q=0.5", "event": "{\"GET\": {\"_\": [\"1673972484605\"]}, \"POST\": {}}", "time": "2023-01-17T16:27:43.520468+00:00", "event_type": "/xblock/block-v1:EPFL+VA_MTU+2022+type@html+block@aab222e4d92c4eb28e53c90171f89681/reorderable_container_child_preview", "event_source": "server", "page": null}
tutor_local-cms-1  | 2023-01-17 16:27:43,539 INFO 27 [openedx.core.djangoapps.cors_csrf.helpers] [user 6] [ip 194.230.158.119] helpers.py:64 - Origin 'https://studio.courses.africancitieslab.org' was not in `CORS_ORIGIN_WHITELIST`; full referer was 'https://studio.courses.africancitieslab.org/container/block-v1:EPFL+VA_MTU+2022+type@vertical+block@179320df1fe1464b8f0dded76714e0c5' and requested host was 'studio.courses.africancitieslab.org'; CORS_ORIGIN_ALLOW_ALL=False
tutor_local-cms-1  | [pid: 27|app: 0|req: 772/3673] 172.18.0.8 () {54 vars in 3842 bytes} [Tue Jan 17 16:27:43 2023] GET /xblock/block-v1:EPFL+VA_MTU+2022+type@html+block@aab222e4d92c4eb28e53c90171f89681/reorderable_container_child_preview?_=1673972484605 => generated 13563 bytes in 260 msecs (HTTP/1.1 200) 7 headers in 508 bytes (1 switches on core 0)
tutor_local-cms-1  | 2023-01-17 16:27:44,003 INFO 26 [tracking] [user 6] [ip 194.230.158.119] logger.py:41 - {"name": "/xblock/container/block-v1:EPFL+VA_MTU+2022+type@vertical+block@179320df1fe1464b8f0dded76714e0c5", "context": {"user_id": 6, "path": "/xblock/container/block-v1:EPFL+VA_MTU+2022+type@vertical+block@179320df1fe1464b8f0dded76714e0c5", "course_id": "", "org_id": "", "enterprise_uuid": ""}, "username": "martibosch", "session": "def6beb60d6f5969296176e4dea92346", "ip": "194.230.158.119", "agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0", "host": "studio.courses.africancitieslab.org", "referer": "https://studio.courses.africancitieslab.org/container/block-v1:EPFL+VA_MTU+2022+type@vertical+block@179320df1fe1464b8f0dded76714e0c5", "accept_language": "en-US,en;q=0.5", "event": "{\"GET\": {}, \"POST\": {}}", "time": "2023-01-17T16:27:44.003494+00:00", "event_type": "/xblock/container/block-v1:EPFL+VA_MTU+2022+type@vertical+block@179320df1fe1464b8f0dded76714e0c5", "event_source": "server", "page": null}
tutor_local-cms-1  | 2023-01-17 16:27:44,022 INFO 26 [openedx.core.djangoapps.cors_csrf.helpers] [user 6] [ip 194.230.158.119] helpers.py:64 - Origin 'https://studio.courses.africancitieslab.org' was not in `CORS_ORIGIN_WHITELIST`; full referer was 'https://studio.courses.africancitieslab.org/container/block-v1:EPFL+VA_MTU+2022+type@vertical+block@179320df1fe1464b8f0dded76714e0c5' and requested host was 'studio.courses.africancitieslab.org'; CORS_ORIGIN_ALLOW_ALL=False
tutor_local-cms-1  | [pid: 26|app: 0|req: 776/3674] 172.18.0.8 () {54 vars in 3797 bytes} [Tue Jan 17 16:27:43 2023] GET /xblock/container/block-v1:EPFL+VA_MTU+2022+type@vertical+block@179320df1fe1464b8f0dded76714e0c5 => generated 11202 bytes in 247 msecs (HTTP/1.1 200) 6 headers in 340 bytes (1 switches on core 0)

At the same time, I just found out that the issue does not happen when using the “Raw” editor (instead of the “Visual”).

Thank you again for your help. Best,
Martí

regis · January 17, 2023, 5:01pm

And there is no error in the javascript console?

martibosch · January 17, 2023, 5:34pm

In blocks with images there are JS GET errors of the form:

GET https://studio.courses.africancitieslab.org/static/Jerome_b-png.jpg

but these are no dealbreakers. One surprising pattern that I have found is this: if I add blank lines (they translate to <p></p>, the edits are saved

Peek 2023-01-17 18-32

regis · January 17, 2023, 6:34pm

Wait, does this happen only with the html blocks? What about videos? Could it be related to this fix? fix: backport fix for html component editing in studio by regisb · Pull Request #780 · overhangio/tutor · GitHub

martibosch · January 18, 2023, 7:06am

It does indeed sound like it.

However, I have tried to fix it and so far I have seen no luck. Here is what I have done:

cp .local/share/tutor/env/build/openedx/Dockerfile Dockerfile.backup
pip uninstall tutor
pip install https://github.com/overhangio/tutor/archive/refs/heads/regisb/fix-tinymce-formatting.zip
tutor config save

Then, the following:

diff .local/share/tutor/env/build/openedx/Dockerfile Dockerfile.backup

outputs:

48,50d47
< # Fix TinyMCE editor for html components
< # https://github.com/openedx/edx-platform/pull/31500
< RUN curl -fsSL https://github.com/openedx/edx-platform/commit/5af42f8ba3.patch | git am

So far so good. Then, I have done

tutor local launch -I

which ran succesfully. However the problem persists.

Have I done everything correctly?

CC: @Anh_Vu_Nguy_n @pdpinch ?

regis · January 18, 2023, 7:22am

Did you re-build the Docker image? If not, do:

tutor images build openedx
tutor local launch -I

martibosch · January 19, 2023, 6:12am

yes, that did it. I rebuilt our custom image (because we use a custom theme) in a dedicated github workflow using tutor from the master branch (after you merged the PR): acl-tutor-do/build.yaml at fix-temp-tinymce · martibosch/acl-tutor-do · GitHub

Everything seems to be working smoothly now.

Immense thanks!
Martí

regis · January 19, 2023, 9:00am

Awesome. I merged the pr and will make a new release later today.

system · April 19, 2023, 9:01am

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Cannot import or export classes (or run grading, or submit to problems) Site Operations Help nutmeg	8	862	June 8, 2023
Tutor: Error during Upgrading from Nutmeg to Olive Tutor Help how-to , tutor , nutmeg , olive , tutor-plugin	2	137	February 7, 2024
Not able to access view course in Open edX Olive version Tutor Help	1	306	May 30, 2023
ERROR/ForkPoolWorker-5 "Could not create course outline for course" Site Operations Help	2	131	May 21, 2024
eduNEXT tutor-contrib-codejail - update to Nutmeg fails because wrong lms / cms config Site Operations Help tutor	5	299	September 14, 2022

Cannot edit course blocks in studio - CMS worker not receving jobs

Related topics