Cannot edit course blocks in studio - CMS worker not receving jobs

Hello,

I recently successfully upgraded our tutor platform to olive (from nutmeg). Everything seemed to work fine, but one day after we noticed that we are unable to edit course blocks, e.g., I click on “EDIT” and change some text or images, and then click “Save”, but nothing happens and the “Publish” button remains disabled as if no changes were made whatsoever.

I have restarted tutor and tested it numerous times and managed to save the edits only twice, and I have not detected any pattern on why it worked when it worked.

Digging into the logs, the issue seems clear: the CMS worker is not receiving any task. It only shows logs for the two times that the edits worked.

I can also provide the full CMS logs for an edit block attempt if needed.

Why is this happening?

Thank you. Best,
Martí

PD: FYI:

tutor==15.1.0
tutor-android==15.0.0
tutor-discovery==15.0.0
tutor-ecommerce==15.0.2
tutor-forum==15.0.0
tutor-license==15.0.0
tutor-mfe==15.0.4
tutor-minio==15.0.0
tutor-notes==15.0.0
tutor-webui==15.0.0
tutor-xqueue==15.0.0
  • tutor plugins list in the server gives:
android            (disabled)  15.0.0
coursediscovery                /home/ubuntu/.local/share/tutor-plugins/coursediscovery.py
customcert                     /home/ubuntu/.local/share/tutor-plugins/customcert.py
discovery                      15.0.0
ecommerce          (disabled)  15.0.2
forum                          15.0.0
license            (disabled)  15.0.0
mfe                            15.0.4
minio              (disabled)  15.0.0
notes              (disabled)  15.0.0
webui              (disabled)  15.0.0
xqueue                         15.0.0
  • the two custom plugins are:
from tutor import hooks

hooks.Filters.ENV_PATCHES.add_item(
    (
        "openedx-lms-common-settings",
        """
FEATURES["ENABLE_COURSE_DISCOVERY"] = False
FEATURES["ENABLE_COURSEWARE_SEARCH"] = False
FEATURES["ENABLE_DASHBOARD_SEARCH"] = False
"""
    )
)

and

from tutor import hooks

hooks.Filters.ENV_PATCHES.add_item(
    (
        "openedx-lms-common-settings",
        "FEATURES['CUSTOM_CERTIFICATE_TEMPLATES_ENABLED'] = True",
    )
)

Hi @martibosch,

When you make changes to a course from the studio, it is not supposed to trigger an asynchronous task. A call to the cms-worker is only made once you click “publish” – but the problem occurs sooner than that, right?

Your problem occurs whenever you make a change to a course unit and hit “save”, right?

Let’s start in the browser. When you hit save, it should trigger the following requests:

You should not have any error in the javascript console.

Do you observe the same behaviour?

Can you also post the logs from the cms container? The first line after you hit “save” should be similar to this one:

cms_1                        | 2023-01-17 15:29:44,980 INFO 25 [tracking] [user 5] [ip 77.195.23.18] logger.py:41 - {"name": "/xblock/block-v1:edX+DemoX+Demo_Course+type@html+block@9a9703d932d14c28a2a9eff53c86327b", "context": {"user_id": 5, "path": "/xblock/block-v1:edX+DemoX+Demo_Course+type@html+block@9a9703d932d14c28a2a9eff53c86327b", "course_id": "", "org_id": "", "enterprise_uuid": ""}, "username": "admin", "session": "e8c635b4e265349a9f846c6c7eae9ca2", "ip": "77.195.23.18", "agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0", "host": "studio.demo.openedx.overhang.io", "referer": "https://studio.demo.openedx.overhang.io/container/block-v1:edX+DemoX+Demo_Course+type@vertical+block@46f3f1d9ddd04c0da624c91dfcd6fb08?action=new", "accept_language": "fr,en;q=0.5", "event": "{\"GET\": {}, \"POST\": {}}", "time": "2023-01-17T15:29:44.979645+00:00", "event_type": "/xblock/block-v1:edX+DemoX+Demo_Course+type@html+block@9a9703d932d14c28a2a9eff53c86327b", "event_source": "server", "page": null}

You’ll probably get many worrying errors concerning CORS errors, such as this one:

cms_1                        | 2023-01-17 15:30:11,359 INFO 7 [openedx.core.djangoapps.cors_csrf.helpers] [user 7] [ip 31.223.46.44] helpers.py:64 - Origin 'https://studio.demo.openedx.overhang.io' was not in `CORS_ORIGIN_WHITELIST`; full referer was 'https://studio.demo.openedx.overhang.io/learning/course/course-v1:edX+DemoX+Demo_Course/home' and requested host was 'studio.demo.openedx.overhang.io'; CORS_ORIGIN_ALLOW_ALL=False

You can safely ignore those errors.

It would also help if you managed to replicate the issue on a vanilla Open edX instance. Can you try to replicate the error on the demo Open edX platform?

1 Like

Hello,

thank you for your thorough answer. It basically seems that one of the POST requests is not triggered:

The CMS logs seem quite equivalent:

tutor_local-cms-1  | 2023-01-17 16:05:46,293 INFO 7 [tracking] [user 6] [ip 194.230.158.119] logger.py:41 - {"name": "/xblock/container/block-v1:EPFL+VA_MTU+2022+type@vertical+block@179320df1fe1464b8f0dded76714e0c5", "context": {"user_id": 6, "path": "/xblock/container/block-v1:EPFL+VA_MTU+2022+type@vertical+block@179320df1fe1464b8f0dded76714e0c5", "course_id": "", "org_id": "", "enterprise_uuid": ""}, "username": "martibosch", "session": "def6beb60d6f5969296176e4dea92346", "ip": "194.230.158.119", "agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0", "host": "studio.courses.africancitieslab.org", "referer": "https://studio.courses.africancitieslab.org/container/block-v1:EPFL+VA_MTU+2022+type@vertical+block@179320df1fe1464b8f0dded76714e0c5", "accept_language": "en-US,en;q=0.5", "event": "{\"GET\": {}, \"POST\": {}}", "time": "2023-01-17T16:05:46.293179+00:00", "event_type": "/xblock/container/block-v1:EPFL+VA_MTU+2022+type@vertical+block@179320df1fe1464b8f0dded76714e0c5", "event_source": "server", "page": null}

I observe the same requests when editing the demo course in the demo platform:

However, the block is properly updated.

In a few cases, the save button works properly in our platform, however the triggered requests seem to be the same:

and the corresponding logs (I am attaching all the corresponding requests in case they are useful):

tutor_local-cms-1  | 2023-01-17 16:27:43,208 INFO 25 [tracking] [user 6] [ip 194.230.158.119] logger.py:41 - {"name": "/xblock/block-v1:EPFL+VA_MTU+2022+type@html+block@aab222e4d92c4eb28e53c90171f89681", "context": {"user_id": 6, "path": "/xblock/block-v1:EPFL+VA_MTU+2022+type@html+block@aab222e4d92c4eb28e53c90171f89681", "course_id": "", "org_id": "", "enterprise_uuid": ""}, "username": "martibosch", "session": "def6beb60d6f5969296176e4dea92346", "ip": "194.230.158.119", "agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0", "host": "studio.courses.africancitieslab.org", "referer": "https://studio.courses.africancitieslab.org/container/block-v1:EPFL+VA_MTU+2022+type@vertical+block@179320df1fe1464b8f0dded76714e0c5", "accept_language": "en-US,en;q=0.5", "event": "{\"GET\": {}, \"POST\": {}}", "time": "2023-01-17T16:27:43.208446+00:00", "event_type": "/xblock/block-v1:EPFL+VA_MTU+2022+type@html+block@aab222e4d92c4eb28e53c90171f89681", "event_source": "server", "page": null}
tutor_local-cms-1  | 2023-01-17 16:27:43,227 INFO 25 [openedx.core.djangoapps.cors_csrf.helpers] [user 6] [ip 194.230.158.119] helpers.py:64 - Origin 'https://studio.courses.africancitieslab.org' was not in `CORS_ORIGIN_WHITELIST`; full referer was 'https://studio.courses.africancitieslab.org/container/block-v1:EPFL+VA_MTU+2022+type@vertical+block@179320df1fe1464b8f0dded76714e0c5' and requested host was 'studio.courses.africancitieslab.org'; CORS_ORIGIN_ALLOW_ALL=False
tutor_local-cms-1  | [pid: 25|app: 0|req: 768/3672] 172.18.0.8 () {62 vars in 3930 bytes} [Tue Jan 17 16:27:43 2023] POST /xblock/block-v1%3AEPFL%2BVA_MTU%2B2022%2Btype%40html%2Bblock%40aab222e4d92c4eb28e53c90171f89681 => generated 4949 bytes in 223 msecs (HTTP/1.1 200) 7 headers in 379 bytes (1 switches on core 0)
tutor_local-cms-1  | 2023-01-17 16:27:43,520 INFO 27 [tracking] [user 6] [ip 194.230.158.119] logger.py:41 - {"name": "/xblock/block-v1:EPFL+VA_MTU+2022+type@html+block@aab222e4d92c4eb28e53c90171f89681/reorderable_container_child_preview", "context": {"user_id": 6, "path": "/xblock/block-v1:EPFL+VA_MTU+2022+type@html+block@aab222e4d92c4eb28e53c90171f89681/reorderable_container_child_preview", "course_id": "", "org_id": "", "enterprise_uuid": ""}, "username": "martibosch", "session": "def6beb60d6f5969296176e4dea92346", "ip": "194.230.158.119", "agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0", "host": "studio.courses.africancitieslab.org", "referer": "https://studio.courses.africancitieslab.org/container/block-v1:EPFL+VA_MTU+2022+type@vertical+block@179320df1fe1464b8f0dded76714e0c5", "accept_language": "en-US,en;q=0.5", "event": "{\"GET\": {\"_\": [\"1673972484605\"]}, \"POST\": {}}", "time": "2023-01-17T16:27:43.520468+00:00", "event_type": "/xblock/block-v1:EPFL+VA_MTU+2022+type@html+block@aab222e4d92c4eb28e53c90171f89681/reorderable_container_child_preview", "event_source": "server", "page": null}
tutor_local-cms-1  | 2023-01-17 16:27:43,539 INFO 27 [openedx.core.djangoapps.cors_csrf.helpers] [user 6] [ip 194.230.158.119] helpers.py:64 - Origin 'https://studio.courses.africancitieslab.org' was not in `CORS_ORIGIN_WHITELIST`; full referer was 'https://studio.courses.africancitieslab.org/container/block-v1:EPFL+VA_MTU+2022+type@vertical+block@179320df1fe1464b8f0dded76714e0c5' and requested host was 'studio.courses.africancitieslab.org'; CORS_ORIGIN_ALLOW_ALL=False
tutor_local-cms-1  | [pid: 27|app: 0|req: 772/3673] 172.18.0.8 () {54 vars in 3842 bytes} [Tue Jan 17 16:27:43 2023] GET /xblock/block-v1:EPFL+VA_MTU+2022+type@html+block@aab222e4d92c4eb28e53c90171f89681/reorderable_container_child_preview?_=1673972484605 => generated 13563 bytes in 260 msecs (HTTP/1.1 200) 7 headers in 508 bytes (1 switches on core 0)
tutor_local-cms-1  | 2023-01-17 16:27:44,003 INFO 26 [tracking] [user 6] [ip 194.230.158.119] logger.py:41 - {"name": "/xblock/container/block-v1:EPFL+VA_MTU+2022+type@vertical+block@179320df1fe1464b8f0dded76714e0c5", "context": {"user_id": 6, "path": "/xblock/container/block-v1:EPFL+VA_MTU+2022+type@vertical+block@179320df1fe1464b8f0dded76714e0c5", "course_id": "", "org_id": "", "enterprise_uuid": ""}, "username": "martibosch", "session": "def6beb60d6f5969296176e4dea92346", "ip": "194.230.158.119", "agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0", "host": "studio.courses.africancitieslab.org", "referer": "https://studio.courses.africancitieslab.org/container/block-v1:EPFL+VA_MTU+2022+type@vertical+block@179320df1fe1464b8f0dded76714e0c5", "accept_language": "en-US,en;q=0.5", "event": "{\"GET\": {}, \"POST\": {}}", "time": "2023-01-17T16:27:44.003494+00:00", "event_type": "/xblock/container/block-v1:EPFL+VA_MTU+2022+type@vertical+block@179320df1fe1464b8f0dded76714e0c5", "event_source": "server", "page": null}
tutor_local-cms-1  | 2023-01-17 16:27:44,022 INFO 26 [openedx.core.djangoapps.cors_csrf.helpers] [user 6] [ip 194.230.158.119] helpers.py:64 - Origin 'https://studio.courses.africancitieslab.org' was not in `CORS_ORIGIN_WHITELIST`; full referer was 'https://studio.courses.africancitieslab.org/container/block-v1:EPFL+VA_MTU+2022+type@vertical+block@179320df1fe1464b8f0dded76714e0c5' and requested host was 'studio.courses.africancitieslab.org'; CORS_ORIGIN_ALLOW_ALL=False
tutor_local-cms-1  | [pid: 26|app: 0|req: 776/3674] 172.18.0.8 () {54 vars in 3797 bytes} [Tue Jan 17 16:27:43 2023] GET /xblock/container/block-v1:EPFL+VA_MTU+2022+type@vertical+block@179320df1fe1464b8f0dded76714e0c5 => generated 11202 bytes in 247 msecs (HTTP/1.1 200) 6 headers in 340 bytes (1 switches on core 0)

At the same time, I just found out that the issue does not happen when using the “Raw” editor (instead of the “Visual”).

Thank you again for your help. Best,
Martí

And there is no error in the javascript console?

In blocks with images there are JS GET errors of the form:

GET https://studio.courses.africancitieslab.org/static/Jerome_b-png.jpg

but these are no dealbreakers. One surprising pattern that I have found is this: if I add blank lines (they translate to <p></p>, the edits are saved :person_shrugging:

Peek 2023-01-17 18-32

Wait, does this happen only with the html blocks? What about videos? Could it be related to this fix? fix: backport fix for html component editing in studio by regisb · Pull Request #780 · overhangio/tutor · GitHub

It does indeed sound like it.

However, I have tried to fix it and so far I have seen no luck. Here is what I have done:

cp .local/share/tutor/env/build/openedx/Dockerfile Dockerfile.backup
pip uninstall tutor
pip install https://github.com/overhangio/tutor/archive/refs/heads/regisb/fix-tinymce-formatting.zip
tutor config save

Then, the following:

diff .local/share/tutor/env/build/openedx/Dockerfile Dockerfile.backup

outputs:

48,50d47
< # Fix TinyMCE editor for html components
< # https://github.com/openedx/edx-platform/pull/31500
< RUN curl -fsSL https://github.com/openedx/edx-platform/commit/5af42f8ba3.patch | git am

So far so good. Then, I have done

tutor local launch -I

which ran succesfully. However the problem persists.

Have I done everything correctly?

CC: @Anh_Vu_Nguy_n @pdpinch ?

Did you re-build the Docker image? If not, do:

tutor images build openedx
tutor local launch -I

yes, that did it. I rebuilt our custom image (because we use a custom theme) in a dedicated github workflow using tutor from the master branch (after you merged the PR): acl-tutor-do/build.yaml at fix-temp-tinymce · martibosch/acl-tutor-do · GitHub

Everything seems to be working smoothly now.

Immense thanks!
Martí

Awesome. I merged the pr and will make a new release later today.