Hello everyone. We’re encountering problems with translations. Sometimes, the LMS gets to a point where it’s going into infinite recursion on translation.fallback when translating a message, which causes a 500 error page, which goes into infinite recursion translating the messages in the error page template.
It’s been happening very rarely for a couple of years at least, and we’ve been unable to reproduce the problem ourselves. It just happens for one request, the LMS crashes after a while (actually, it seems to get stopped by uWSGI, which looks like a graceful termination, and thus doesn’t get automatically restarted by Docker if the restart condition is set to on-failure,) and then all is fine, until next time.
Here’s a stack trace that seems to indicate the problem comes from XBlock translations, but we’re not sure, since it also happens on other pages, like `/courses`:
Traceback (most recent call last):
File "/openedx/venv/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
response = get_response(request)
^^^^^^^^^^^^^^^^^^^^^
File "/openedx/venv/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/openedx/venv/lib/python3.11/site-packages/django/views/decorators/http.py", line 43, in inner
return func(request, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/openedx/edx-platform/common/djangoapps/util/views.py", line 63, in inner
response = view_func(request, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/openedx/venv/lib/python3.11/site-packages/django/views/decorators/clickjacking.py", line 58, in wrapper_view
resp = view_func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/openedx/venv/lib/python3.11/site-packages/django/utils/decorators.py", line 134, in _wrapper_view
response = view_func(request, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/openedx/edx-platform/lms/djangoapps/courseware/views/views.py", line 1695, in render_xblock
fragment = block.render(requested_view, context=student_view_context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/openedx/venv/lib/python3.11/site-packages/xblock/core.py", line 818, in render
return self.runtime.render(self, view, context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/openedx/edx-platform/xmodule/x_module.py", line 994, in render
return super().render(block, view_name, context=context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/openedx/venv/lib/python3.11/site-packages/xblock/runtime.py", line 823, in render
frag = view_fn(context)
^^^^^^^^^^^^^^^^
File "/openedx/edx-platform/xmodule/vertical_block.py", line 203, in student_view
return self._student_or_public_view(context, STUDENT_VIEW)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/openedx/edx-platform/xmodule/vertical_block.py", line 130, in _student_or_public_view
rendered_child = child.render(view, child_block_context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/openedx/venv/lib/python3.11/site-packages/xblock/core.py", line 818, in render
return self.runtime.render(self, view, context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/openedx/edx-platform/xmodule/x_module.py", line 994, in render
return super().render(block, view_name, context=context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/openedx/venv/lib/python3.11/site-packages/xblock/runtime.py", line 823, in render
frag = view_fn(context)
^^^^^^^^^^^^^^^^
File "/openedx/venv/lib/python3.11/site-packages/done/done.py", line 81, in student_view
frag.add_content(resource_loader.render_django_template(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/openedx/venv/lib/python3.11/site-packages/xblock/utils/resources.py", line 48, in render_django_template
rendered = template.render(Context(context))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/openedx/venv/lib/python3.11/site-packages/django/template/base.py", line 175, in render
return self._render(context)
^^^^^^^^^^^^^^^^^^^^^
File "/openedx/venv/lib/python3.11/site-packages/django/template/base.py", line 167, in _render
return self.nodelist.render(context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/openedx/venv/lib/python3.11/site-packages/django/template/base.py", line 1005, in render
return SafeString("".join([node.render_annotated(context) for node in self]))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/openedx/venv/lib/python3.11/site-packages/django/template/base.py", line 1005, in <listcomp>
return SafeString("".join([node.render_annotated(context) for node in self]))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/openedx/venv/lib/python3.11/site-packages/django/template/base.py", line 966, in render_annotated
return self.render(context)
^^^^^^^^^^^^^^^^^^^^
File "/openedx/venv/lib/python3.11/site-packages/xblock/utils/templatetags/i18n.py", line 54, in render
with self.merge_translation(context):
File "/opt/pyenv/versions/3.11.8/lib/python3.11/contextlib.py", line 137, in __enter__
return next(self.gen)
^^^^^^^^^^^^^^
File "/openedx/venv/lib/python3.11/site-packages/xblock/utils/templatetags/i18n.py", line 40, in merge_translation
translation.merge(i18n_service)
File "/openedx/venv/lib/python3.11/site-packages/django/utils/translation/trans_real.py", line 264, in merge
self.add_fallback(other._fallback)
File "/opt/pyenv/versions/3.11.8/lib/python3.11/gettext.py", line 279, in add_fallback
self._fallback.add_fallback(fallback)
File "/opt/pyenv/versions/3.11.8/lib/python3.11/gettext.py", line 279, in add_fallback
self._fallback.add_fallback(fallback)
File "/opt/pyenv/versions/3.11.8/lib/python3.11/gettext.py", line 279, in add_fallback
self._fallback.add_fallback(fallback)
[Previous line repeated 831 more times]
RecursionError: maximum recursion depth exceeded
My theory is that something at runtime causes fallback languages to be added to the translation service, causing one of:
-
A language gets configured as its own fallback (one-element recursion loop);
-
A language’s fallback is configured to one of its parent languages (N-element recursion loop)
-
Fallbacks keep being added without a loop, but it eventually results in a chain of 1000 elements and triggers the recursion limit error.
Potentially, this poisons the translation service, which causes a recursion error to happen when translating another page, which doesn’t have an XBlock.
Has this happened to anyone else?
We’ve been investigating this for a week, and finally today I manage to find a reproduction: when the language is set to German (de-de,) loading a unit that uses the done/Completion XBlock causes the infinite recursion bug, at least in Redwood and Sumac. I reproduced it with a unit that contains only this XBlock, and it happens every single time. If I remove the XBlock, it stops (after restarting the LMS to stop the recursion, of course.) It doesn’t seem to happen with French or Italian.
Disabling German in the DarkLang config is not sufficient, because while it will hide it from the language menu, a user that already has their cookie set to de-de will still trigger the bug. Removing German from settings.LANGUAGES will prevent the bug from happening. Another way to prevent the bug it is to remove the four translated strings from /openedx/venv/lib/python3.11/site-packages/done/templates/done.html.
What I noticed is that out of 6 XBlocks I looked at (DoneXBlock, xblock-free-text-response, xblock-qualtrics-survey, xblock-google-drive, xblock-image-explorer, xblock-drag-and-drop-v2,) DoneXBlock is the only one that doesn’t have a symlink called translations that points to conf/locale. I have to idea if it has anything to do with the matter.
Does anybody have an idea of what’s happening? At the moment, I’m thinking of replacing that XBlock with a fork that doesn’t translate its strings, but I’d be happy with a cleaner solution. Thanks in advance for your insights.