MySQL error after upgrade from olive to palm

FatemeKhodayari · September 27, 2023, 10:37am

After upgrading from Olive (tutor 15.3.7) to Palm (tutor 16.1.1) some of users who have activities in discussions, exams, assignments, etc get 500 error on dashboard. The log for the said error is as follows:

ERROR 35 [openedx.core.djangoapps.content.course_overviews.models] models.py:339 - Saving CourseOverview for course course-v1:sample-organization-33001+T1-1402 failed with unexpected exception!
Traceback (most recent call last):
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/connections.py", line 254, in query
    _mysql.connection.query(self, query)
MySQLdb.OperationalError: (1366, "Incorrect string value: '\\xF0\\x9F\\x93\\xA3\\xD8\\xAA...' for column 'short_description' at row 1")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/openedx/edx-platform/./openedx/core/djangoapps/content/course_overviews/models.py", line 305, in load_from_module_store
    course_overview.save()
  File "/openedx/venv/lib/python3.8/site-packages/model_utils/models.py", line 38, in save
    super().save(*args, **kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/base.py", line 739, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/base.py", line 776, in save_base
    updated = self._save_table(
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/base.py", line 858, in _save_table
    updated = self._do_update(base_qs, using, pk_val, values, update_fields,
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/base.py", line 912, in _do_update
    return filtered._update(values) > 0
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/query.py", line 802, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1559, in execute_sql
    cursor = super().execute_sql(result_type)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
    cursor.execute(sql, params)
  File "/openedx/venv/lib/python3.8/site-packages/sentry_sdk/integrations/django/__init__.py", line 616, in execute
    return real_execute(self, sql, params)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/connections.py", line 254, in query
    _mysql.connection.query(self, query)
django.db.utils.OperationalError: (1366, "Incorrect string value: '\\xF0\\x9F\\x93\\xA3\\xD8\\xAA...' for column 'short_description' at row 1")
2023-09-27 08:48:28,131 ERROR 35 [root] [user None] [ip None] signals.py:22 - Uncaught exception from None
Traceback (most recent call last):
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/connections.py", line 254, in query
    _mysql.connection.query(self, query)
MySQLdb.OperationalError: (1366, "Incorrect string value: '\\xF0\\x9F\\x93\\xA3\\xD8\\xAA...' for column 'short_description' at row 1")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/sentry_sdk/integrations/django/views.py", line 84, in sentry_wrapped_callback
    return callback(request, *args, **kwargs)
  File "/opt/pyenv/versions/3.8.15/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/openedx/venv/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/openedx/edx-platform/./lms/djangoapps/course_home_api/course_metadata/views.py", line 80, in get
    course = course_detail(request, request.user.username, course_key)
  File "/openedx/edx-platform/./lms/djangoapps/course_api/api.py", line 76, in course_detail
    overview = get_course_overview_with_access(
  File "/openedx/edx-platform/./lms/djangoapps/courseware/courses.py", line 128, in get_course_overview_with_access
    course_overview = CourseOverview.get_from_id(course_key)
  File "/openedx/edx-platform/./openedx/core/lib/cache_utils.py", line 74, in decorator
    result = wrapped(*args, **kwargs)
  File "/openedx/edx-platform/./openedx/core/djangoapps/content/course_overviews/models.py", line 407, in get_from_id
    course_overview = cls.load_from_module_store(course_id)
  File "/openedx/edx-platform/./openedx/core/djangoapps/content/course_overviews/models.py", line 305, in load_from_module_store
    course_overview.save()
  File "/openedx/venv/lib/python3.8/site-packages/model_utils/models.py", line 38, in save
    super().save(*args, **kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/base.py", line 739, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/base.py", line 776, in save_base
    updated = self._save_table(
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/base.py", line 858, in _save_table
    updated = self._do_update(base_qs, using, pk_val, values, update_fields,
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/base.py", line 912, in _do_update
    return filtered._update(values) > 0
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/query.py", line 802, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1559, in execute_sql
    cursor = super().execute_sql(result_type)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
    cursor.execute(sql, params)
  File "/openedx/venv/lib/python3.8/site-packages/sentry_sdk/integrations/django/__init__.py", line 616, in execute
    return real_execute(self, sql, params)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/connections.py", line 254, in query
    _mysql.connection.query(self, query)
django.db.utils.OperationalError: (1366, "Incorrect string value: '\\xF0\\x9F\\x93\\xA3\\xD8\\xAA...' for column 'short_description' at row 1")
[pid: 7|app: 0|req: 957/2492] 172.18.0.4 () {60 vars in 3613 bytes} [Wed Sep 27 08:48:28 2023] GET /api/notifications/count/ => generated 10117 bytes in 133 msecs (HTTP/1.1 404) 10 headers in 939 bytes (1 switches on core 0)
2023-09-27 08:48:28,208 ERROR 35 [django.request] [user 40] [ip 37.98.105.95] log.py:224 - Internal Server Error: /api/course_home/course_metadata/course-v1:Sharifstar+IOI-33001+T1-1402
Traceback (most recent call last):
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/connections.py", line 254, in query
    _mysql.connection.query(self, query)
MySQLdb.OperationalError: (1366, "Incorrect string value: '\\xF0\\x9F\\x93\\xA3\\xD8\\xAA...' for column 'short_description' at row 1")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/sentry_sdk/integrations/django/views.py", line 84, in sentry_wrapped_callback
    return callback(request, *args, **kwargs)
  File "/opt/pyenv/versions/3.8.15/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/openedx/venv/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/openedx/edx-platform/./lms/djangoapps/course_home_api/course_metadata/views.py", line 80, in get
    course = course_detail(request, request.user.username, course_key)
  File "/openedx/edx-platform/./lms/djangoapps/course_api/api.py", line 76, in course_detail
    overview = get_course_overview_with_access(
  File "/openedx/edx-platform/./lms/djangoapps/courseware/courses.py", line 128, in get_course_overview_with_access
    course_overview = CourseOverview.get_from_id(course_key)
  File "/openedx/edx-platform/./openedx/core/lib/cache_utils.py", line 74, in decorator
    result = wrapped(*args, **kwargs)
  File "/openedx/edx-platform/./openedx/core/djangoapps/content/course_overviews/models.py", line 407, in get_from_id
    course_overview = cls.load_from_module_store(course_id)
  File "/openedx/edx-platform/./openedx/core/djangoapps/content/course_overviews/models.py", line 305, in load_from_module_store
    course_overview.save()
  File "/openedx/venv/lib/python3.8/site-packages/model_utils/models.py", line 38, in save
    super().save(*args, **kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/base.py", line 739, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/base.py", line 776, in save_base
    updated = self._save_table(
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/base.py", line 858, in _save_table
    updated = self._do_update(base_qs, using, pk_val, values, update_fields,
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/base.py", line 912, in _do_update
    return filtered._update(values) > 0
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/query.py", line 802, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1559, in execute_sql
    cursor = super().execute_sql(result_type)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
    cursor.execute(sql, params)
  File "/openedx/venv/lib/python3.8/site-packages/sentry_sdk/integrations/django/__init__.py", line 616, in execute
    return real_execute(self, sql, params)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/connections.py", line 254, in query
    _mysql.connection.query(self, query)
django.db.utils.OperationalError: (1366, "Incorrect string value: '\\xF0\\x9F\\x93\\xA3\\xD8\\xAA...' for column 'short_description' at row 1")

Does any one know how I should fix this? Should I restore from a backup and redo the upgrade?

regis · October 2, 2023, 5:16am

Hi @FatemeKhodayari,
What are the logs coming from the mysql container? tutor local logs mysql

FatemeKhodayari · October 2, 2023, 6:09am

Hi @regis and thanks for the reply. Actually I have no specific error in mysql logs. Here is all the log I get from mysql:

[Warning] [MY-013360] [Server] Plugin mysql_native_password reported: ''mysql_native_password' is deprecated and will be removed in a future release. Please use caching_sha2_password instead'

regis · October 2, 2023, 6:30am

The string error seems to come from the short_description column. Please have a look at this data with:

tutor local do sqlshell openedx
select id, display_name, short_description from course_overviews_courseoverview;

FatemeKhodayari · October 2, 2023, 7:10am

Two of the courses have value in their short_description column but those two courses are archived. I tried removing (unenroll) the user from these courses but she is still getting 500 on dashboard.

regis · October 2, 2023, 7:29am

Is one of these two courses course-v1:sample-organization-33001+T1-1402?

FatemeKhodayari · October 2, 2023, 7:40am

Nope. These are the course ids that are displayed after the said command.

course-v1:Sharifstar+C-22003+AT2-1402
course-v1:Sharifstar+cdc+2ws
course-v1:Sharifstar+cdc+test-rerun
course-v1:Sharifstar+CDT-22001+AT2-1402
course-v1:Sharifstar+CDT-22001+ST1-1402 course-v1:Sharifstar+CRE-11002+AT2-1402 course-v1:Sharifstar+CRE-11002+ST1-1402 course-v1:Sharifstar+CS101+2023_T2 course-v1:Sharifstar+DTM-11004+AT2-1402
course-v1:Sharifstar+DTM-11004+ST1-1402
course-v1:Sharifstar+ESD-22005+AT2-1402
course-v1:Sharifstar+GPS-11001+AT2-1402
course-v1:Sharifstar+GPS-11001+ST1-1402 course-v1:Sharifstar+IO-11006+AT2-1402
course-v1:Sharifstar+IOI-33001+ST1-1402
course-v1:Sharifstar+IOI-33001+T1-1402
course-v1:Sharifstar+LST-33006+AT2-1402
course-v1:Sharifstar+Marketing-33007+AT1-1402
course-v1:Sharifstar+Marketing-33007+AT2-1402
course-v1:Sharifstar+PFE-22006+AT2-1402
course-v1:Sharifstar+PFE-22007+AT2-1402
course-v1:Sharifstar+PPTM-22002+AT2-1402
course-v1:Sharifstar+SET-11005+AT2-1402
course-v1:Sharifstar+SHT-11003+AT2-1402
course-v1:Sharifstar+SHT-11003+ST1-1402 course-v1:Sharifstar+WAM-22004+ST1-1402
course-v1:SharifStars+C01+summer01
course-v1:smartstars+123345+T1_2023
course-v1:test-orginization+56787854+lunch-course-field

regis · October 2, 2023, 8:32am

I suggest you try to identify the course which includes the “incorrect” short description. Maybe override the short description with update short_description = null from course_overviews_courseoverview;?

I suspect that all courses are being updated when the user loads the course index for the first time, and the fact that courses are archived does not help.

FatemeKhodayari · October 2, 2023, 10:29am

Found what the problem was. In one of the short descriptions, some emojis were used. My guess is that these emojis were causing the error. I removed the short description from studio, saved and added them again and the 500 error was gone. But what if we had more data? Do you know the right way for fixing this issue?

regis · October 2, 2023, 11:53am

What were the emojis?

FatemeKhodayari · October 2, 2023, 11:58am

The emojis were and

regis · October 2, 2023, 1:21pm

I managed to reproduce the issue, it’s now tracked here: Emojis in course description cause failures after Olive -> Palm upgrade · Issue #910 · overhangio/tutor · GitHub
Thanks for the thorough feedback!

FatemeKhodayari · October 2, 2023, 1:28pm

Thanks for the follow up. In the issue you have mentioned: “…then the LMS will fail entirely.” Just wanted to add that the dashboard gets 500 error but the user is still able to go to the discover courses tab from header, enroll and view course details in the learning MFE, etc. It’s just the dashboard that fails.

system · December 31, 2023, 1:29pm

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Tutor local launch on pi5 fails with MySQLdb.OperationalError: (2013, 'Lost connection to MySQL server during query') Tutor Help tutor	4	68	July 10, 2024
Immigration error when upgrading Quince from Palm Site Operators tutor	2	38	October 28, 2024
500 Server Error when deleting a user from the amin console after upgrading from Nutmeg to Palm Site Operators devstack , tutor	3	212	September 25, 2023
Upgrade to Maple fails due to missing table "openedx.agreements_integritysignature" Tutor Help tutor , maple	3	417	May 3, 2023
Mysql initialisation error when upgrading tutor Tutor Help	0	70	August 15, 2024

MySQL error after upgrade from olive to palm

Related topics