Load missing lesson information in mysql table

asappa · May 6, 2023, 9:05pm

Hi Team,

Our team is using nutmeg version of openedx. When we import a new course or are exporting a course and then re-importing it in CMS, we are seeing the new courses’s lesson information like usage_key, title etc in learning_sequences_learningsequence table in mysql. But we are not seeing records for some of the older//historical courses in studio and LMS in learning_sequences_learningsequence table.

Could anyone please let me know if there is a way to update or load the lesson information for existing courses in learning_sequences_learningsequence table?

dave · May 8, 2023, 3:11am

It updates when you publish any changes to a course, but you can run a management command to backfill:

python ./manage.py cms backfill_course_outlines

(Please note that it’s a cms/studio management command, not LMS.)

asappa · May 8, 2023, 4:02pm

Thanks Dave! We are using tutor and tried the below commands to backfill

tutor local run cms ./manage.py cms backfill_course_outlines
tutor local run cms ./manage.py cms backfill_course_outlines --force

We are still not able to see records in learning_sequences_learningsequence table for one of the courses in CMS. Could you please suggest anything in this regard?

dave · May 8, 2023, 4:42pm

When you look at your Django admin for “Learning Sequences an Outlines” (/admin/learning_sequences/coursecontext/), do you see the course listed there? If so, click on it and see if there are errors recorded there.

If not, did the management command itself output any errors? You can re-create a course outline for a single course by using the update_course_outline management command and passing the course key as an argument–and seeing what errors show up.

asappa · May 8, 2023, 6:22pm

Hi Dave,

On the admin screen, it shows 0 errors, 0 sequences and 0 sections but we do have sections and sequences for that course. When we run the update_course_outline, we are seeing the below error:

tutor local run cms ./manage.py cms update_course_outline course-v1:MyUx+D312+2021_T3
2023-05-08 18:12:09,029 INFO 1 [openedx.core.djangoapps.content.learning_sequences.api.outlines] [user None] [ip None] outlines.py:389 - Replacing CourseOutline for course-v1:MyUx+D312+2021_T3 (version 64493e308ca193a4eb0048c0, 70 sequences)
2023-05-08 18:12:09,042 INFO 1 [openedx.core.djangoapps.content.learning_sequences.api.outlines] [user None] [ip None] outlines.py:435 - Found CourseContext for course-v1:MyUx+D312+2021_T3, updating...
Traceback (most recent call last):
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/query.py", line 581, in get_or_create
    return self.get(**kwargs), False
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/query.py", line 435, in get
    raise self.model.DoesNotExist(
openedx.core.djangoapps.content.learning_sequences.models.DoesNotExist: LearningSequence matching query does not exist.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/connections.py", line 254, in query
    _mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (1366, "Incorrect string value: '\\xE2\\x80\\xAFSke...' for column 'title' at row 1")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "./manage.py", line 106, in <module>
    execute_from_command_line([sys.argv[0]] + django_args)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/openedx/edx-platform/cms/djangoapps/contentstore/management/commands/update_course_outline.py", line 27, in handle
    update_outline_from_modulestore(course_key)
  File "/openedx/edx-platform/cms/djangoapps/contentstore/outlines.py", line 385, in update_outline_from_modulestore
    replace_course_outline(course_outline_data, content_errors=content_errors)
  File "/opt/pyenv/versions/3.8.12/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/openedx/edx-platform/openedx/core/djangoapps/content/learning_sequences/api/outlines.py", line 406, in replace_course_outline
    _update_sequences(course_outline, course_context)
  File "/openedx/edx-platform/openedx/core/djangoapps/content/learning_sequences/api/outlines.py", line 474, in _update_sequences
    LearningSequence.objects.update_or_create(
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/query.py", line 608, in update_or_create
    obj, created = self.select_for_update().get_or_create(defaults, **kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/query.py", line 588, in get_or_create
    return self.create(**params), True
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/query.py", line 453, in create
    obj.save(force_insert=True, using=self.db)
  File "/openedx/venv/lib/python3.8/site-packages/model_utils/models.py", line 38, in save
    super().save(*args, **kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/base.py", line 739, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/base.py", line 776, in save_base
    updated = self._save_table(
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/base.py", line 881, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/base.py", line 919, in _do_insert
    return manager._insert(
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/query.py", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
    cursor.execute(sql, params)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/connections.py", line 254, in query
    _mysql.connection.query(self, query)
django.db.utils.OperationalError: (1366, "Incorrect string value: '\\xE2\\x80\\xAFSke...' for column 'title' at row 1")
ERROR: 1
Error: Command failed with status 1: docker-compose -f /home/ec2-user/.local/share/tutor/env/local/docker-compose.yml -f /home/ec2-user/.local/share/tutor/env/local/docker-compose.prod.yml -f /home/ec2-user/.local/share/tutor/env/local/docker-compose.tmp.yml --project-name tutor_local run --rm cms ./manage.py cms update_course_outline course-v1:MyUx+D312+2021_T3

dave · May 8, 2023, 6:48pm

That sounds like a unicode encoding error. What is the character encoding for that table?

asappa · May 8, 2023, 8:46pm

latin1 is the character encoding for that table.

dave · May 8, 2023, 10:52pm

Yeah, that would do it. You ideally want utf8mb4, though some databases are stuck on utf8 and that will still work in this case (but would break for emojis).

dave · May 9, 2023, 12:29am

This is true for pretty much all tables that Open edX services create, FWIW. You’ll run into a lot of issues around storing anything that isn’t English unless your tables have a Unicode encoding (utf8mb4 or utf8). I’m not sure how it got configured that way for your install, but you should convert it as soon as you can.

dave · May 17, 2023, 12:46pm

@asappa: BTW, how did your database get set up? I ask because Tutor should be setting the defaults to utf8 for the charset with utf8_general_ci collation (which is itself a bit problematic, but vastly better than latin1). I’m trying to figure out how much work it would be to get uniformity in database configurations, and so I’m curious about how your database got to its current state.

Thank you.

CourtneyMaybe · June 15, 2023, 10:12pm

Hey Dave!

We’ve had this instance of OpenEdX for some time. It looks like latin1 was the default encoding for MySQL for the version we are running (5.7).

Do you have any best practices or suggestions for the migration to utf8mb4?

dave · June 16, 2023, 2:25pm

@CourtneyMaybe: I don’t have any suggestions personally, since I haven’t done it for a live site myself. But there is a thread where folks walk through it to fix another issue: Nutmeg upgrade: outlines.py: "openedx.core.djangoapps.content.learning_sequences.models.LearningContext.DoesNotExist: LearningContext matching query does not exist"

system · September 14, 2023, 2:26pm

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Course outline not loading in LMS (solved) Architecture how-to , palm	1	41	August 13, 2024
Nutmeg upgrade: outlines.py: "openedx.core.djangoapps.content.learning_sequences.models.LearningContext.DoesNotExist: LearningContext matching query does not exist" Site Operations Help	30	2892	November 5, 2022
Nutmeg Feature Round-up Blog	4	491	July 12, 2022
ERROR/ForkPoolWorker-5 "Could not create course outline for course" Site Operations Help	2	129	May 21, 2024
Problem refreshing content and creating course in Quince Site Operations Help how-to , lms , quince	3	158	June 25, 2024

Load missing lesson information in mysql table

Related topics