Lost grade progress while updating to Palm

Site Operators Site Operations Help
1

I recently moved my server from AWS to Azure. I started from Lilac on AWS, and therefore started from Lilac on Azure too as I imported my data. The data import was done as below, but at the end when I checked in Palm, I see that no student grades were kept! :astonished: I would greatly appreciate if someone can point me to where the grades specifically are held, because the rest of the class data seems to have copied over.

Export MySQL data from Lilac:

docker exec tutor_local_mysql_1 /usr/bin/mysqldump -Q -d -uroot -p`echo $(tutor config printvalue MYSQL_ROOT_PASSWORD)` --default-character-set=utf8 --skip-set-charset openedx | sed 's/utf8/utf8mb4/gi' | sed 's/utf8mb4mb4/utf8mb4/gi' > openedx_schema.sql
docker exec tutor_local_mysql_1 /usr/bin/mysqldump -Q --insert-ignore -t -uroot -p`echo $(tutor config printvalue MYSQL_ROOT_PASSWORD)` --default-character-set=utf8 --skip-set-charset openedx > openedx_data.sql

(Note I’m doing some utf8mb4 fixing up here to deal with the prior emoji support breakage

Import MySQL data to Lilac:

tutor local stop
tutor local start mysql -d
tutor local run mysql bash
mysql -uroot -pPASSWORD --host "mysql" --port 3306
DROP DATABASE openedx;
create database openedx default charset utf8mb4 collate utf8mb4_general_ci;
exit
exit
cat ~/data/openedx_schema.sql | docker exec -i tutor_local_mysql_1 /usr/bin/mysql -u root --password=`echo $(tutor config printvalue MYSQL_ROOT_PASSWORD)` openedx
cat ~/data/openedx_data.sql | docker exec -i tutor_local_mysql_1 /usr/bin/mysql -u root --password=`echo $(tutor config printvalue MYSQL_ROOT_PASSWORD)` openedx

Export MongoDB data from Lilac:

tutor local exec mongodb mongodump --out=/data/db/dump.mongodb

Import MongoDB data to Lilac:

tutor local start mongodb -d
tutor local exec -it mongodb bash
mongorestore --drop /data/db/dump.mongodb
exit
2

@dave since I saw you mentioned on another thread with expertise on grades (and since you were super helpful in the past with MySQL emoji issues): Could you tell me how I could force the grades to be written to mysql on my old Lilac server and then export only the relevant MySQL table and import it into my new Palm instance?

It’s possible(likely?) my original Lilac didn’t have persistent grades enabled, which is why the mysql export/import didn’t carry it along (whereas it did for a separate Nutmeg server)

3

@ABC: My understanding is that you can generate the grades directly in Palm (i.e. you don’t have to try to generate in Lilac and port over if you already have the rest of the stuff moved). The management command is described here:

https://openedx.atlassian.net/wiki/spaces/AC/pages/755171487/Migrating+to+Persistent+Grading

Just to be clear: I was aware of this work when it was being built, but I didn’t write this part of the code and I don’t have personal operational experience with running this migration. I’ve been warned that it will enqueue many celery tasks though.

The compute_grades management command takes a list of courses as an argument (--courses), so you might want to start with one of your smallest courses and see how long it takes to run.

@gabor or @nsprenkle might know more.

Good luck. Please let me know how it goes for you.

4

The issue is that when I create a grade report for students, they all come up as 0% done. So it’s not exactly a question of “calculate on the fly” vs. “calculate persistently” (which I believe Palm should be doing by default anyway?), it seems that the core data somehow is missing despite the full mysql/mongo db backups and moves over to the new server. Which is why I’m looking to figure out and pull over just the grades and then see if grade reports stop showing up as 0% for everyone.

@gabor or @nsprenkle any ideas?

5

The tables for grading should all start with grades_. The one that has persistent grades should be grades_persistentsubsectiongrade. Though all the tables in this models file are important.

When an individual student looks at a specific problem in the courseware that they’ve completed, do they see the score they earned for it? If that’s true, then at least the user state and raw scores moved over.

If that’s the case, and the grades_ tables are empty for both of your installs, and running the compute_grades management command for a course doesn’t fill out this table for your students–then try forcing a course publish by making some trivial change to that course, and waiting for all the post-publish processes to finish. (It might be that the grading system is missing some content-related data that it normally generates after publishing a course.) Then try running compute_grades for that course again, and try downloading a grades report after it’s completed.

Hope this helps. Good luck.

6

Unfortunately, I’m not familiar with this particular part of the courses. I was working with grade reports and completions, but that was a while ago. Maybe @pkulkark or @braden have some hints.

Although I’m only in the dark, but have you checked if all the xblocks, plugins, etc., are installed for the new instance that you have in the old one? I know some particular plugins can create other database tables too, like the completion aggregator.

Also, have you checked you have the same number of database tables compared to the old one?

7

Thanks @dave and @gabor. As an initial result, I found that the grades_persistentcoursegrade and grades_persistentsubsectiongrade mysql tables were empty on my previous production instance. So it seems like I had never enabled persistent grades for lilac. I’m doing that now, and manually re-calculating all the grades

I have three follow-up questions then: (for @pkulkark and @braden if they’re more likely to understand how grading works)

  1. Once I get all these grades, what are all the db tables I should export in order to import on the new server? Just all the grades_* ones?

  2. Is there any way at import time to avoid destroying the 300 or so users’ grade data which has been updated on the new production server while I’ve been trying to figure this out?

  3. Where do the grades’ source information come from, to be calculated and put in the db? I.e. if it’s ultimately just coming from somewhere else, perhaps I should just be checking that somewhere else and whether it got copied over between servers, so that maybe the data could be re-computed properly? Because if that source data isn’t there, won’t future grade re-computes still be wrong if they’re currently computing zero grades on the new server?

8

Hmm… and once a manual report from the admin UI completed, unfortunately it was named as grade_report_err and had only 7 entries like this (for a 1000-person class):

(1062, "Duplicate entry 'COURSEID-STUDENTID' for key 'grades_persistentsubsect_course_id_user_id_usage__42820224_uniq'")

Edit: OK, actually it generated a normal report + this error report (I just didn’t notice the normal report below the error report.)

9

I really don’t know much about grading, sorry.

10

Sorry if I sound like a broken record, but I have to ask again: Have you tried to run the backfill command compute_grades as described here, on your new instance?

https://openedx.atlassian.net/wiki/spaces/AC/pages/755171487/Migrating+to+Persistent+Grading#The-Persistent-Grades-Backfill

My concern with generating stuff in your old instance and copying the data across is that it might have implicit assumptions about other data that should have been generated from the publish, e.g. links back to S3 stored files.

If you do something like this from the LMS:

python manage.py lms compute_grades -v1 --settings=production --courses {YOUR COURSE KEY}

Does that fill in values for those tables as you would expect? Or does it cause an error? After that command is run for a specific course, does downloading a grade report look like you would expect it to?

Where do the grades’ source information come from, to be calculated and put in the db? I.e. if it’s ultimately just coming from somewhere else, perhaps I should just be checking that somewhere else and whether it got copied over between servers, so that maybe the data could be re-computed properly? Because if that source data isn’t there, won’t future grade re-computes still be wrong if they’re currently computing zero grades on the new server?

There are at least a couple of inputs into this:

  • The Course Blocks API, which is where grades gets its “what does the content look like for this particular user”. The ultimate source for that data is in the ModuleStore (i.e. MongoDB), but this API builds its own representation of that data after every course publish. If you’re moving the data across instances, you should probably run the simulate_publish command from a Studio instance, so you can trigger it across all courses without having to edit each one through the UI.
  • The student’s state in courseware_studentmodule and the submissions_score table.

Running the compute_grades management command is supposed to then read from those and materialize all the students grade information into the grades_* tables so that future download tasks can read from that. So when you ask for a grade report, it should be reading purely from those tables and will make the assumption that if someone doesn’t have an entry there, it means they never did the content. (Every time a user submits an answer, it also triggers a celery task that updates those grades_ tables, particularly grades_persistentsubsectiongrade.) If the compute_grades command doesn’t fill in those tables, does it give some kind of error in its logs?

:slightly_frowning_face: Okay, that sounds like a race condition tried to shove duplicate data into that table for a particular (user, subsection) combination. I don’t know why it’s doing that.

1 Like
11

Thanks for re-raising that. I got too focused on starting from my previous instance, because initially I couldn’t imagine that if I ran a grade from within the instructor UI, and it came up with 0% grades, that running something from the CLI would be any different. However, at preliminary test for the single largest class (tutor local run lms ./manage.py lms compute_grades --courses <COURSEID>) seems to have worked! So now I’m running tutor local run lms ./manage.py lms compute_grades --all_courses and will report back tomorrow if it worked for a class where I didn’t specify it explicitly. Thanks!!! :bangbang::bangbang:

1 Like
12

I can confirm now that the grades all recalcualted successfully. Thanks again @dave, you’re a life-saver!

13

:grin: I’m glad it worked out for you. Though in this case, all credit should go to the team that worked on the grades project (I know @nimisha and @iloveagent57 worked on that–I don’t remember if there were others). Grades are one of the trickiest parts of the platform, and they spent a lot of time taking great pains to document all the stuff they were doing and why.

1 Like
14

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.