# Lost grade progress while updating to Palm

**URL:** <https://discuss.openedx.org/t/lost-grade-progress-while-updating-to-palm/13311>\
**Category:** Site Operations Help\
**Created:** [July 2, 2024, 12:47pm UTC](https://discuss.openedx.org/t/lost-grade-progress-while-updating-to-palm/13311 "2024-07-02T12:47:13Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![ABC](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/abc/32/4946_2.png) [@ABC](https://discuss.openedx.org/u/ABC)\
**Post date:** [July 2, 2024, 12:47pm UTC](https://discuss.openedx.org/t/lost-grade-progress-while-updating-to-palm/13311/1 "2024-07-02T12:47:13Z")

</div>

I recently moved my server from AWS to Azure. I started from Lilac on AWS, and therefore started from Lilac on Azure too as I imported my data. The data import was done as below, but at the end when I checked in Palm, I see that **no student grades were kept**! 😲 **I would greatly appreciate if someone can point me to where the grades specifically are held** , because the rest of the class data seems to have copied over.

Export MySQL data from Lilac:

```auto
docker exec tutor_local_mysql_1 /usr/bin/mysqldump -Q -d -uroot -p`echo $(tutor config printvalue MYSQL_ROOT_PASSWORD)` --default-character-set=utf8 --skip-set-charset openedx | sed 's/utf8/utf8mb4/gi' | sed 's/utf8mb4mb4/utf8mb4/gi' > openedx_schema.sql
docker exec tutor_local_mysql_1 /usr/bin/mysqldump -Q --insert-ignore -t -uroot -p`echo $(tutor config printvalue MYSQL_ROOT_PASSWORD)` --default-character-set=utf8 --skip-set-charset openedx > openedx_data.sql

```

(Note I’m doing some utf8mb4 fixing up here to deal with the prior [emoji support breakage](https://discuss.openedx.org/t/emoji-in-subsection-title-not-supported-since-nutmeg/7856)

Import MySQL data to Lilac:

```auto
tutor local stop
tutor local start mysql -d
tutor local run mysql bash
mysql -uroot -pPASSWORD --host "mysql" --port 3306
DROP DATABASE openedx;
create database openedx default charset utf8mb4 collate utf8mb4_general_ci;
exit
exit
cat ~/data/openedx_schema.sql | docker exec -i tutor_local_mysql_1 /usr/bin/mysql -u root --password=`echo $(tutor config printvalue MYSQL_ROOT_PASSWORD)` openedx
cat ~/data/openedx_data.sql | docker exec -i tutor_local_mysql_1 /usr/bin/mysql -u root --password=`echo $(tutor config printvalue MYSQL_ROOT_PASSWORD)` openedx

```

Export MongoDB data from Lilac:

```auto
tutor local exec mongodb mongodump --out=/data/db/dump.mongodb

```

Import MongoDB data to Lilac:

```auto
tutor local start mongodb -d
tutor local exec -it mongodb bash
mongorestore --drop /data/db/dump.mongodb
exit

```

---

<div class="post-metadata">

**Author:** ![ABC](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/abc/32/4946_2.png) [@ABC](https://discuss.openedx.org/u/ABC)\
**Post date:** [July 3, 2024, 11:39am UTC](https://discuss.openedx.org/t/lost-grade-progress-while-updating-to-palm/13311/2 "2024-07-03T11:39:53Z")

</div>

@dave since I saw you mentioned on [another thread](https://discuss.openedx.org/t/need-information-about-persistent-grades/1412) with expertise on grades (and since you were super helpful in the past with MySQL emoji issues): Could you tell me how I could force the grades to be written to mysql on my old Lilac server and then export only the relevant MySQL table and import it into my new Palm instance?

It’s possible(likely?) my original Lilac didn’t have persistent grades enabled, which is why the mysql export/import didn’t carry it along (whereas it did for a separate Nutmeg server)

---

<div class="post-metadata">

**Author:** ![dave](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/dave/32/263_2.png) [@dave](https://discuss.openedx.org/u/dave)\
**Post date:** [July 4, 2024, 9:42pm UTC](https://discuss.openedx.org/t/lost-grade-progress-while-updating-to-palm/13311/3 "2024-07-04T21:42:37Z")

</div>

@ABC: My understanding is that you can generate the grades directly in Palm (i.e. you don’t have to try to generate in Lilac and port over if you already have the rest of the stuff moved). The management command is described here:

[https://openedx.atlassian.net/wiki/spaces/AC/pages/755171487/Migrating+to+Persistent+Grading](https://openedx.atlassian.net/wiki/spaces/AC/pages/755171487/Migrating+to+Persistent+Grading)

Just to be clear: I was aware of this work when it was being built, but I didn’t write this part of the code and I don’t have personal operational experience with running this migration. I’ve been warned that it will enqueue many celery tasks though.

The `compute_grades` management command takes a list of courses as an argument (`--courses`), so you might want to start with one of your smallest courses and see how long it takes to run.

@gabor or @nsprenkle might know more.

Good luck. Please let me know how it goes for you.

---

<div class="post-metadata">

**Author:** ![ABC](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/abc/32/4946_2.png) [@ABC](https://discuss.openedx.org/u/ABC)\
**Post date:** [July 7, 2024, 12:06pm UTC](https://discuss.openedx.org/t/lost-grade-progress-while-updating-to-palm/13311/4 "2024-07-07T12:06:03Z")

</div>

The issue is that when I create a grade report for students, they all come up as 0% done. So it’s not exactly a question of “calculate on the fly” vs. “calculate persistently” (which I believe Palm should be doing by default anyway?), it seems that the core data somehow is missing despite the full mysql/mongo db backups and moves over to the new server. Which is why I’m looking to figure out and pull over just the grades and then see if grade reports stop showing up as 0% for everyone.

@gabor or @nsprenkle any ideas?

---

<div class="post-metadata">

**Author:** ![dave](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/dave/32/263_2.png) [@dave](https://discuss.openedx.org/u/dave)\
**Post date:** [July 8, 2024, 5:13pm UTC](https://discuss.openedx.org/t/lost-grade-progress-while-updating-to-palm/13311/5 "2024-07-08T17:13:38Z")

</div>

> [@ABC](#):
>
> The issue is that when I create a grade report for students, they all come up as 0% done. So it’s not exactly a question of “calculate on the fly” vs. “calculate persistently” (which I believe Palm should be doing by default anyway?), it seems that the core data somehow is missing despite the full mysql/mongo db backups and moves over to the new server. Which is why I’m looking to figure out and pull over just the grades and then see if grade reports stop showing up as 0% for everyone.

The tables for grading should all start with `grades_`. The one that has persistent grades should be `grades_persistentsubsectiongrade`. Though all the tables in this [models file](https://github.com/openedx/edx-platform/blob/open-release/palm.master/lms/djangoapps/grades/models.py) are important.

When an individual student looks at a specific problem in the courseware that they’ve completed, do they see the score they earned for it? If that’s true, then at least the user state and raw scores moved over.

If that’s the case, and the `grades_` tables are empty for both of your installs, and running the `compute_grades` management command for a course doesn’t fill out this table for your students–then try forcing a course publish by making some trivial change to that course, and waiting for all the post-publish processes to finish. (It might be that the grading system is missing some content-related data that it normally generates after publishing a course.) Then try running `compute_grades` for that course again, and try downloading a grades report after it’s completed.

Hope this helps. Good luck.

---

<div class="post-metadata">

**Author:** ![gabor](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/gabor/32/2318_2.png) [@gabor](https://discuss.openedx.org/u/gabor)\
**Post date:** [July 9, 2024, 7:12am UTC](https://discuss.openedx.org/t/lost-grade-progress-while-updating-to-palm/13311/6 "2024-07-09T07:12:15Z")

</div>

Unfortunately, I’m not familiar with this particular part of the courses. I was working with grade reports and completions, but that was a while ago. Maybe @pkulkark or @braden have some hints.

Although I’m only in the dark, but have you checked if all the xblocks, plugins, etc., are installed for the new instance that you have in the old one? I know some particular plugins can create other database tables too, like the [completion aggregator](https://github.com/open-craft/openedx-completion-aggregator).

Also, have you checked you have the same number of database tables compared to the old one?

---

<div class="post-metadata">

**Author:** ![ABC](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/abc/32/4946_2.png) [@ABC](https://discuss.openedx.org/u/ABC)\
**Post date:** [July 10, 2024, 11:32am UTC](https://discuss.openedx.org/t/lost-grade-progress-while-updating-to-palm/13311/7 "2024-07-10T11:32:35Z")

</div>

Thanks @dave and @gabor. As an initial result, I found that the grades\_persistentcoursegrade and grades\_persistentsubsectiongrade mysql tables were empty on my previous production instance. So it seems like I had never enabled persistent grades for lilac. I’m doing that now, and manually re-calculating all the grades

I have three follow-up questions then: (for @pkulkark and @braden if they’re more likely to understand how grading works)

1. Once I get all these grades, what are _all_ the db tables I should export in order to import on the new server? Just all the grades\_\* ones?

2. Is there any way at import time to avoid destroying the 300 or so users’ grade data which has been updated on the new production server while I’ve been trying to figure this out?

3. Where do the grades’ source information come from, to be calculated and put in the db? I.e. if it’s ultimately just coming from somewhere else, perhaps I should just be checking that somewhere else and whether _it_ got copied over between servers, so that maybe the data could be re-computed properly? Because if that source data isn’t there, won’t future grade re-computes still be wrong if they’re currently computing zero grades on the new server?

---

<div class="post-metadata">

**Author:** ![ABC](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/abc/32/4946_2.png) [@ABC](https://discuss.openedx.org/u/ABC)\
**Post date:** [July 10, 2024, 11:36am UTC](https://discuss.openedx.org/t/lost-grade-progress-while-updating-to-palm/13311/8 "2024-07-10T11:36:19Z")

</div>

Hmm… and once a manual report from the admin UI completed, unfortunately it was named as **grade\_report\_err** and had only 7 entries like this (for a 1000-person class):

```auto
(1062, "Duplicate entry 'COURSEID-STUDENTID' for key 'grades_persistentsubsect_course_id_user_id_usage__42820224_uniq'")

```

Edit: OK, actually it generated a normal report + this error report (I just didn’t notice the normal report below the error report.)

---

<div class="post-metadata">

**Author:** ![braden](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/braden/32/46_2.png) [@braden](https://discuss.openedx.org/u/braden)\
**Post date:** [July 10, 2024, 12:11pm UTC](https://discuss.openedx.org/t/lost-grade-progress-while-updating-to-palm/13311/9 "2024-07-10T12:11:10Z")

</div>

I really don’t know much about grading, sorry.

---

<div class="post-metadata">

**Author:** ![dave](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/dave/32/263_2.png) [@dave](https://discuss.openedx.org/u/dave)\
**Post date:** [July 10, 2024, 4:58pm UTC](https://discuss.openedx.org/t/lost-grade-progress-while-updating-to-palm/13311/10 "2024-07-10T16:58:45Z")

</div>

Sorry if I sound like a broken record, but I have to ask again: Have you tried to run the backfill command `compute_grades` as described here, _on your new instance_?

[https://openedx.atlassian.net/wiki/spaces/AC/pages/755171487/Migrating+to+Persistent+Grading#The-Persistent-Grades-Backfill](https://openedx.atlassian.net/wiki/spaces/AC/pages/755171487/Migrating+to+Persistent+Grading#The-Persistent-Grades-Backfill)

My concern with generating stuff in your old instance and copying the data across is that it might have implicit assumptions about other data that should have been generated from the publish, e.g. links back to S3 stored files.

If you do something like this from the LMS:

```auto
python manage.py lms compute_grades -v1 --settings=production --courses {YOUR COURSE KEY}

```

Does that fill in values for those tables as you would expect? Or does it cause an error? After that command is run for a specific course, does downloading a grade report look like you would expect it to?

> Where do the grades’ source information come from, to be calculated and put in the db? I.e. if it’s ultimately just coming from somewhere else, perhaps I should just be checking that somewhere else and whether it got copied over between servers, so that maybe the data could be re-computed properly? Because if that source data isn’t there, won’t future grade re-computes still be wrong if they’re currently computing zero grades on the new server?

There are at least a couple of inputs into this:

- The [Course Blocks API](https://github.com/openedx/edx-platform/blob/5f9a30e0f24b7d136e63c10481654ec5a85b9d82/lms/djangoapps/course_blocks/api.py#L57), which is where grades gets its “what does the content look like for this particular user”. The ultimate source for that data is in the ModuleStore (i.e. MongoDB), but this API builds its own representation of that data after every course publish. If you’re moving the data across instances, you should probably run the `simulate_publish` command from a Studio instance, so you can trigger it across all courses without having to edit each one through the UI.
- The student’s state in courseware\_studentmodule and the `submissions_score` table.

Running the `compute_grades` management command is supposed to then read from those and materialize all the students grade information into the `grades_*` tables so that future download tasks can read from that. So when you ask for a grade report, it should be reading purely from those tables and will make the assumption that if someone doesn’t have an entry there, it means they never did the content. (Every time a user submits an answer, it also triggers a celery task that updates those `grades_` tables, particularly `grades_persistentsubsectiongrade`.) If the `compute_grades` command doesn’t fill in those tables, does it give some kind of error in its logs?

> [@ABC](#):
>
> Hmm… and once a manual report from the admin UI completed, unfortunately it was named as **grade\_report\_err** and had only 7 entries like this (for a 1000-person class):
> 
> ```auto
> (1062, "Duplicate entry 'COURSEID-STUDENTID' for key 'grades_persistentsubsect_course_id_user_id_usage__42820224_uniq'")
> 
> ```
> 
> Edit: OK, actually it generated a normal report + this error report (I just didn’t notice the normal report below the error report.)

🙁 Okay, that sounds like a race condition tried to shove duplicate data into that table for a particular (user, subsection) combination. I don’t know why it’s doing that.

---

<div class="post-metadata">

**Author:** ![ABC](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/abc/32/4946_2.png) [@ABC](https://discuss.openedx.org/u/ABC)\
**Post date:** [July 11, 2024, 4:08pm UTC](https://discuss.openedx.org/t/lost-grade-progress-while-updating-to-palm/13311/11 "2024-07-11T16:08:16Z")

</div>

> [@dave](#):
>
> Sorry if I sound like a broken record, but I have to ask again: Have you tried to run the backfill command `compute_grades` as described here, _on your new instance_?

Thanks for re-raising that. I got too focused on starting from my previous instance, because initially I couldn’t imagine that if I ran a grade from within the instructor UI, and it came up with 0% grades, that running something from the CLI would be any different. However, at preliminary test for the single largest class (`tutor local run lms ./manage.py lms compute_grades --courses <COURSEID>`) seems to have worked! So now I’m running `tutor local run lms ./manage.py lms compute_grades --all_courses` and will report back tomorrow if it worked for a class where I didn’t specify it explicitly. Thanks!!! ‼‼

---

<div class="post-metadata">

**Author:** ![ABC](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/abc/32/4946_2.png) [@ABC](https://discuss.openedx.org/u/ABC)\
**Post date:** [July 14, 2024, 10:43am UTC](https://discuss.openedx.org/t/lost-grade-progress-while-updating-to-palm/13311/12 "2024-07-14T10:43:50Z")

</div>

I can confirm now that the grades all recalcualted successfully. Thanks again @dave, you’re a life-saver!

---

<div class="post-metadata">

**Author:** ![dave](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/dave/32/263_2.png) [@dave](https://discuss.openedx.org/u/dave)\
**Post date:** [July 15, 2024, 1:26am UTC](https://discuss.openedx.org/t/lost-grade-progress-while-updating-to-palm/13311/13 "2024-07-15T01:26:07Z")

</div>

😁 I’m glad it worked out for you. Though in this case, all credit should go to the team that worked on the grades project (I know @nimisha and @iloveagent57 worked on that–I don’t remember if there were others). Grades are one of the trickiest parts of the platform, and they spent a _lot_ of time taking great pains to [document all the stuff they were doing and why](https://openedx.atlassian.net/wiki/spaces/AC/pages/159458764/Grades+Architecture).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex020/uploads/openedx/original/2X/e/e0015f89e879bc75925b71fd11133ab15fe98297.png) [@system](https://discuss.openedx.org/u/system)\
**Post date:** [October 13, 2024, 1:26am UTC](https://discuss.openedx.org/t/lost-grade-progress-while-updating-to-palm/13311/14 "2024-10-13T01:26:19Z")

</div>

Closed on October 13, 2024, 1:26am UTC

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.