Debugging persistent grade issues

We’ve been having ongoing issues with persistent grades, and we could use some pointers on how to debug them. We’re running off master (minus a couple of weeks), but I think the issues started for us with palm.

The problem manifests as users who complete a unit and can see their marks in the courseware. But when they look at their progress page, there are no grades.

Similarly, when the instructor looks at the gradebook, there are no grades there either.

If the instructor rescores a problem in the unit, then the progress page and the gradebook will update and then the courseware and the progress page and the gradebook are all consistent.

We’ve been looking through the error logs, hoping to find some failure with persistent grading, but we haven’t found a smoking gun yet.

We’re not 100% sure, but this may be happening with restricted content. At least, it seems to mostly happen with problems that are “Access restricted to: verified”

Any suggestions for further investigation or theories would be greatly appreciated.

Peter Pinch
MIT Open Learning

My one quick suggestion is, can you tell if the PersistentSubsectionGrade records in the database are created/modified when the user completes the unit? Or are they only updated after an instructor forces a rescore. And can you tell if the celery tasks to update such records are submitted to the broker after the learner completes the unit (usually you can grep the task name in the logs to figure this out)?

Thanks, those are great suggestions.

Do you recall what the task name is for student-initiated grading? We’ve found the rescoring tasks generated by the instructor. They weren’t helpful.

I belive it’s this one: https://github.com/openedx/edx-platform/blob/548d302610b9f39edd457c4c50b5feba38f2b188/lms/djangoapps/grades/tasks.py#L182

We believe we’ve found one issue. It seems like if you start a unit with audit-only content, the grading code will skip over graded content that appears later. We still have some work to do to verify this finding, though.

@pdpinch this was captured on the CC working group agenda. Are you all set on this or do you still need further support with this? If not we’ll close the issue.

Sorry I missed your follow-up @e0d. As it happens, we are still working on this. We think we made a discovery today – but not yet a solution – which we’ll share shortly.

This is an issue related to the caching of course blocks and how we utilize the course blocks cache during the update of persistent grades.

Steps to Reproduce

Here are the steps to reproduce this locally.

  • Create a course with multiple subsections and add some problems.
  • Some of the subsections should be restricted to the Verified learners.
  • Create a new learner and enroll in Audit Mode.
  • Access the course with the Audit Learner
  • Visit some of the subsections and verify that the restricted subsection content is not accessible.
  • Attempt the problems, and verify the grades on the Progress page.
  • Create a new learner and enroll in verified mode.
  • Access the restricted subsection.
  • Attempt problems. Submissions are successful but grades are not updated on the Progress page.

Temporary Fix

It can be fixed by clearing the course cache and then attempting a new problem or rescoring the restricted problem. The course cache can be cleared by running this code snippet. Once the cache is cleared, Verified learners can attempt a new problem and verify the submission on the Progress page.

from opaque_keys.edx.locator import CourseLocator
from xmodule.modulestore.django import modulestore
store = modulestore()
course_key = CourseLocator.from_string("REPLACE WITH COURSE ID")
from openedx.core.djangoapps.content.block_structure.store import BlockStructureStore
from django.core.cache import cache
BlockStructureStore(cache).delete(store.make_course_usage_key(course_key))

Another Fix: Change the Grading Policy

Another workaround to fix the issue is to change the grading policy. This was identified by the course teams as they noticed that when the grading policy changes, all of the grades are updated successfully.

While looking at the code that updates the grades on policy change, we have identified that it does not use the cached blocks.

THE ISSUE

Here is the code for _update_subsection_grades

def _update_subsection_grades(
        course_key, scored_block_usage_key, only_if_higher, user_id, score_deleted, force_update_subsections=False
):
    """
    A helper function to update subsection grades in the database
    for each subsection containing the given block, and to signal
    that those subsection grades were updated.
    """
    student = User.objects.get(id=user_id)
    store = modulestore()
    with store.bulk_operations(course_key):
        course_structure = get_course_blocks(student, store.make_course_usage_key(course_key))
        subsections_to_update = course_structure.get_transformer_block_field(
            scored_block_usage_key,
            GradesTransformer,
            'subsections',
            set(),
        )

        course = store.get_course(course_key, depth=0)
        subsection_grade_factory = SubsectionGradeFactory(student, course, course_structure)

        for subsection_usage_key in subsections_to_update:
            if subsection_usage_key in course_structure:
                subsection_grade = subsection_grade_factory.update(
                    course_structure[subsection_usage_key],
                    only_if_higher,
                    score_deleted,
                    force_update_subsections,
                )
                SUBSECTION_SCORE_CHANGED.send(
                    sender=None,
                    course=course,
                    course_structure=course_structure,
                    user=student,
                    subsection_grade=subsection_grade,
                )

In the above code snippet, when course_structure does not contain the restricted blocks then subsections_to_update is an empty set. Hence, it won’t make a call to subsection_grade_factory.update

What happens when the Grading Policy is Changed?

Any Permanent Fix?

I am not sure about a permanent fix to this issue as I need to look more into it.

1 Like