Course Block Cache Resulting in Inconsistent Course Outline, Progress Page, and Instructor Reports

Followup of Debugging persistent grade issues

Summary
Course blocks are cached to boost the performance and this performance boost is resulting in inconsistent Outline, Progress Page, and Instructor Reports. As it is strongly related to how cache behaves, it isn’t easy to reproduce it. The issue arises when the same course blocks cache is used for Audit and Verified learners.

Let’s Assume:

  • All the problems of a subsection are accessible to the Verified learners.
  • A few Subsections are accessible to the Audit Learners.
  • Audit learner accesses the course.
    • Course block cache was empty.
    • Blocks are cached for the Audit learner.
    • These cached blocks do not contain the inaccessible blocks due to the access checks.
  • A Verified learner accesses the course, and the same course block cache is used.
    • User cannot see an updated Course Outline, and Progress Page.
  • Instructors are unable to download the correct grade reports.

Description

To develop an initial understanding of the issue, read through this doc listing the steps to reproduce this issue.

Now, let’s discuss the inconsistencies.

Inconsistent Outline

Progress Page (Old)

Progress Page (New)

Instructor Reports:
These reports can be generated from the instructor dashboard /courses/<COURSE_KEY>/instructor#view-data_download

  • Generate Problem Grade Report:

    • Bug when Course blocks are cached for Audit Learner (Restricted Problems are not listed):
Student ID,Email,Username,Enrollment Status,Grade,Homework 1: Subsection - Checkboxes (Earned),Homework 1: Subsection - Checkboxes (Possible),Homework 1: Subsection - Dropdown (Earned),Homework 1: Subsection - Dropdown (Possible)
3,edx@example.com,edx,enrolled,0.02,1.0,1.0,1.0,1.0
13,asad.ali+cbc1@arbisoft.com,asadalicbc1,enrolled,0.0,Not Attempted,1.0,Not Attempted,1.0
14,asad.ali+cbc2@arbisoft.com,asadalicbc2,enrolled,0.4,Not Attempted,1.0,Not Attempted,1.0
  • FINE after cache Update/restart (Restricted Problems are listed):
Student ID,Email,Username,Enrollment Status,Grade,Homework 1: Subsection - Checkboxes (Earned),Homework 1: Subsection - Checkboxes (Possible),Homework 1: Subsection - Dropdown (Earned),Homework 1: Subsection - Dropdown (Possible),Final Exam 1: Proctored Exam - Checkboxes (Earned),Final Exam 1: Proctored Exam - Checkboxes (Possible),Final Exam 1: Proctored Exam - Dropdown (Earned),Final Exam 1: Proctored Exam - Dropdown (Possible),Final Exam 1: Proctored Exam - Multiple Choice (Earned),Final Exam 1: Proctored Exam - Multiple Choice (Possible)
3,edx@example.com,edx,enrolled,0.02,1.0,1.0,1.0,1.0,Not Attempted,1.0,Not Attempted,1.0,Not Attempted,1.0
13,asad.ali+cbc1@arbisoft.com,asadalicbc1,enrolled,0.0,Not Attempted,1.0,Not Attempted,1.0,Not Available,Not Available,Not Available,Not Available,Not Available,Not Available
14,asad.ali+cbc2@arbisoft.com,asadalicbc2,enrolled,0.4,Not Attempted,1.0,Not Attempted,1.0,1.0,1.0,1.0,1.0,1.0,1.0

Code/Implementation Details

How Course Blocks Caching is Managed?

Almost all of the views use get_course_blocks to get the course blocks. get_course_blocks calls course blocks manager BlockStructureManager.

BlockStructureManager uses BlockStructureStore and manages the course blocks. BlockStructureStore is just a Cache interface to manage the caching of the blocks.

BlockStructureManager.get_collected gets the block structure from BlockStructureStore if it exists (Cache HIT), otherwise (Cache MISS), it calls BlockStructureManager._update_collected to get the block structure from the module store and it adds the block structure to the cache using the BlockStructureStore interface.

Access Checks

get_block_for_descriptor is responsible for the access checks on the blocks. When a User does not have access, it will return None and the block won’t be cached. This results in inconsistent course blocks cache.

Proposed Solution

  • [Recommended] Cache all blocks and do not check access when Caching course blocks.
    • When blocks are cached for a Verified learner (It means all blocks are cached), things work well for both the Audit and Verified learners.
  • Split course blocks cache based on the Access
    • Create multiple course block caches for the different access types.

PS: These changes might have a huge impact, so it might be a good idea to get thoughts from someone with good domain knowledge.

Fixed Issues

Persistent grades were not saved due to course block cache and I fixed it in fix: fix _update_subsection_grades when restricted blocks are not cached by asadali145 · Pull Request #33963 · openedx/edx-platform · GitHub

Related Issues

Related PRs

@Agrendalath Would you be able to look at this post? I see that you have worked on refactor: reuse services and wrappers between XBlocks (fixed) [FC-0026] by Agrendalath · Pull Request #32779 · openedx/edx-platform · GitHub. What are your thoughts about this post?

@kmccormick @dave this is the issue with “persistent grades” I mentioned to you at the conference office hours – in much more detail than I went into.

Do you have any suggestions on what the cause could be? Or other folks we should bring into the discussion?

Thanks for the detailed investigation. I’m off for the next two weeks, but I’ll take a look at your post when I’m back.