Followup of Debugging persistent grade issues
Summary
Course blocks are cached to boost the performance and this performance boost is resulting in inconsistent Outline, Progress Page, and Instructor Reports. As it is strongly related to how cache behaves, it isn’t easy to reproduce it. The issue arises when the same course blocks cache is used for Audit and Verified learners.
Let’s Assume:
- All the problems of a subsection are accessible to the
Verifiedlearners. - A few Subsections are accessible to the
AuditLearners. - Audit learner accesses the course.
- Course block cache was empty.
- Blocks are cached for the
Auditlearner. - These cached blocks do not contain the inaccessible blocks due to the access checks.
- A
Verifiedlearner accesses the course, and the same course block cache is used.- User cannot see an updated Course Outline, and Progress Page.
- Instructors are unable to download the correct grade reports.
Description
To develop an initial understanding of the issue, read through this doc listing the steps to reproduce this issue.
Now, let’s discuss the inconsistencies.
Inconsistent Outline
-
Outline for Audit Learner: You cannot see the number of questions against the
Proctored Exam, which is fine as the Learner does not have access to those questions.
-
Outline for Verified Learner: You cannot see the number of questions against the
Proctored Exam, which isa bug. The learner can access those questions and should see the number of questions.
-
Outline for Verified Learner if we clear the cache: Learner can see the number of questions against
Proctored Exam.
Progress Page (Old)
-
Progress page with inconsistent Cache for a Verified learner. Problem Scores are not visible and it says
No problem scores in this section. Subsection grades and the progress graph is fine.
-
The Progress Page is fine after the cache update. Problem scores are visible.
Progress Page (New)
-
Problem Scores are not visible with Audit Cache.
-
Problem Scores are Visible after cache is updated.
Instructor Reports:
These reports can be generated from the instructor dashboard /courses/<COURSE_KEY>/instructor#view-data_download
-
Generate Problem Grade Report:- Bug when Course blocks are cached for Audit Learner (Restricted Problems are not listed):
Student ID,Email,Username,Enrollment Status,Grade,Homework 1: Subsection - Checkboxes (Earned),Homework 1: Subsection - Checkboxes (Possible),Homework 1: Subsection - Dropdown (Earned),Homework 1: Subsection - Dropdown (Possible)
3,edx@example.com,edx,enrolled,0.02,1.0,1.0,1.0,1.0
13,asad.ali+cbc1@arbisoft.com,asadalicbc1,enrolled,0.0,Not Attempted,1.0,Not Attempted,1.0
14,asad.ali+cbc2@arbisoft.com,asadalicbc2,enrolled,0.4,Not Attempted,1.0,Not Attempted,1.0
- FINE after cache Update/restart (Restricted Problems are listed):
Student ID,Email,Username,Enrollment Status,Grade,Homework 1: Subsection - Checkboxes (Earned),Homework 1: Subsection - Checkboxes (Possible),Homework 1: Subsection - Dropdown (Earned),Homework 1: Subsection - Dropdown (Possible),Final Exam 1: Proctored Exam - Checkboxes (Earned),Final Exam 1: Proctored Exam - Checkboxes (Possible),Final Exam 1: Proctored Exam - Dropdown (Earned),Final Exam 1: Proctored Exam - Dropdown (Possible),Final Exam 1: Proctored Exam - Multiple Choice (Earned),Final Exam 1: Proctored Exam - Multiple Choice (Possible)
3,edx@example.com,edx,enrolled,0.02,1.0,1.0,1.0,1.0,Not Attempted,1.0,Not Attempted,1.0,Not Attempted,1.0
13,asad.ali+cbc1@arbisoft.com,asadalicbc1,enrolled,0.0,Not Attempted,1.0,Not Attempted,1.0,Not Available,Not Available,Not Available,Not Available,Not Available,Not Available
14,asad.ali+cbc2@arbisoft.com,asadalicbc2,enrolled,0.4,Not Attempted,1.0,Not Attempted,1.0,1.0,1.0,1.0,1.0,1.0,1.0
-
Create a Report of Problem Response-
With Audit Cache (Cannot see the problems to check the state):
-
After Cache Update (Can see the arrow and the list of restricted problems):
-
Code/Implementation Details
How Course Blocks Caching is Managed?
Almost all of the views use get_course_blocks to get the course blocks. get_course_blocks calls course blocks manager BlockStructureManager.
BlockStructureManager uses BlockStructureStore and manages the course blocks. BlockStructureStore is just a Cache interface to manage the caching of the blocks.
BlockStructureManager.get_collected gets the block structure from BlockStructureStore if it exists (Cache HIT), otherwise (Cache MISS), it calls BlockStructureManager._update_collected to get the block structure from the module store and it adds the block structure to the cache using the BlockStructureStore interface.
Access Checks
get_block_for_descriptor is responsible for the access checks on the blocks. When a User does not have access, it will return None and the block won’t be cached. This results in inconsistent course blocks cache.
Proposed Solution
- [Recommended] Cache all blocks and do not check access when Caching course blocks.
- When blocks are cached for a
Verifiedlearner (It means all blocks are cached), things work well for both theAuditandVerifiedlearners.
- When blocks are cached for a
- Split course blocks cache based on the Access
- Create multiple course block caches for the different access types.
PS: These changes might have a huge impact, so it might be a good idea to get thoughts from someone with good domain knowledge.
Fixed Issues
Persistent grades were not saved due to course block cache and I fixed it in fix: fix _update_subsection_grades when restricted blocks are not cached by asadali145 · Pull Request #33963 · openedx/edx-platform · GitHub
Related Issues
Related PRs