I realize that courseware is far more CPU intensive than it should be in pretty much all cases. But there are a number of contributing factors to poor performance, and a number of surprising edge cases that can make things dramatically worse (e.g. inline discussion blocks in CCX courses). It’s not clear to me at the moment which parts are hurting you the worst right now. Given how large your quiz is (~100 problems), it seems likely that you’re suffering from the general issues we have around rendering large subsections of content. But there are other possibilities as well.
For instance, say CPU is high and sluggish, and people start hitting reload or pressing the buttons many times. One thing that used to happen is that we’d throw a giant implicit transaction around certain courseware views, and multiple loads of that same view by the same user would cause workers to hang because multiple transactions were blocking on trying to update the same few rows of XBlock student state. This would essentially remove workers from the available pool, and dump the requests that they could have served onto other workers. The slower it got, the more frustrated users became, and the more likely it was to dogpile because they’d re-submit.
If you’re looking for something actionable, I think the most likely thing to work is the bit I mentioned earlier about breaking the quiz up into smaller, separate quizzes for future runs. Another option is of course to scale up the cluster. We are working on these issues–all four tickets being worked on by my team this week relate to improving server side performance in courseware–but those fixes will be part of the Maple release. It is possible that you’re running into an edge case that could be fixed by dropping a plugin, fixing a config value, flipping a feature toggle, or cherry-picking some particular commit. But that’s likely going to be hard to debug, with an uncertain outcome.
Another thing to consider in terms of future content is that we are definitely making optimizations on navigation (so not loading the whole sequence), and we will probably be optimizing composition (what things are in a particular Unit, which will help with content library module performance), but we will likely not be able to make a significant dent in Unit rendering performance without breaking backwards compatibility. Meaning that if you put 100 problem blocks in the same Unit in the Maple or Nutmeg release of Open edX, it’s very likely that none of our optimizations will make a significant difference.
I’m sorry that I can’t give better news. As @regis says, this is near and dear to my heart. We’re actively working on it and making progress, but there’s no easy fix–the courseware is slow because of deeply rooted data access issues that date back to the original prototype courseware that Open edX evolved from. In order to address these problems, we’re essentially creating new applications and data models and translating from one system to the other at publish time. I did a short writeup of that architectural shift, if you’re interested.
Take care.