I have been making some updates on my tutor installation (mainly themes and openedx plugins) and suddenly after a very basic update on a plugin (updating a condition on an if statement) I started getting this error when running “tutor local quickstart -I”
Has anyone ran into this before?
When I scroll up on the console I also see this message, which seems to point that this is probably a common issue, but with my limited knowledge im not sure where to run the commands suggested to fix it.
If anyone have seen this error before I would appreciate some pointers to be able to not see this ugly error when reinitializing my tutor installation. It seems like its not really a big deal because everything in my site seems to still work properly, but Im scared that this error is causing some initialization steps down the road to not happen. Maybe its not the case and only this step is skipped and everything is still running but I would love to fix this.
UPDATE: After hours of debugging I was able to fix it, my openedx image has a custom eduNext proctoring version and changing stuff in the requirements folder triggered all the requirements (including custom proctoring version) to be rebuilt, and it seems like having the platform running while updating proctoring caused some conflict with the forums initialization… So I tried just making a small update on my openedx plugin, bringing the platform down and then rebuilding the openedx image and that seemed to fix it.
I hope this is helpful for someone in the future, cheers!
UPDATE: So it seems like the error came back after I kept running “tutor local quickstart” to reinitialize my tutor installation…
It seems like elasticsearch reached the maximum amount of shards (1000/1000) but honestly I have no idea what could be causing this, or how to rebuild my indices… my knowledge is very limited when it comes to elasticsearch.
This is an image of the full stack trace highlighting the shards error:
I did find other forum threads mentioning similar issues, but they give instructions for the native openedx installation and I dont really know how to translate those to a tutor installation.
For example I found someone suggesting to run this:
sudo -sHu forum bash
cd ~/cs_comments_service/
source ~/forum_env
rake -AT # list available rake commands
rake search:initialize
If someone could tell me how to run this on tutor (probably really basic stuff but I am lost here), or any other suggestions to fix this I would appreciate it!
There’s something which I don’t understand: it’s a deeper issue which might be causing you trouble. Unless I’m mistaken, all processes which create Elasticsearch indices actually duplicate the data in new indices, and older indices are not deleted.
Take a look at the list of Elasticsearch indices on my local Open edX installation:
My understanding is that for every one of these aliases, the process in charge of indexing the data does the following:
Create a new index <myindex>_<yyyymmdd_hhmmss> and fill it with all data.
If it does not exist, create the <myindex> alias.
Point the <myindex> alias to the newly-created index
Of course, the problem with this approach is that older data that resides in existing indices is never deleted, which causes infinitely growing disk usage. I did not manage to find any command which would automatically delete stale data. Well, it’s possible to delete indices from last years with:
But there are many reasons why these commands are not really sufficient.
This situation is not tenable. How should we delete stale data? How does edX.org handle stale data? @dave I’m taking the liberty to ping you: are you familiar with the management of ES indices in Open edX?
Any followup on this? I am still having the issue that at the moment I just put a bandaid on increasing the elasticsearch “max_shards_per_node” setting but I dont think this will hold on forever.
Are there any recommendations to properly clean up ES? I cant go with a simple date filter to clean up entries because some of my old data is actually still used. Or is it safe to just delete everything and let it get rebuilt when tutor quickstart runs again?
It is still weird how when running tutor local quickstart ES indices get duplicated instead of entirely remade, my current ES has only ~600 indices (which is still alot), so when running local quickstart and duplicates everything it hits a limit even tho everything should be fine if its started from scratch.
Again my knowledge of ES is very limited, so im not sure what is the best solution.
I am tagging @Diana_Huang again, it seems like the last mention got ignored… I hope we can find a solution for this.
