Hello!
I have been making some updates on my tutor installation (mainly themes and openedx plugins) and suddenly after a very basic update on a plugin (updating a condition on an if statement) I started getting this error when running “tutor local quickstart -I”
Has anyone ran into this before?
When I scroll up on the console I also see this message, which seems to point that this is probably a common issue, but with my limited knowledge im not sure where to run the commands suggested to fix it.
If anyone have seen this error before I would appreciate some pointers to be able to not see this ugly error when reinitializing my tutor installation. It seems like its not really a big deal because everything in my site seems to still work properly, but Im scared that this error is causing some initialization steps down the road to not happen. Maybe its not the case and only this step is skipped and everything is still running but I would love to fix this.
Thanks in advance,
UPDATE: After hours of debugging I was able to fix it, my openedx image has a custom eduNext proctoring version and changing stuff in the requirements folder triggered all the requirements (including custom proctoring version) to be rebuilt, and it seems like having the platform running while updating proctoring caused some conflict with the forums initialization… So I tried just making a small update on my openedx plugin, bringing the platform down and then rebuilding the openedx image and that seemed to fix it.
I hope this is helpful for someone in the future, cheers!
2 Likes
UPDATE: So it seems like the error came back after I kept running “tutor local quickstart” to reinitialize my tutor installation…
It seems like elasticsearch reached the maximum amount of shards (1000/1000) but honestly I have no idea what could be causing this, or how to rebuild my indices… my knowledge is very limited when it comes to elasticsearch.
This is an image of the full stack trace highlighting the shards error:
I did find other forum threads mentioning similar issues, but they give instructions for the native openedx installation and I dont really know how to translate those to a tutor installation.
For example I found someone suggesting to run this:
sudo -sHu forum bash
cd ~/cs_comments_service/
source ~/forum_env
rake -AT # list available rake commands
rake search:initialize
Source: Forum error message in Ginkgo - #2 by andrey.kryachko
If someone could tell me how to run this on tutor (probably really basic stuff but I am lost here), or any other suggestions to fix this I would appreciate it!
Thanks!
regis
June 29, 2022, 4:25pm
#4
There’s something which I don’t understand: it’s a deeper issue which might be causing you trouble. Unless I’m mistaken, all processes which create Elasticsearch indices actually duplicate the data in new indices, and older indices are not deleted .
Take a look at the list of Elasticsearch indices on my local Open edX installation:
[elasticsearch@21d2210b13ec ~]$ curl --silent http://elasticsearch:9200/_cat/indices | sort
yellow open comment_threads_20210614111946097 uS67G4EXRguxuLqDiJIoQg 1 1 0 0 208b 208b
yellow open comment_threads_20210614111948647 qQENrHUKT3eNqSAnD32czQ 1 1 4 0 7.4kb 7.4kb
yellow open comment_threads_20210622160727569 SiZeNYxxQWCUsSZ9ekmEtQ 1 1 4 0 7.4kb 7.4kb
yellow open comment_threads_20210701045453001 5O8MiyXbQaiRM_MaElnRfQ 1 1 4 0 7.4kb 7.4kb
yellow open comment_threads_20210705084304088 d6mChfixQIiU7sIsZZWJnA 1 1 4 0 7.4kb 7.4kb
yellow open comment_threads_20210712145212276 ykrTv-YuS7-OM359sQzV8Q 1 1 4 0 7.4kb 7.4kb
yellow open comment_threads_20210810093451015 MHIlL-4jSGafQ2MFYi7GrQ 1 1 4 0 7.4kb 7.4kb
yellow open comment_threads_20210830132919302 19u-AAaiTXWGnw8AJqb_-A 1 1 4 0 7.4kb 7.4kb
yellow open comment_threads_20210830135510565 79iZr-oLRNiZIO_DEDXTuA 1 1 4 0 7.4kb 7.4kb
yellow open comment_threads_20210901113143949 UKWHEXi8QW6dDx2dEZWliQ 1 1 5 1 32.3kb 32.3kb
yellow open comment_threads_20210902131447175 BpGej4tOQnactdzp8CMITw 1 1 5 0 7.7kb 7.7kb
yellow open comment_threads_20210906083123957 5R_twDlqRuivjGEeDKTh8g 1 1 5 0 7.7kb 7.7kb
yellow open comment_threads_20210907071357495 LyFWvF5LQmWfmh-7_bohqw 1 1 5 0 7.7kb 7.7kb
yellow open comment_threads_20210914171448399 3ZB-nw4nQ2SFy3j8EslziA 1 1 5 0 7.7kb 7.7kb
yellow open comment_threads_20210921191039423 upgGpplYSUSpHqGOH2drhA 1 1 5 0 7.7kb 7.7kb
yellow open comment_threads_20210921213936429 BBJ83rSuQaCLisipVmr7cA 1 1 5 0 7.7kb 7.7kb
yellow open comment_threads_20210922140530874 EerZcTUGQc2XERTcSFSn-A 1 1 5 0 7.7kb 7.7kb
yellow open comment_threads_20210923100056950 i5wuvEjBTVOuxEToSHTRyg 1 1 6 0 14.3kb 14.3kb
yellow open comment_threads_20210928101103870 Wmp2FXWYTqaKCI2kKWLafA 1 1 6 0 8kb 8kb
yellow open comment_threads_20211004221746368 TjuVnCk9T066jjY2I-yWNw 1 1 6 0 8kb 8kb
yellow open comment_threads_20211007080449685 F8ERu5FdTHa9-AhaMVv43A 1 1 6 0 8kb 8kb
yellow open comment_threads_20211007113238619 l1FUs99vSuudRy03e2zvfg 1 1 6 0 8kb 8kb
yellow open comment_threads_20211007142427770 e9H_jJ1WTA6AtKpB9YnFvA 1 1 6 0 8kb 8kb
yellow open comment_threads_20211013091811804 Yy14DNnqRMOsD1i0Krupqg 1 1 6 0 8kb 8kb
yellow open comment_threads_20211013151753791 19mhCAhuSH-rdCpmKX6X5w 1 1 6 0 8kb 8kb
yellow open comment_threads_20211013152537591 wlVrpM8zSDWETj_XGm74ZA 1 1 6 0 8kb 8kb
yellow open comment_threads_20211020082911201 uvscoXvdTdaSna9V4P3rlw 1 1 6 0 8kb 8kb
yellow open comment_threads_20211025174352329 4nJRifl0T6u4W-veuMIjTw 1 1 7 0 14.5kb 14.5kb
yellow open comment_threads_20211028084237247 QjSxJTepTAypYaisIh8_Zw 1 1 7 0 8.2kb 8.2kb
yellow open comment_threads_20211028084934323 wrVOtJkoT8KWp9yrEw-ChA 1 1 7 0 8.2kb 8.2kb
yellow open comment_threads_20211125112919378 whbDHm6vQ2WfHFl1z6Ictw 1 1 7 0 8.2kb 8.2kb
yellow open comment_threads_20211125121950088 9qqUXTU_QHqv5B4UbqsOyg 1 1 7 0 8.2kb 8.2kb
yellow open comment_threads_20220107140740478 KC6RQGcjRDymZybqYyOsQg 1 1 7 0 11.2kb 11.2kb
yellow open comment_threads_20220120131459412 pXLAsXqSQaGrkvIgBWpsbA 1 1 7 0 11.2kb 11.2kb
yellow open comment_threads_20220629154311072 wBOF7uZsQtmbn5Dt66lgfw 1 1 7 0 11.4kb 11.4kb
yellow open comment_threads_20220629154802917 FSStWetBRRGSkao2WsYaGg 1 1 7 0 11.4kb 11.4kb
yellow open comments_20210614111946097 XeauI9gpSjGO0iiki_O_JQ 1 1 0 0 208b 208b
yellow open comments_20210614111948647 cH6ST4G8Qm-a68SBYJReHQ 1 1 4 0 6.6kb 6.6kb
yellow open comments_20210622160727569 Mf9X05NTR7K89QHTgRHd4w 1 1 4 0 6.6kb 6.6kb
yellow open comments_20210701045453001 1sGmawqTSQmbaayw8iNJJQ 1 1 4 0 6.6kb 6.6kb
yellow open comments_20210705084304088 22FydXGoSryUe3YFiKYPsA 1 1 4 0 6.6kb 6.6kb
yellow open comments_20210712145212276 xQshZIK6Q06lwaJQ-Zxptw 1 1 4 0 6.6kb 6.6kb
yellow open comments_20210810093451015 CCLnicURT6Ogh1ICeUho_g 1 1 4 0 6.6kb 6.6kb
yellow open comments_20210830132919302 u3IPZ37bRlKS9zWYVbyhDg 1 1 4 0 6.6kb 6.6kb
yellow open comments_20210830135510565 QrOVCyGLTf6x849kmMLymw 1 1 4 0 6.6kb 6.6kb
yellow open comments_20210901113143949 jEylw27XSeae1t8BEisOHw 1 1 8 0 13.1kb 13.1kb
yellow open comments_20210902131447175 DSwE1U36QMeFU4eKh-GH2g 1 1 8 0 7.3kb 7.3kb
yellow open comments_20210906083123957 pRGilPcYRhq_KC1LFRTDfg 1 1 8 0 7.3kb 7.3kb
yellow open comments_20210907071357495 ObXXcd5sTLCq-ppZGGr2Tw 1 1 8 0 7.3kb 7.3kb
yellow open comments_20210914171448399 -JdNEC5tQQuaYZG8CqIqEw 1 1 8 0 7.3kb 7.3kb
yellow open comments_20210921191039423 3pU6OZ-WQtS3nOz25BZigg 1 1 8 0 7.3kb 7.3kb
yellow open comments_20210921213936429 _udIyFyiQlO3rOy0AbKTSQ 1 1 8 0 7.3kb 7.3kb
yellow open comments_20210922140530874 O5SyXRlrQuuqfAYse6lpeg 1 1 8 0 7.3kb 7.3kb
yellow open comments_20210923100056950 Tyx9Tx-rRjyDu7m9eQUoSw 1 1 8 0 7.3kb 7.3kb
yellow open comments_20210928101103870 oDyhtHnJTi6cLUMYWbTFhA 1 1 8 0 7.3kb 7.3kb
yellow open comments_20211004221746368 -K5EJLL6SnWHpr2DXvr8fw 1 1 8 0 7.3kb 7.3kb
yellow open comments_20211007080449685 cuabrm43Snaxqfnc2lX-zA 1 1 8 0 7.3kb 7.3kb
yellow open comments_20211007113238619 _Mk1TKN8Q_OhrKC7XWACPw 1 1 8 0 7.3kb 7.3kb
yellow open comments_20211007142427770 uyS3Hqe1TF6R5jVHEZbXDA 1 1 8 0 7.3kb 7.3kb
yellow open comments_20211013091811804 ZQtwgAnNTGmaHYYAljYC1g 1 1 8 0 7.3kb 7.3kb
yellow open comments_20211013151753791 wQK_MNS8QfGjNV7MoNBt6w 1 1 8 0 7.3kb 7.3kb
yellow open comments_20211013152537591 EZCHdwgeTiq09-GIwEU-uA 1 1 8 0 7.3kb 7.3kb
yellow open comments_20211020082911201 Z0KWwaNdRySOlh05o5g9sQ 1 1 8 0 7.3kb 7.3kb
yellow open comments_20211025174352329 NJZqlLxyTJajLFKarB4UUw 1 1 8 0 7.3kb 7.3kb
yellow open comments_20211028084237247 EpgNh7F_TYmL6kFIO4-MBA 1 1 8 0 7.3kb 7.3kb
yellow open comments_20211028084934323 o0Jr6s00SjCZAx0MRSI_ng 1 1 8 0 7.3kb 7.3kb
yellow open comments_20211125112919378 CLYWC8_bRr2XHFU5oOWrXw 1 1 8 0 7.3kb 7.3kb
yellow open comments_20211125121950088 hVWQ6MxTQl2bvigfcs_rww 1 1 8 0 7.3kb 7.3kb
yellow open comments_20220107140740478 KBlYBmQCR42XZJWH8MUYmg 1 1 8 0 10.7kb 10.7kb
yellow open comments_20220120131459412 AadJGMMpTLCeSEGYgmXaJg 1 1 8 0 10.7kb 10.7kb
yellow open comments_20220629154311072 UhQ_WzbnTnCIkIj_7yH4RQ 1 1 8 0 10.7kb 10.7kb
yellow open comments_20220629154802917 YZZ-DX--Rw--OJgiRfYmmQ 1 1 8 0 10.7kb 10.7kb
yellow open course_20210928_101212 52tLHqp6QtysWDMtSL4rAw 1 1 0 0 208b 208b
yellow open course_20210928_101223 3IF3hV5TTqaIc0iKT8gkDg 1 1 9 0 21.2kb 21.2kb
yellow open course_20211007_080600 9VLOLiIlS0urofCL_gQK7g 1 1 9 0 21.2kb 21.2kb
yellow open course_20211007_080739 0l1nailDRgyfj2ZMxSb4lQ 1 1 9 0 21.2kb 21.2kb
yellow open course_20211007_081433 _FvKrdmvRR-46opEDTZ3FQ 1 1 9 0 21.3kb 21.3kb
yellow open course_20211007_082133 X8ng6kOZSy6-b6q4Z6LuJw 1 1 9 0 21.3kb 21.3kb
yellow open course_20211007_113400 XLEZkdG3Sm6b4gUS1udT6g 1 1 9 0 21.3kb 21.3kb
yellow open course_20211007_115340 yNdVuKGnTti1qOBYDinz1w 1 1 9 0 21.3kb 21.3kb
yellow open course_20211007_120106 967j1G9tSa-c_lE5ZTqTaA 1 1 9 0 21.2kb 21.2kb
yellow open course_20211007_142549 qcNdjx0KTmqT-_pKcK6AMg 1 1 9 0 21.2kb 21.2kb
yellow open course_20220627_143723 tMdhTopJTeOGuEvBYIrtoQ 1 1 6 0 23.9kb 23.9kb
yellow open course_20220628_081337 wBeTyYBHRRCUddM-NGa3jQ 1 1 6 0 23.8kb 23.8kb
yellow open course_20220628_081741 ILOLATCzSbexv21eGbyspg 1 1 6 0 23.9kb 23.9kb
yellow open course_info J5_1OXV_RjaYopiMeWy5SQ 1 1 14 6 103.7kb 103.7kb
yellow open course_run_20210928_101213 3gNG3tBzTOGb5CKhLb4NXQ 1 1 0 0 208b 208b
yellow open course_run_20210928_101223 A5ohN6GyRfGLtqW4MeRCOw 1 1 0 0 208b 208b
yellow open course_run_20211007_080601 GLclq1zxTaqopW9nf0crrA 1 1 0 0 208b 208b
yellow open course_run_20211007_080739 BWFUdoAWSe6yKoW-tSUE1Q 1 1 0 0 208b 208b
yellow open course_run_20211007_081434 qK575v7DQ3Cq3x_SJRnUyA 1 1 0 0 208b 208b
yellow open course_run_20211007_082133 0Ph-ti1_QtmFjZ_b7hUFJg 1 1 0 0 208b 208b
yellow open course_run_20211007_113400 Habym_HIQYmNg2ku5OWLQA 1 1 0 0 208b 208b
yellow open course_run_20211007_115339 3N2y2r03TZqSGbOkGJuJSQ 1 1 0 0 208b 208b
yellow open course_run_20211007_120106 Shx7iazCRuu4XPBveT2Akw 1 1 0 0 208b 208b
yellow open course_run_20211007_142549 zDvP08BqQJiZLkVzKSiPxQ 1 1 0 0 208b 208b
yellow open course_run_20220627_143723 Vx51uWWlREmkHkNSIODDnA 1 1 0 0 208b 208b
yellow open course_run_20220628_081336 yuW7MODATSef4QpxwPyXsg 1 1 0 0 208b 208b
yellow open course_run_20220628_081741 W3oukzLnQRSfEAqZcSnOOg 1 1 0 0 208b 208b
yellow open courseware_content DurfKDsGSA6IidfmvJgquA 1 1 310 23 325.2kb 325.2kb
yellow open edx_notes_api q8FV0PSxTmKJEGuCiuk-Gg 1 1 1 0 9kb 9kb
yellow open learner_pathway_20220627_143709 K2sO0EzjQyKR0AIKwY3BOQ 1 1 0 0 208b 208b
yellow open learner_pathway_20220627_143723 sI4hHsgPTjCUAFNysggsFw 1 1 0 0 208b 208b
yellow open learner_pathway_20220628_081336 EZjULw2UT-C3x4Q_eYfc1Q 1 1 0 0 208b 208b
yellow open learner_pathway_20220628_081741 nKMNGxJdRJ-pX45OOiZH6A 1 1 0 0 208b 208b
yellow open person_20210928_101212 iSjb2A5HTYqoe5hUJL2jOw 1 1 0 0 208b 208b
yellow open person_20210928_101223 _7vO4RegS8Kh8AsySE4NuQ 1 1 0 0 208b 208b
yellow open person_20211007_080601 4aBoVjKSR-6c8wg3n9TKRw 1 1 0 0 208b 208b
yellow open person_20211007_080739 6ChVGHDdQHGQCnBsrqK5Iw 1 1 0 0 208b 208b
yellow open person_20211007_081434 KFvsOSZiQY-RLjeDQ6i6jQ 1 1 0 0 208b 208b
yellow open person_20211007_082133 7Cfd9_S0Q_C34Bf7tVk6SQ 1 1 0 0 208b 208b
yellow open person_20211007_113400 ggzo530HQgOnr3h7aZ6Itw 1 1 0 0 208b 208b
yellow open person_20211007_115339 xU8ByJv_SGaCMhJAMIYzjQ 1 1 0 0 208b 208b
yellow open person_20211007_120106 zLawG09pTK61eKnygH8IZg 1 1 0 0 208b 208b
yellow open person_20211007_142549 vNp1cnrSTVemmBkWvQV4Qw 1 1 0 0 208b 208b
yellow open person_20220627_143724 xbjJnWjZSGSjBn3LQDHXbQ 1 1 0 0 208b 208b
yellow open person_20220628_081336 dCPcltcyQfWP4wW8e_3Dag 1 1 0 0 208b 208b
yellow open person_20220628_081742 pkQKVtCBTGWO8aGtvrjSXQ 1 1 0 0 208b 208b
yellow open program_20210928_101213 LMryLSrERQq_M0Yguy7NvQ 1 1 0 0 208b 208b
yellow open program_20210928_101223 didHMwgASo6s8fD-luW3aA 1 1 0 0 208b 208b
yellow open program_20211007_080600 DhYxU-1HSoqNNMWxHFkdVg 1 1 0 0 208b 208b
yellow open program_20211007_080739 esK-qUJsTI2Hd8dNCs_J2w 1 1 0 0 208b 208b
yellow open program_20211007_081435 qGR-v4tASaSRisBRSo9tHQ 1 1 0 0 208b 208b
yellow open program_20211007_082133 Lbiyue-VRt2V5rmO69_FXQ 1 1 0 0 208b 208b
yellow open program_20211007_113400 HnWHyYJFTkqAA9_Ajg3u6g 1 1 1 0 11.2kb 11.2kb
yellow open program_20211007_115340 _osWwP00TU-DTZicxVxOXw 1 1 1 0 11.3kb 11.3kb
yellow open program_20211007_120106 _eZryvd3TWaEO8V5v0gEHA 1 1 1 0 11kb 11kb
yellow open program_20211007_142549 rUIe5DsFRjmxRYO7LQPsaQ 1 1 1 0 11kb 11kb
yellow open program_20220627_143724 DFrbbl_dRSqP8P8A2fXNXw 1 1 0 0 208b 208b
yellow open program_20220628_081337 2mGW36EBRyaBjVX48DBpxA 1 1 0 0 208b 208b
yellow open program_20220628_081741 rSwQ018DTsKOwNstZBQ0FA 1 1 0 0 208b 208b
yellow open richie_categories_2022-01-03-14h14m32.840996s dDj1PqaaTYyCZE25DBBrhg 1 1 0 0 230b 230b
yellow open richie_categories_2022-05-12-16h15m52.679308s CtWe4TvYRJeUkYLcl13Gfw 1 1 0 0 230b 230b
yellow open richie_courses_2022-01-03-14h14m33.506948s L9p2kLqXSc67zqYvRMLNrg 1 1 2 0 11.4kb 11.4kb
yellow open richie_courses_2022-05-12-16h15m53.438552s ALWqj765SL-ziYv3vzTp-Q 1 1 2 0 12.7kb 12.7kb
yellow open richie_organizations_2022-01-03-14h14m34.234992s EdXpE4niTG6yJ-dpQrgW2A 1 1 1 0 6.4kb 6.4kb
yellow open richie_organizations_2022-05-12-16h15m54.392167s CSta3eFeQfulXt-vuEM8fQ 1 1 1 0 6.4kb 6.4kb
yellow open richie_persons_2022-01-03-14h14m34.889455s uRu5LSqxQr6Yv_b1pmfr4g 1 1 0 0 230b 230b
yellow open richie_persons_2022-05-12-16h15m55.347471s YTam7UKUSmatxBphqQ-KZw 1 1 0 0 230b 230b
That’s 139 indices!
And here is the list of aliases:
$ curl http://elasticsearch:9200/_cat/aliases
course_run course_run_20220628_081741 - - - -
richie_persons richie_persons_2022-05-12-16h15m55.347471s - - - -
person person_20220628_081742 - - - -
program program_20220628_081741 - - - -
comment_threads comment_threads_20220629160638572 - - - -
richie_courses richie_courses_2022-05-12-16h15m53.438552s - - - -
learner_pathway learner_pathway_20220628_081741 - - - -
richie_categories richie_categories_2022-05-12-16h15m52.679308s - - - -
course course_20220628_081741 - - - -
comments comments_20220629160638572 - - - -
richie_organizations richie_organizations_2022-05-12-16h15m54.392167s - - - -
My understanding is that for every one of these aliases, the process in charge of indexing the data does the following:
Create a new index <myindex>_<yyyymmdd_hhmmss> and fill it with all data.
If it does not exist, create the <myindex> alias.
Point the <myindex> alias to the newly-created index
Of course, the problem with this approach is that older data that resides in existing indices is never deleted, which causes infinitely growing disk usage. I did not manage to find any command which would automatically delete stale data. Well, it’s possible to delete indices from last years with:
curl -X DELETE http://elasticsearch:9200/*_2020*
curl -X DELETE http://elasticsearch:9200/*_2021*
But there are many reasons why these commands are not really sufficient.
This situation is not tenable. How should we delete stale data? How does edX.org handle stale data? @dave I’m taking the liberty to ping you: are you familiar with the management of ES indices in Open edX?
dave
June 29, 2022, 4:38pm
#5
I am not. @Diana_Huang might be… (or know someone who is)?
