Tutor initialization error on "bundle exec rake search:rebuild_indices"

Hello!

I have been making some updates on my tutor installation (mainly themes and openedx plugins) and suddenly after a very basic update on a plugin (updating a condition on an if statement) I started getting this error when running “tutor local quickstart -I”

image

Has anyone ran into this before?

When I scroll up on the console I also see this message, which seems to point that this is probably a common issue, but with my limited knowledge im not sure where to run the commands suggested to fix it.

image

If anyone have seen this error before I would appreciate some pointers to be able to not see this ugly error when reinitializing my tutor installation. It seems like its not really a big deal because everything in my site seems to still work properly, but Im scared that this error is causing some initialization steps down the road to not happen. Maybe its not the case and only this step is skipped and everything is still running but I would love to fix this.

Thanks in advance,
Diego

UPDATE: After hours of debugging I was able to fix it, my openedx image has a custom eduNext proctoring version and changing stuff in the requirements folder triggered all the requirements (including custom proctoring version) to be rebuilt, and it seems like having the platform running while updating proctoring caused some conflict with the forums initialization… So I tried just making a small update on my openedx plugin, bringing the platform down and then rebuilding the openedx image and that seemed to fix it.

I hope this is helpful for someone in the future, cheers!

2 Likes

UPDATE: So it seems like the error came back after I kept running “tutor local quickstart” to reinitialize my tutor installation…

It seems like elasticsearch reached the maximum amount of shards (1000/1000) but honestly I have no idea what could be causing this, or how to rebuild my indices… my knowledge is very limited when it comes to elasticsearch.

This is an image of the full stack trace highlighting the shards error:

I did find other forum threads mentioning similar issues, but they give instructions for the native openedx installation and I dont really know how to translate those to a tutor installation.

For example I found someone suggesting to run this:

sudo -sHu forum bash
cd ~/cs_comments_service/
source ~/forum_env
rake -AT # list available rake commands
rake search:initialize

Source: Forum error message in Ginkgo - #2 by andrey.kryachko

If someone could tell me how to run this on tutor (probably really basic stuff but I am lost here), or any other suggestions to fix this I would appreciate it!

Thanks!
Diego

There’s something which I don’t understand: it’s a deeper issue which might be causing you trouble. Unless I’m mistaken, all processes which create Elasticsearch indices actually duplicate the data in new indices, and older indices are not deleted.

Take a look at the list of Elasticsearch indices on my local Open edX installation:

[elasticsearch@21d2210b13ec ~]$ curl --silent http://elasticsearch:9200/_cat/indices | sort
yellow open comment_threads_20210614111946097                uS67G4EXRguxuLqDiJIoQg 1 1   0  0    208b    208b
yellow open comment_threads_20210614111948647                qQENrHUKT3eNqSAnD32czQ 1 1   4  0   7.4kb   7.4kb
yellow open comment_threads_20210622160727569                SiZeNYxxQWCUsSZ9ekmEtQ 1 1   4  0   7.4kb   7.4kb
yellow open comment_threads_20210701045453001                5O8MiyXbQaiRM_MaElnRfQ 1 1   4  0   7.4kb   7.4kb
yellow open comment_threads_20210705084304088                d6mChfixQIiU7sIsZZWJnA 1 1   4  0   7.4kb   7.4kb
yellow open comment_threads_20210712145212276                ykrTv-YuS7-OM359sQzV8Q 1 1   4  0   7.4kb   7.4kb
yellow open comment_threads_20210810093451015                MHIlL-4jSGafQ2MFYi7GrQ 1 1   4  0   7.4kb   7.4kb
yellow open comment_threads_20210830132919302                19u-AAaiTXWGnw8AJqb_-A 1 1   4  0   7.4kb   7.4kb
yellow open comment_threads_20210830135510565                79iZr-oLRNiZIO_DEDXTuA 1 1   4  0   7.4kb   7.4kb
yellow open comment_threads_20210901113143949                UKWHEXi8QW6dDx2dEZWliQ 1 1   5  1  32.3kb  32.3kb
yellow open comment_threads_20210902131447175                BpGej4tOQnactdzp8CMITw 1 1   5  0   7.7kb   7.7kb
yellow open comment_threads_20210906083123957                5R_twDlqRuivjGEeDKTh8g 1 1   5  0   7.7kb   7.7kb
yellow open comment_threads_20210907071357495                LyFWvF5LQmWfmh-7_bohqw 1 1   5  0   7.7kb   7.7kb
yellow open comment_threads_20210914171448399                3ZB-nw4nQ2SFy3j8EslziA 1 1   5  0   7.7kb   7.7kb
yellow open comment_threads_20210921191039423                upgGpplYSUSpHqGOH2drhA 1 1   5  0   7.7kb   7.7kb
yellow open comment_threads_20210921213936429                BBJ83rSuQaCLisipVmr7cA 1 1   5  0   7.7kb   7.7kb
yellow open comment_threads_20210922140530874                EerZcTUGQc2XERTcSFSn-A 1 1   5  0   7.7kb   7.7kb
yellow open comment_threads_20210923100056950                i5wuvEjBTVOuxEToSHTRyg 1 1   6  0  14.3kb  14.3kb
yellow open comment_threads_20210928101103870                Wmp2FXWYTqaKCI2kKWLafA 1 1   6  0     8kb     8kb
yellow open comment_threads_20211004221746368                TjuVnCk9T066jjY2I-yWNw 1 1   6  0     8kb     8kb
yellow open comment_threads_20211007080449685                F8ERu5FdTHa9-AhaMVv43A 1 1   6  0     8kb     8kb
yellow open comment_threads_20211007113238619                l1FUs99vSuudRy03e2zvfg 1 1   6  0     8kb     8kb
yellow open comment_threads_20211007142427770                e9H_jJ1WTA6AtKpB9YnFvA 1 1   6  0     8kb     8kb
yellow open comment_threads_20211013091811804                Yy14DNnqRMOsD1i0Krupqg 1 1   6  0     8kb     8kb
yellow open comment_threads_20211013151753791                19mhCAhuSH-rdCpmKX6X5w 1 1   6  0     8kb     8kb
yellow open comment_threads_20211013152537591                wlVrpM8zSDWETj_XGm74ZA 1 1   6  0     8kb     8kb
yellow open comment_threads_20211020082911201                uvscoXvdTdaSna9V4P3rlw 1 1   6  0     8kb     8kb
yellow open comment_threads_20211025174352329                4nJRifl0T6u4W-veuMIjTw 1 1   7  0  14.5kb  14.5kb
yellow open comment_threads_20211028084237247                QjSxJTepTAypYaisIh8_Zw 1 1   7  0   8.2kb   8.2kb
yellow open comment_threads_20211028084934323                wrVOtJkoT8KWp9yrEw-ChA 1 1   7  0   8.2kb   8.2kb
yellow open comment_threads_20211125112919378                whbDHm6vQ2WfHFl1z6Ictw 1 1   7  0   8.2kb   8.2kb
yellow open comment_threads_20211125121950088                9qqUXTU_QHqv5B4UbqsOyg 1 1   7  0   8.2kb   8.2kb
yellow open comment_threads_20220107140740478                KC6RQGcjRDymZybqYyOsQg 1 1   7  0  11.2kb  11.2kb
yellow open comment_threads_20220120131459412                pXLAsXqSQaGrkvIgBWpsbA 1 1   7  0  11.2kb  11.2kb
yellow open comment_threads_20220629154311072                wBOF7uZsQtmbn5Dt66lgfw 1 1   7  0  11.4kb  11.4kb
yellow open comment_threads_20220629154802917                FSStWetBRRGSkao2WsYaGg 1 1   7  0  11.4kb  11.4kb
yellow open comments_20210614111946097                       XeauI9gpSjGO0iiki_O_JQ 1 1   0  0    208b    208b
yellow open comments_20210614111948647                       cH6ST4G8Qm-a68SBYJReHQ 1 1   4  0   6.6kb   6.6kb
yellow open comments_20210622160727569                       Mf9X05NTR7K89QHTgRHd4w 1 1   4  0   6.6kb   6.6kb
yellow open comments_20210701045453001                       1sGmawqTSQmbaayw8iNJJQ 1 1   4  0   6.6kb   6.6kb
yellow open comments_20210705084304088                       22FydXGoSryUe3YFiKYPsA 1 1   4  0   6.6kb   6.6kb
yellow open comments_20210712145212276                       xQshZIK6Q06lwaJQ-Zxptw 1 1   4  0   6.6kb   6.6kb
yellow open comments_20210810093451015                       CCLnicURT6Ogh1ICeUho_g 1 1   4  0   6.6kb   6.6kb
yellow open comments_20210830132919302                       u3IPZ37bRlKS9zWYVbyhDg 1 1   4  0   6.6kb   6.6kb
yellow open comments_20210830135510565                       QrOVCyGLTf6x849kmMLymw 1 1   4  0   6.6kb   6.6kb
yellow open comments_20210901113143949                       jEylw27XSeae1t8BEisOHw 1 1   8  0  13.1kb  13.1kb
yellow open comments_20210902131447175                       DSwE1U36QMeFU4eKh-GH2g 1 1   8  0   7.3kb   7.3kb
yellow open comments_20210906083123957                       pRGilPcYRhq_KC1LFRTDfg 1 1   8  0   7.3kb   7.3kb
yellow open comments_20210907071357495                       ObXXcd5sTLCq-ppZGGr2Tw 1 1   8  0   7.3kb   7.3kb
yellow open comments_20210914171448399                       -JdNEC5tQQuaYZG8CqIqEw 1 1   8  0   7.3kb   7.3kb
yellow open comments_20210921191039423                       3pU6OZ-WQtS3nOz25BZigg 1 1   8  0   7.3kb   7.3kb
yellow open comments_20210921213936429                       _udIyFyiQlO3rOy0AbKTSQ 1 1   8  0   7.3kb   7.3kb
yellow open comments_20210922140530874                       O5SyXRlrQuuqfAYse6lpeg 1 1   8  0   7.3kb   7.3kb
yellow open comments_20210923100056950                       Tyx9Tx-rRjyDu7m9eQUoSw 1 1   8  0   7.3kb   7.3kb
yellow open comments_20210928101103870                       oDyhtHnJTi6cLUMYWbTFhA 1 1   8  0   7.3kb   7.3kb
yellow open comments_20211004221746368                       -K5EJLL6SnWHpr2DXvr8fw 1 1   8  0   7.3kb   7.3kb
yellow open comments_20211007080449685                       cuabrm43Snaxqfnc2lX-zA 1 1   8  0   7.3kb   7.3kb
yellow open comments_20211007113238619                       _Mk1TKN8Q_OhrKC7XWACPw 1 1   8  0   7.3kb   7.3kb
yellow open comments_20211007142427770                       uyS3Hqe1TF6R5jVHEZbXDA 1 1   8  0   7.3kb   7.3kb
yellow open comments_20211013091811804                       ZQtwgAnNTGmaHYYAljYC1g 1 1   8  0   7.3kb   7.3kb
yellow open comments_20211013151753791                       wQK_MNS8QfGjNV7MoNBt6w 1 1   8  0   7.3kb   7.3kb
yellow open comments_20211013152537591                       EZCHdwgeTiq09-GIwEU-uA 1 1   8  0   7.3kb   7.3kb
yellow open comments_20211020082911201                       Z0KWwaNdRySOlh05o5g9sQ 1 1   8  0   7.3kb   7.3kb
yellow open comments_20211025174352329                       NJZqlLxyTJajLFKarB4UUw 1 1   8  0   7.3kb   7.3kb
yellow open comments_20211028084237247                       EpgNh7F_TYmL6kFIO4-MBA 1 1   8  0   7.3kb   7.3kb
yellow open comments_20211028084934323                       o0Jr6s00SjCZAx0MRSI_ng 1 1   8  0   7.3kb   7.3kb
yellow open comments_20211125112919378                       CLYWC8_bRr2XHFU5oOWrXw 1 1   8  0   7.3kb   7.3kb
yellow open comments_20211125121950088                       hVWQ6MxTQl2bvigfcs_rww 1 1   8  0   7.3kb   7.3kb
yellow open comments_20220107140740478                       KBlYBmQCR42XZJWH8MUYmg 1 1   8  0  10.7kb  10.7kb
yellow open comments_20220120131459412                       AadJGMMpTLCeSEGYgmXaJg 1 1   8  0  10.7kb  10.7kb
yellow open comments_20220629154311072                       UhQ_WzbnTnCIkIj_7yH4RQ 1 1   8  0  10.7kb  10.7kb
yellow open comments_20220629154802917                       YZZ-DX--Rw--OJgiRfYmmQ 1 1   8  0  10.7kb  10.7kb
yellow open course_20210928_101212                           52tLHqp6QtysWDMtSL4rAw 1 1   0  0    208b    208b
yellow open course_20210928_101223                           3IF3hV5TTqaIc0iKT8gkDg 1 1   9  0  21.2kb  21.2kb
yellow open course_20211007_080600                           9VLOLiIlS0urofCL_gQK7g 1 1   9  0  21.2kb  21.2kb
yellow open course_20211007_080739                           0l1nailDRgyfj2ZMxSb4lQ 1 1   9  0  21.2kb  21.2kb
yellow open course_20211007_081433                           _FvKrdmvRR-46opEDTZ3FQ 1 1   9  0  21.3kb  21.3kb
yellow open course_20211007_082133                           X8ng6kOZSy6-b6q4Z6LuJw 1 1   9  0  21.3kb  21.3kb
yellow open course_20211007_113400                           XLEZkdG3Sm6b4gUS1udT6g 1 1   9  0  21.3kb  21.3kb
yellow open course_20211007_115340                           yNdVuKGnTti1qOBYDinz1w 1 1   9  0  21.3kb  21.3kb
yellow open course_20211007_120106                           967j1G9tSa-c_lE5ZTqTaA 1 1   9  0  21.2kb  21.2kb
yellow open course_20211007_142549                           qcNdjx0KTmqT-_pKcK6AMg 1 1   9  0  21.2kb  21.2kb
yellow open course_20220627_143723                           tMdhTopJTeOGuEvBYIrtoQ 1 1   6  0  23.9kb  23.9kb
yellow open course_20220628_081337                           wBeTyYBHRRCUddM-NGa3jQ 1 1   6  0  23.8kb  23.8kb
yellow open course_20220628_081741                           ILOLATCzSbexv21eGbyspg 1 1   6  0  23.9kb  23.9kb
yellow open course_info                                      J5_1OXV_RjaYopiMeWy5SQ 1 1  14  6 103.7kb 103.7kb
yellow open course_run_20210928_101213                       3gNG3tBzTOGb5CKhLb4NXQ 1 1   0  0    208b    208b
yellow open course_run_20210928_101223                       A5ohN6GyRfGLtqW4MeRCOw 1 1   0  0    208b    208b
yellow open course_run_20211007_080601                       GLclq1zxTaqopW9nf0crrA 1 1   0  0    208b    208b
yellow open course_run_20211007_080739                       BWFUdoAWSe6yKoW-tSUE1Q 1 1   0  0    208b    208b
yellow open course_run_20211007_081434                       qK575v7DQ3Cq3x_SJRnUyA 1 1   0  0    208b    208b
yellow open course_run_20211007_082133                       0Ph-ti1_QtmFjZ_b7hUFJg 1 1   0  0    208b    208b
yellow open course_run_20211007_113400                       Habym_HIQYmNg2ku5OWLQA 1 1   0  0    208b    208b
yellow open course_run_20211007_115339                       3N2y2r03TZqSGbOkGJuJSQ 1 1   0  0    208b    208b
yellow open course_run_20211007_120106                       Shx7iazCRuu4XPBveT2Akw 1 1   0  0    208b    208b
yellow open course_run_20211007_142549                       zDvP08BqQJiZLkVzKSiPxQ 1 1   0  0    208b    208b
yellow open course_run_20220627_143723                       Vx51uWWlREmkHkNSIODDnA 1 1   0  0    208b    208b
yellow open course_run_20220628_081336                       yuW7MODATSef4QpxwPyXsg 1 1   0  0    208b    208b
yellow open course_run_20220628_081741                       W3oukzLnQRSfEAqZcSnOOg 1 1   0  0    208b    208b
yellow open courseware_content                               DurfKDsGSA6IidfmvJgquA 1 1 310 23 325.2kb 325.2kb
yellow open edx_notes_api                                    q8FV0PSxTmKJEGuCiuk-Gg 1 1   1  0     9kb     9kb
yellow open learner_pathway_20220627_143709                  K2sO0EzjQyKR0AIKwY3BOQ 1 1   0  0    208b    208b
yellow open learner_pathway_20220627_143723                  sI4hHsgPTjCUAFNysggsFw 1 1   0  0    208b    208b
yellow open learner_pathway_20220628_081336                  EZjULw2UT-C3x4Q_eYfc1Q 1 1   0  0    208b    208b
yellow open learner_pathway_20220628_081741                  nKMNGxJdRJ-pX45OOiZH6A 1 1   0  0    208b    208b
yellow open person_20210928_101212                           iSjb2A5HTYqoe5hUJL2jOw 1 1   0  0    208b    208b
yellow open person_20210928_101223                           _7vO4RegS8Kh8AsySE4NuQ 1 1   0  0    208b    208b
yellow open person_20211007_080601                           4aBoVjKSR-6c8wg3n9TKRw 1 1   0  0    208b    208b
yellow open person_20211007_080739                           6ChVGHDdQHGQCnBsrqK5Iw 1 1   0  0    208b    208b
yellow open person_20211007_081434                           KFvsOSZiQY-RLjeDQ6i6jQ 1 1   0  0    208b    208b
yellow open person_20211007_082133                           7Cfd9_S0Q_C34Bf7tVk6SQ 1 1   0  0    208b    208b
yellow open person_20211007_113400                           ggzo530HQgOnr3h7aZ6Itw 1 1   0  0    208b    208b
yellow open person_20211007_115339                           xU8ByJv_SGaCMhJAMIYzjQ 1 1   0  0    208b    208b
yellow open person_20211007_120106                           zLawG09pTK61eKnygH8IZg 1 1   0  0    208b    208b
yellow open person_20211007_142549                           vNp1cnrSTVemmBkWvQV4Qw 1 1   0  0    208b    208b
yellow open person_20220627_143724                           xbjJnWjZSGSjBn3LQDHXbQ 1 1   0  0    208b    208b
yellow open person_20220628_081336                           dCPcltcyQfWP4wW8e_3Dag 1 1   0  0    208b    208b
yellow open person_20220628_081742                           pkQKVtCBTGWO8aGtvrjSXQ 1 1   0  0    208b    208b
yellow open program_20210928_101213                          LMryLSrERQq_M0Yguy7NvQ 1 1   0  0    208b    208b
yellow open program_20210928_101223                          didHMwgASo6s8fD-luW3aA 1 1   0  0    208b    208b
yellow open program_20211007_080600                          DhYxU-1HSoqNNMWxHFkdVg 1 1   0  0    208b    208b
yellow open program_20211007_080739                          esK-qUJsTI2Hd8dNCs_J2w 1 1   0  0    208b    208b
yellow open program_20211007_081435                          qGR-v4tASaSRisBRSo9tHQ 1 1   0  0    208b    208b
yellow open program_20211007_082133                          Lbiyue-VRt2V5rmO69_FXQ 1 1   0  0    208b    208b
yellow open program_20211007_113400                          HnWHyYJFTkqAA9_Ajg3u6g 1 1   1  0  11.2kb  11.2kb
yellow open program_20211007_115340                          _osWwP00TU-DTZicxVxOXw 1 1   1  0  11.3kb  11.3kb
yellow open program_20211007_120106                          _eZryvd3TWaEO8V5v0gEHA 1 1   1  0    11kb    11kb
yellow open program_20211007_142549                          rUIe5DsFRjmxRYO7LQPsaQ 1 1   1  0    11kb    11kb
yellow open program_20220627_143724                          DFrbbl_dRSqP8P8A2fXNXw 1 1   0  0    208b    208b
yellow open program_20220628_081337                          2mGW36EBRyaBjVX48DBpxA 1 1   0  0    208b    208b
yellow open program_20220628_081741                          rSwQ018DTsKOwNstZBQ0FA 1 1   0  0    208b    208b
yellow open richie_categories_2022-01-03-14h14m32.840996s    dDj1PqaaTYyCZE25DBBrhg 1 1   0  0    230b    230b
yellow open richie_categories_2022-05-12-16h15m52.679308s    CtWe4TvYRJeUkYLcl13Gfw 1 1   0  0    230b    230b
yellow open richie_courses_2022-01-03-14h14m33.506948s       L9p2kLqXSc67zqYvRMLNrg 1 1   2  0  11.4kb  11.4kb
yellow open richie_courses_2022-05-12-16h15m53.438552s       ALWqj765SL-ziYv3vzTp-Q 1 1   2  0  12.7kb  12.7kb
yellow open richie_organizations_2022-01-03-14h14m34.234992s EdXpE4niTG6yJ-dpQrgW2A 1 1   1  0   6.4kb   6.4kb
yellow open richie_organizations_2022-05-12-16h15m54.392167s CSta3eFeQfulXt-vuEM8fQ 1 1   1  0   6.4kb   6.4kb
yellow open richie_persons_2022-01-03-14h14m34.889455s       uRu5LSqxQr6Yv_b1pmfr4g 1 1   0  0    230b    230b
yellow open richie_persons_2022-05-12-16h15m55.347471s       YTam7UKUSmatxBphqQ-KZw 1 1   0  0    230b    230b

That’s 139 indices!

And here is the list of aliases:

$ curl http://elasticsearch:9200/_cat/aliases
course_run           course_run_20220628_081741                       - - - -
richie_persons       richie_persons_2022-05-12-16h15m55.347471s       - - - -
person               person_20220628_081742                           - - - -
program              program_20220628_081741                          - - - -
comment_threads      comment_threads_20220629160638572                - - - -
richie_courses       richie_courses_2022-05-12-16h15m53.438552s       - - - -
learner_pathway      learner_pathway_20220628_081741                  - - - -
richie_categories    richie_categories_2022-05-12-16h15m52.679308s    - - - -
course               course_20220628_081741                           - - - -
comments             comments_20220629160638572                       - - - -
richie_organizations richie_organizations_2022-05-12-16h15m54.392167s - - - -

My understanding is that for every one of these aliases, the process in charge of indexing the data does the following:

  1. Create a new index <myindex>_<yyyymmdd_hhmmss> and fill it with all data.
  2. If it does not exist, create the <myindex> alias.
  3. Point the <myindex> alias to the newly-created index

Of course, the problem with this approach is that older data that resides in existing indices is never deleted, which causes infinitely growing disk usage. I did not manage to find any command which would automatically delete stale data. Well, it’s possible to delete indices from last years with:

curl -X DELETE http://elasticsearch:9200/*_2020*
curl -X DELETE http://elasticsearch:9200/*_2021*

But there are many reasons why these commands are not really sufficient.

This situation is not tenable. How should we delete stale data? How does edX.org handle stale data? @dave I’m taking the liberty to ping you: are you familiar with the management of ES indices in Open edX?

I am not. @Diana_Huang might be… (or know someone who is)?