Shared ElasticSearch/OpenSearch

Development Architecture
1

Hi EdX.

As I mentioned here, we are looking for a way to use one Elastic cluster for more than one EdX organization and we didn’t have any luck.
after reading the source code, I found that Index names were hardcoded in lots of places like:

I found that you are removing Elastic dependency in some places like:

course-discovery
edx-notes
edx-analytics-api

and switching these parts of code to use OpenSearch as well:

cs_comments_service
blockstore
edx-search (courseware search)

is that possible to read index names from settings (environment variable or something) and have the possibility to change the values?
if so, where are all the indexes used in elastic?

2

Hi! I don’t think we are planning on making the index names configurable as part of our migration to use OpenSearch, and I don’t believe there’s any current configuration available for it. Currently, edx.org uses separate ES clusters for each of our services as it is, so even comments service and edx-search don’t share a cluster.

It might be something that the community could put the work in and submit a PR for, but I think using separate clusters is probably the easiest thing to do here.

3

@Diana_Huang it sounds like 2U doesn’t have need for this functionality but if others in the community did and submitted PRs to make the indexes configurable, those changes would likely be accepted, is that right?

That said, at the moment, I think none of the index names for any of the existing repos conflict with each other. So it should be possible to run all of them off of one ES cluster. You just wouldn’t be able to do multi-tenancy easily because running 2 instances of the same service with independent data but the same ES cluster would not work.

4

Thanks for the response @Diana_Huang.

so If the community makes a PR for this matter, would be accepted?

well, each newly deployed EdX instance needs at least 1.5 GB of RAM for Elastic Search, and unlike other parts of the EdX like MongoDB and MySQL, Elastic is the only part that is not configurable.

5

I would be willing to help review/push through a PR that makes the indexes configurable.

The big considerations that it would have to take into account would be:

  • backwards compatibility with existing installations
  • working with/around the OpenSearch migration/ES removal efforts

There is a #search-migration channel on the Open edX Slack for talking about the migration going forward.

1 Like
6

I Appreciate it @Diana_Huang.

@feanil, do we know exactly which apps are using elastic right now and where are the indexes in each app?
as far as I know, these apps are using it:

cs_comments_service https://github.com/openedx/cs_comments_service
blockstore https://github.com/openedx/blockstore
edx-search (courseware search) https://github.com/openedx/edx-search

and as they said using elastic will be removed in the apps below in the future:
course-discovery, edx-notes, edx-analytics-api

7

I think everything mentioned in the DEPR ticket is all the services and uses we’re aware of so that should be a comprehensive list.