Recognizing that there has already been substantial investment in the adoption of Meilisearch as the de facto search backend for edx-platform, I wanted to follow up with this topic.
I had not been following this work closely, but after revisiting the conversations around high availability/redundancy/failover in Meilisearch it seems that there has still been no real progress in that direction. All of the GitHub issue and discussion threads peter out in the same manner of noting the challenge of implementing distributed consensus (e.g. Paxos, Raft) and the lack of high-quality libraries in Rust to handle them. All of the recommended methods of handling failures rely on persistent disk and restarting the process, which fundamentally fails to address high availability and shared-nothing architectures. Instead it forces you to have a distributed storage layer (e.g. NFS, GlusterFS, Ceph) or some other means of data replication to be able to handle server failures, disk corruption, etc.
I understand that the majority case of edX installations, and the primary mode of operation supported by e.g. Tutor is to have a single server or virtual machine, but for cases where someone is not operating in that fashion Meilisearch continues to pose an operational risk. Granted, the search functionality is not mission critical for the use of edx-platform, but for anyone who operates the system a failure in any element of the system can still lead to a degradation of trust in the system or the ability of the operator.
I recognize that there is no perfect answer to the challenge of search, and I do like the performance promises that Meilisearch offers. That being said, it seems that Typesense would be a more appropriate alternative? It offers similar performance benefits, a longer development history with wider adoption, and an out-of-the-box HA story (Algolia vs Elasticsearch vs Meilisearch vs Typesense Comparison). Looking at the comparison document it seems that the primary downside is the in-memory nature of the engine?