Openfaas and opensearch in harmony

Development Architecture
1

I see that the harmony repo can deploy openfaas but I can’t see anywhere that it is used. Neither Google nor the forum search turn up much. I found OpenFAAS - Grove Documentation - but that just says “The OpenFAAS installation is used for sending filtered periodic build failure notifications, though it may be used for other purposes by users of Grove.” Is that why it’s currently included in harmony? Or are there plans for moving stuff to Openfaas that Google didn’t turn up?

I may be misunderstanding but it also looks like meilisearch is taking over some of the functions elastic used to perform. harmony also looks like it still can install either elastic or opensearch (though lets tutor take care of meili) - is that for the same monitoring use that Grove has for them?

2

Hi @AntonOfTheWoods I don’t know the answer or who to tag here but wanted to alert you to the Large Instances WG which perhaps you could attend and ask some of these questions. They meet Tuesdays. “Kubernetes & Large/Multi Instances DevOps collaboration” is the name of the meeting. You can find it on the WG calendar, link here: https://openedx.atlassian.net/wiki/spaces/COMM/pages/3707371565/Active+Working+Groups

3

Thanks @sarina . I know large projects are hard to document - and it’s even harder to keep documentation up-to-date. I’ll confess it does seem like in many critical places the team has just given up though… “Just use tutor, you don’t need documentation”… It’s quite challenging to get productive if you can’t use tutor for various reasons!

4

@AntonOfTheWoods I think most people have focused their energy on Tutor, not “given up”, which feels unfair to say to an open-source community. If you would like to take on the project of documenting how to not use Tutor, you are welcome to do so. From what I’ve seen on the forums, this just simply isn’t something really anyone is doing.

5

Those of us who are maintain instances that don’t use Tutor create documentation prolifically, but understandably we write documentation about the components that we maintain, modify, or otherwise prioritize. We welcome more people who write non-Tutor-specific documentation, @AntonOfTheWoods, if you would like to join in our efforts.

6

Thanks @deborahgu - I definitely hope to write plenty! Is there a place where us Tutor-rejects can hang out and exchange best practices and battle stories?

I have a compose-less k3d + helm alternative that replaces mysql with postgresql and replaces mongo with ferretdb (so also postgres) that is almost ready to make public for first comments. It has a set of infrastructure bricks that fill out what the harmony chart doesn’t (yet) do (and could combine well with that) and then I converted a decent chunk of what tutor k8s does also to helm. Secrets are in secrets, variables are pod env vars, TLS everywhere, etc. And it’s pretty one-clicky to boot!

7

Hey @AntonOfTheWoods, let me chime in! I’m part of those people developing Grove and in the same time being involved in the development of Harmony.

As @sarina very well pointed out, nobody “gave up” anything, however there is something that must be clarified: the Harmony project is (mostly) maintained by major providers (OpenCraft, eduNEXT, Racoon Gang) as a community effort. Therefore, all working group members are trying their best to keep documentation up to date and follow the changes in the platform/ecosystem.

The goal of the project is to harmonize (hence the name) the approaches we took on deploying Open edX instances on scale. Just as the project README says:

This project aims to provide an easy and standardized approach that incorporates industry best practices and lessons learned.

If I recall correctly, Meilisearch was discussed in the working group a few months ago: “should or shouldn’t we install it using Helm?”. To provide context, we decided to setup OpenSearch and ElasticSearch with Harmony to allow people using one single cluster for all their instances. Not to have 1-1 installation for every instance. ElasticSearch uses 1.3GB memory IDLE in the Kubernetes cluster. Having 15 instances with 15 IDLE ElasticSearch installations wouldn’t make sense. Or at least it wouldn’t be cost-effective. If we could have left Tutor handle this, we would have been let it do; but it couldn’t. That being said, the project is not a replacement to Tutor. In fact, it provides a Tutor plugin itself to make some ElasticSearch/OpenSearch users and some more things.

However, nothing prevents anyone to use the terraform modules and such provided by the Harmony project without Tutor. I highly encourage you to participate on the next working group meeting, the 7th of October. There, you can ask questions and join the conversation easier.

Regarding OpenFAAS being part of Harmony: Grove uses it, but the reason why it was added to Harmony is that we were planning to move features to Harmony that we never did and deprecated since then. This is a really good point; we will need to remove OpenFAAS installation from Harmony. If you would like to, feel free to create a new GitHub issue, and if you want to do that on your own, I’m happy to review that PR.

Regarding OpenSearch being part of Harmony: OpenSearch and ElasticSearch (from Open edX point of view) should be interchangeable. Hence, if someone would like to use something else than ElasticSearch, the project offers OpenSearch as an alternative.

8

Thanks very much for your reply Gabor! I understand that the community has settled on tutor as a tool for the last mile. There are a few aspects of the way it currently does things that are incompatible with the projects I’m going to be working on, so have developed an alternative, 100% helm/kube-native solution, that does all the key things I need (hot reloading in dev, fully-independent helm deploy for staging/prod, etc.). It’s just started so there are many things that will need to be added eventually and cleaned up (security-wise amongst others).

For several decades now I’ve been a “dev workstations should look as close as possible to prod”, so I have a helmfile-based deployment of the infrastructure stuff - via helm-installed operators (CNPG, ot-redis, minio) or normal charts (meilisearch, argo workflows, cert-manager), that then gets installed/configured in a infra chart. There is then a final chart that installs the openedx-specific stuff. These charts are usable for dev and prod.

I then have a set of setup scripts that bootstrap either a dev workstation via k3d, or generate certs/secrets for deployment to a staging/prod cluster via the same charts.

It might add a few hundred MB to dev workstation RAM requirements but the current openedx-required images and memory requirements are fairly high already, so percentage-wise I’m not convinced the difference will be huge.

While I think that the setup I have could be adapted to scale relatively easily to at least “minor provider” status, my main goal is to have a dev setup that is as similar as possible to prod, and will scale to at least relatively robust low-medium requirements without changing the workflow. All this to say that I’m not sure I’m really useful to the large service provider WG yet - most of the stuff you will consider out of scope (because in the scope of Tutor), so maybe I could join in a few meetings time if it seems relevant.