How to configure MFE's on Kubernetes Olive.1 with tutor version 15.x?

[lpm0073 (Lawrence McDaniel)]

PROLOGUE: When i first attempted to deploy Olive.1 on Kubernetes using tutor 15.x I got blank screens for all enabled MFEs. Interestingly , the same MFEs worked fine when deploying to an AWS EC2 instance using tutor’s quick start instructions. Hence, the problem was clearly and directly related to Kubernetes configuration. Following were the two hard-fought battles that I went through to get MFEs working on Kubernetes.

For any enabled MFE url end point deployed to Kubernetes, i initally saw the following three errors in Chrome’s JS console window:

My configuration work with regards to MFEs was limited to the following two command lines:

         pip install tutor-mfe
         tutor plugins enable mfe

I verified the following:

• the Docker container deployed, hub.docker.com/r/overhangio/openedx-mfe, is the same for both k8s and local
• k8s ingress correctly reaches the static assets in the container. That is, the MFE endpoints return the same contents for both k8s and local.
• the bash environment variable values in the container are the same for both k8s and local
• the mfe container runs caddy run --config /etc/caddy/Caddyfile --adapter caddyfile for both k8s and local

Problem 1: Proxy service setup

One difference that i found is in the proxy services. Tutor local uses Caddy to listen on ports 80 and 443 and to proxy upstream, whereas k8s (in my case) is using nginx. Tutor’s Caddy proxy includes a couple of entries that were not present in the nginx.conf of my nginx container on k8s. These included the following:

I duplicated the path and additional header in the nginx controller as follows:

Afterwards, the MFE container correctly orchestrated the generation of the mfe_config url end point:

Note however, that all of the urls in the dict above are “http” rather than “https”, leading to the other problem.

Problem 2: Mixed Content errors in the browser

the tutor-mfe plugin is designed for a single-server deployment in which the tutor setting ENABLE_HTTPS works well as a once-size-fits-all toggle for whether or not to use http or https. But in Kubernetes deployments this setting requires more finesse. I ultimately created this tutor patch to iron out the http protocol settings.

And now, everything works!

[regis (Régis Behmo)]

Hi Lawrence,

1. Are you not running the Caddy container in Kubernetes? If not, why?
2. You mention that the ENABLE_HTTPS toggle does not work for you. Can you explain why?

The MFEs should definitely work in Kubernetes out of the box, and if they don’t then it’s a bug.

[arbrandes (Adolfo R. Brandes)]

Hi @lpm0073,

As reviewer of the PR that introduced dynamic config into tutor-mfe for Olive, I’m sorry to hear it has caused you trouble. Two things, though:

1. Can you reproduce the problem with the default tutor k8s deployment of Caddy?

2. Can you provide a diff -u of your openedx-lms-production-settings patch against the one in tutor-mfe? I’d like to understand what needed to be changed. Plus, if it makes sense, we might pull that into the repo for general consumption.

Finally, I’d also like to understand why nginx instead of Caddy.

[lpm0073 (Lawrence McDaniel)]

hey @regis, thank you very much for inquiring! :D.
I’ll respectfully opine that it is not a tutor bug. tutor presently, correctly, handles two use cases: a.) all url endpoints are accessed via http, or b.) they’re all accessed via https. Contrastingly, the Kubernetes platforms that i manage via the cookiecutter are hybrid configurations that use https with TLS termination up to the load balancer, and then http for all internal traffic behind the load balancer. To date, none of my clients have expressed any interest in improving this to a zero-trust platform, which is what would be required in order to rely solely on tutor’s ENABLE_HTTPS setting. To me, my use case seems like an edge case that wouldn’t merit any changes to tutor. At any rate, the patch that i created, which modifies the urls in the MFE_CONFIG dict, is simple and ought to work fine as a long-term solution.

Regarding your question about Caddy: to be clear, I agree that Caddy could certainly work fine inside a Kubernetes-based deployment, as you’ve asserted. The Cookiecutter uses Nginx simply because the original legacy codebase that i leveraged as seed material was itself based on Nginx. Nginx also works fine, and I personally don’t have a technical preference either way. However it turns out that of my clients who have technology teams, all of them are quite familiar with Nginx and some of them are familiar with Caddy. This, along w the fact that the proxy services configuration either way is pretty simple, leaves me with little incentive to make any changes.

[lpm0073 (Lawrence McDaniel)]

Hi @arbrandes and thanks for following up. to clarify, it’s a feature request to handle an edge case rather than a bug fix. My PR only regards Kubernetes-based installations that both a.) rely on a remote Redis cluster as an alternative to the Redis pod that tutor provides natively, and b.) run multiple open edx environments within the same Kubernetes cluster. For the moment I have three clients who want this.

To accomplish this, it will be necessary to parameterize the KEY_PREFIX key in this dict so as to differentiate objects from multiple environments that reside in the same cluster.

While this remains pending, the only pain point to my clients regards user experience in the prod environments of these three platforms when re-deploying, which potentially has the affect of ending the sessions of logged-in users. Also, in the interest of completeness, there are multiple workarounds. Namely, creating bespoke cache configurations for each environment.

definitely an edge case

[regis (Régis Behmo)]

@lpm0073

Contrastingly, the Kubernetes platforms that i manage via the cookiecutter are hybrid configurations that use https with TLS termination up to the load balancer, and then http for all internal traffic behind the load balancer.

This is also the case with the default Tutor configuration for K8s. Tutor does not use https for internal traffic. What it does, is to make some calls (not all) to the external API endpoints for some services. Thus, when ENABLE_HTTPS=true, calls are made to the https external endpoints; this is not internal traffic, but external. This is for instance the case for ecommerce/discovery; it’s due not to the Tutor way of doing things, but to the Open edX infrastructure.

@lpm0073

The Cookiecutter uses Nginx simply because the original legacy codebase that i leveraged as seed material was itself based on Nginx.

Right, I understand, If you are replacing Caddy with something else, then yes you have to backport all of the Caddy configuration.

@lpm0073

My PR only regards Kubernetes-based installations that both a.) rely on a remote Redis cluster as an alternative to the Redis pod that tutor provides natively, and b.) run multiple open edx environments within the same Kubernetes cluster. For the moment I have three clients who want this.

To accomplish this, it will be necessary to parameterize the KEY_PREFIX key in this dict so as to differentiate objects from multiple environments that

This is the PR, right? add REDIS_KEY_PREFIX by lpm0073 · Pull Request #788 · overhangio/tutor · GitHub I suggested a different approach in my comment.