Tutor k8s deployment on EKS with custom plugin

Site Operators Tutor Help
, ,
1

Hello everyone,

I’m facing an issue with a custom Tutor plugin used in CMS that works correctly in a local Tutor deployment, but does not appear when deploying Open edX on EKS using Tutor k8s.

Local deployment (working)

On a single server, I deployed Open edX using:

tutor local launch

Steps:

  1. Cloned my custom plugin repository on the same server

  2. Installed the plugin (pip install -e .)

  3. Enabled the plugin using tutor plugins enable myplugin

  4. Rebuilt Open edX

  5. Launched Open edX locally

In this setup, the plugin works correctly and shows up in CMS.

EKS deployment (not working)

For Kubernetes, I did the following:

  1. Created an EKS cluster with 4 worker nodes

  2. Created a separate bastion/management server

  3. From the bastion server:

    • Installed Tutor

    • Connected to the EKS cluster

    • Launched Open edX using:

      tutor k8s launch

  4. On the same bastion server:

    • Cloned my custom plugin repository

    • Installed the plugin with pip install -e .

    • Enabled the plugin with tutor plugins enable my plugin

  5. Rebuilt Open edX

  6. Restarted the platform using:

    tutor k8s stop
tutor k8s start

Problem

  • The exact same plugin and enable steps work with tutor local launch

  • But with tutor k8s, the plugin does not appear in CMS

  • The CMS/LMS pods are running successfully, but the plugin is missing

Thank you

2

I would depend on what your plugin does. If I were to take a guess, your plugin is causing a change in the openedx Docker image. On tutor local it would rebuild the image on the server with your changes, but in k8s you need to push the image to a registry of your own.

3

it’s a backend, simple endpoints
ok if i push to registery, then i have to update tutor config to use that image?

4

opnedx image is updated by using the the pushed image.
updating tutor config variable DOCKER_IMAGE_OPENEDX,
what variable to use to update mfe image?

5

The MFE uses MFE_DOCKER_IMAGE. The MFE settings can be found here tutor-mfe/tutormfe/plugin.py at release · overhangio/tutor-mfe · GitHub (note that you should prefix them with MFE_).

You need to set both (MFE image and openedx image) to your registry/repo/tag, push them, update the tutor config you are using for the k8s commands and redeploy.

6

thank you for the guidance. it’s working fine. i would like to ask few questions for the stability of the production environment.
can we separate nodes for cms and lms?
shoud we use ingress or nlb and expose cadddy service loadbalancer to internet facing?
which services should be kept external e.g rds, mongodb

7

can we separate nodes for cms and lms?

Kubernetes does provide a way to constraint pod scheduling in such a way ( Assigning Pods to Nodes | Kubernetes ). You would need to leverage the tools provided by kustomize to set pod affinities accordingly. I mentioned a little bit about kustomize on this answer: How to customise mongodb in the deployment.yaml config? - #2 by MoisesGonzalezS . But for the most part, running lms and cms on different nodes doesn’t accomplish too much.

shoud we use ingress or nlb and expose cadddy service loadbalancer to internet facing?

A lot of people do use an ingress controller and forward the traffic to the caddy service. Setting ENABLE_WEB_PROXY: false in the tutor config would set the caddy service as a ClusterIP instead of LoadBalancer and you can set your ingress rule accordingly.

which services should be kept external e.g rds, mongodb

A rule of thumb is to look at the services that caddy exposes. You can inspect the caddyfile in $TUTOR_ROOT/env/apps/caddy/Caddyfile and see which ones need to be exposed to the internet. DB connections should remain private.