The meeting was held at 15:00 UTC on Zoom. @antoviaque can you post the recording?
@MoisesGonzalezS @gabor @jhony_avella @Felipe @braden @lpm0073 @keithgg @antoviaque @sambapete @mtyaka
Since the previous thread was getting a bit long, we’ve decided to create a new thread to facilitate further discussion.
What was discussed
- To deploy to Kubernetes with tutor still has some pain points:
- It’s not possible to get rid of Caddy.
- The built-in persistance for MongoDB/MySQL/ES and Redis is not that reliable.
- Configuration outside of the scope of
tutoris fairly difficult.
- Building images takes a while, making changes requires rebuilding.
@braden is working on the prototype for Helm +
tutorand is almost ready to demo.
- In the mean time@felipe is working on OEP-59.
- Getting broader community support is important, as this could become an official method of deploying Open edX to Kubernetes.
- @braden will present his prototype at the next meeting (possibly before then) to get feedback.
- @felipe to update OEP-59 so that it’s an amendment to OEP-45 instead.
- We are to involve the wider to community (especially 2U as they’re deploying to k8s already) to get their feedback and explore any other ideas.
- We will have another meeting on the 6th of December at 17:30 UTC. All is welcome
- eduNext has encountered problems with tutor. Specifically with Caddy, mixed with Ingress.
- Can’t get rid of Caddy due to the way it’s specified.
- Persistence in the cluster. ElasticSearch redis and MongoDB/MySQL is not that reliable.
- In a real installation it doesn’t fulfil the needs.
- Default settings aren’t enough. Have to add plugins to manage configuration.
- Plugin ecosystem of tutor has two use-cases.
- IDAs → Ecommerce, etc.
- Configuration, it’s difficult to manage configuration for what we need.
- It’s not just changing the plugin, but then overriding templates to override everything.
- Using tutor as a deploy tool isn’t working
- The way tutor manages jobs, hooks are injected into the manifest and applied.
- Agrees. For him, building the Open edX container is a problem, because it’s so big.
- He has to rebuild the container, because an eg. an XBlock needs to be added.
- It’s much slower than working natively.
- It’s very difficult to make small changes to the build.
- Caddy is a problem, persistance mismatch, availability zones, templates and plugins all have issues.
- Open edX container is much too large and brittle. Could turn something minor into taking up a lot of time.
- Helm is worthwhile.
- Is it possible to that the Open Edx container can be standalone and that everything else like Xblocks and such, could be added after the fact?
- Suggested using Docker layers for a base Open edX image might be an option.
- Lawrence is unable to reliably build and deploy unless there’s a codified workflow.
- He’s currently using Github actions, but is open to anything.
- There needs to be a way to codify this workflow.
- It’s difficult currently to get a reliable production build for client.
- Bash scripts seems to work, but would like something better.
- It should be possible to create a Docker layer.
- The image would need to be built every time.
- Clients are now cognizant of the time it takes to build a container and
they are taking it into account, so requests are limited. It’s a distraction
and “dominates the thought process”
- Other than building images, Kubernetes and Docker is resilient and workable
- We could use different ways of caching builds, like BuildKit, etc.
- We could handle it later as part of a working group.
- Making the build work better could be within the scope of the BTR.
- Since it’s something everyone wants, making it available in the wider community is better.
- What is the status of the current helm collabaration?
- Can we announce to the community or is there still some work to do?
- Will share prototype with the group tomorrow/later.
- Is there even anything that’s been agreed upon that we will take on going forward.
- We will definitely use Helm.
- For his clients, Open edX isn’t usually the only piece of software running on their cluster.
- Helm is suited to this task, because it’s usually used to deploy the other software in the cluster as well.
- There might be complaints from other operators that they need to now learn another technology.
- Some customers are not in favour of using helm as they might be tied to different methodologies/tech.
- Helm is fairly widely used so we shouldn’t have that issue.
- Concrete steps:
- Braden finishes prototype and gets feedback.
- Is there anything missing to get the OEP accepted?
- Is the OEP even needed?
- Ideally, the next step is to build on it. We could just start publishing and collabarate on it?
- OEP is preferred since this project is community centric.
- There is value in saying that this could be an official solution.
- OEP can be amendment to the
tutorOEP, rather than standalone as OEP’s require more effort to get accepted.
- This isn’t to replace
tutor, but the resulting project will live downstream from
- In the end, it will replace
- This is a first step to build upon that.
- We need to consider the Social and Governance aspect.
- What’s the shortest path to getting to the first step?
- Amendment to
tutorOEP sounds good.
- Is that something we can do now?
- Is there still things to figure out regarding it?
- Amendment to
- Agree on the social asepct.
- Needing to get people looking at code first > publishing an OEP.
- Willing to write code and assist there where needed.
- We need code, but we also need broader community support.
- Better to progress on both ends.
- We are all agreed, but it’s worth letting the community know that if things work out, things might be official.
- We can added addendum to OEP-45, similar to
- We will test Braden’s code.
- Change the format OEP to a modification of OEP-45.
- Will also show that multiple members of the community are working on it.
- Needs a willingness among the workgroup to standardise on a machine state to build the Open edX images.
- To benefit from this, we need a way to incrementally build Open edX images that doesn’t take 45 minutes.
- Unfortunately it does get complex so we might need more than just a machine to constantly build the images.
- We should investigate changing the tutor Dockerfile to make it more cacheable.
- IHO that’s a separate project.
- Is there a Discovery item to move the exploration forward?
- We should involve Regis next time to get his feedback on the discussion here.
- Let’s schedule the next meeting as we’ve come close to the allotted time here.
- Everyone is happy to meet again in two weeks.
- We should get more folks to join that might be interested in this discussion.
Start a new thread on the forum.
With a recap on the discussion so far.
Involve 2U as well. They don’t use
tutor, but k8s. They could have a different approach.
Ned was suprised that we’re working on this.
6th of December or the next one. Slightly later if possible. Proposed time, 17:30 UTC.
Lawrence offered help with anything code related.
Braden asked for help testing the prototype which was agreed to.