verifying the objective: build openedx docker images faster given the constraint that build and deploy workflows run from Github.
there is no host. there’s a k8s cluster where the deployed applications run, and there are ephemeral nodes at Github where build and deploy workflows are executed.
I’m unfamiliar with Act but thus far at this early stage I understand that it would allow Github Actions to run on a local host. If true, that sounds like a pretty good idea to me.
I use act routinely when I need to test GitHub CI before pushing changes upstream. Act works by running a Docker container locally. But building a Docker image inside a Docker container is not trivial. I used to build the Tutor images in a Kubernetes cluster. But that was just too difficult: very often the nodes were running out of disk space. Upgrading Kubernetes clusters was also a pain. So I moved back to building images locally: for that I use the docker:dind image, and it works great so far – though the setup is a little convoluted.
I suggest you either run a self-hosted GitHub runner, setup Docker caching or figure out how to build Docker images remotely (with the dind images for instance).
thanks @regis. the self-hosted runners look very promising, especial given that these can run locally on Windows and macOS. i’ll read more about these and will probably have time later this week to try to prototype something in the Cookiecutter. i’ll report back after i know more.
This evolved into this working group which is working on tutor-contrib-multi
Obviously with the different issues that it entails
Xavier
The configuration repo has been deprecated and that we were all working on the same thing. Is it possible to get something nice/maintainable.
Last month we started the review of the current approach and with specific steps to verify that the current approach works to validate that this is a good base to be working on.
Adam
Joined edX 4 years ago and researched deploying to EKS clusters.
Took the notes app and containerised it. It’s been the only service running in k8s for 2.5 years.
Was deployed using Kustomize. They needed better customization especially with respect to liveness probes.
It’s still closed source, but it’s running about 7 new Django api endpoints. Some are public, but not the rest due to license concerns, etc.
Considered using an umbrella chart like tutor-contrib-multi, but migration is difficult, especially codejail.
They’re still considering how they’ll be able to deploy all services using k8s instead of the old configuration repo.
Asked who is running codejail behind Flask?
Felipe
eduNext used 3 approaches to deploy codejail.
Docker in docker didn’t work.
Created tutor plugin
Changed edx-platform enough to choose whether it runs within the same process
@keithgg Thank you for the meeting recap! And here is the video recording (chat log):
Next meeting
I’ve sent a calendar invite for the the next meeting, which will be in 2 weeks, on 2023-01-24T17:00:00Z in this Zoom room. I’ve made it recurring to simplify the planning.
My apologies for not having properly followed the agenda for the last meeting btw, but I felt it was useful to hear from the 2U participants who have just joined us. This has delayed a bit the review of the status of the work we had scheduled for yesterday, as we didn’t have enough time to discuss it, but we’ll try to cover this asynchronously until next meeting, and dedicate more time to it during the next meeting.
Here is the proposed agenda for that next meeting - don’t hesitate if you see any changes or additions:
Assign scribe role, greetings & introductions as needed. (5 minutes)
Kubernetes, Tutor & Helm (40 minutes) - Debrief of the work from the tasks list & formal review
DevOps Working Group (10 minutes) - Continue discussing the coordination with the parent DevOps working group, and the formalization of our group (as a “big/multi instances” subgroup of devops?)
Next steps & conclusion (5 minutes)
Current tasks
Since we had very little time to discuss the tasks, the follow-up is happening async on the tickets:
Hey folks, we need to figure out a new name for the project since it’s more about Open edX + Kubernetes + Multiple Instances than it is about Tutor, though it of course assumes use of Tutor for building container images.
Thoughts on any of these names? Vote for 1-2 that you like
Baseline (openedx-k8s-baseline)
Catalyst (openedx-k8s-catalyst)
Harmony (openedx-k8s-harmony)
Ensemble (openedx-k8s-ensemble)
Common Hosting Environment for Containers on Kubernetes (Open edX CHECK)
@braden Is there a recording of the meeting? I didn’t receive one from Zoom, but it looks like the calendar was the host?
@lpm0073@keithgg Did that discussion happen? It would be useful to post an update on the ticket - it looks like some pings don’t get through to you @lpm0073 ?
I also see further down the notes that @Felipe and @braden are also interested in collaborating on this. What would be the next steps on this?
@Felipe Since the formal review period is over, it would be better to merge it provisionally - this way it becomes accessible at https://open-edx-proposals.readthedocs.io/en/latest/index.html and it’s clear that it’s the way we have agreed on for now. Then when refinements are done, we can always update the OEP. It makes it easier for others to open PRs against the document, too.
Yes, both are moving forward. I’ve commented on the first one to follow-up with @regis . For the second one, we need those who want to have access to the repo to mention it on the ticket.
@Felipe@braden Good idea to discuss this at the conference! It could be an occasion to advertise our work, to attract more contributors from the rest of the community?
@antoviaque Unfortunately none of us could figure out how to do a recording. It said we didn’t have permission. I’ll have to learn what it means if the calendar is the host; I assumed it was you.
We looked at the ticket very briefly during the meeting. I posted an update on the ticket just now. IMHO this one is not urgent so there is no need to decide now while some of the questions of scope are still a bit fuzzy. I’m hoping we’ll get more context to make such a decision in the future after other pieces have fallen into place.
Yes we discussed it and @lpm0073 is still planning to do it.
We looked at it briefly and decided to follow up async so we have more time to consider it.
Here are the results of the voting. It is a bit funny as it turns out that even the current time wouldn’t be good
Thursday and Friday have times where only one person couldn’t make it. @jmbowman do you have any suggestions when would be the best for you?
Personally, if possible, I wouldn’t put the meeting on Friday, hence it’s the last day of the week and holidays can have impact on Friday meetings (ie. extended weekends).
That Thursday time slot is already double or triple booked for me: Deprecation WG, Frontend WG, FedX (2U frontend developers), and/or Arch-BOM daily meeting depending on the week. I’d routinely miss the beginning of the Friday time slot, but could join after the Arch-BOM sprint retro/planning meeting ends around 15 minutes in (if I reschedule another biweekly meeting that usually starts 15 minutes after that).
All of the “morning in Eastern time zone” time slots are in high contention for 2U folk both for the reason here (it’s the only good overlap with folk in Europe) and because 2U has a large South Africa office with essentially the same scheduling constraints. Most meetings scoped larger than a single team are in those same hours.
The time slot we’ve been using “works” in the sense that I can attend while eating lunch…
So it means we couldn’t much move the meeting as that would conflict for you anyway? (Just double-checking if I understand correctly) Is it less occupied for you if we think about moving the meeting to odd weeks? (currently it is biweekly on every even week)
The Thursday 11am Eastern time slot is currently triple-booked for me every week even without moving this meeting there. And I have a weekly Friday 10-11:15 meeting also (that sometimes runs over), so I’d miss the beginning of the Friday 11am Eastern time slot also. But if I’m the only one those times don’t work for, maybe you should just move it and I can try to get someone else from 2U to attend?