Deploying Open edX on Kubernetes Using Helm

I’m not sure I follow. You want to build Docker images from within act? Why would you want to do that? Why not run build images directly on the host? Building Docker images from within Docker is not so easy.

Hello @regis ,

The cookie cutter project by @lpm0073 builds the docker images pushes it to ECR by AWS using GitHub workflow. Everytime we build image it takes close to 40 minutes because it’s done by GitHub runners instead of doing it on physical device. Hence there is no cache. To avoide this we can run these workflows locally (from bastion machine) which means while running workflow next time we will have cache and take much lesser time. This helps in making smaller changes easier particularly while developing new features.

Regards.
Sujit Kumar

@regis,

  • verifying the objective: build openedx docker images faster given the constraint that build and deploy workflows run from Github.
  • there is no host. there’s a k8s cluster where the deployed applications run, and there are ephemeral nodes at Github where build and deploy workflows are executed.

I’m unfamiliar with Act but thus far at this early stage I understand that it would allow Github Actions to run on a local host. If true, that sounds like a pretty good idea to me.

I use act routinely when I need to test GitHub CI before pushing changes upstream. Act works by running a Docker container locally. But building a Docker image inside a Docker container is not trivial. I used to build the Tutor images in a Kubernetes cluster. But that was just too difficult: very often the nodes were running out of disk space. Upgrading Kubernetes clusters was also a pain. So I moved back to building images locally: for that I use the docker:dind image, and it works great so far – though the setup is a little convoluted.

I suggest you either run a self-hosted GitHub runner, setup Docker caching or figure out how to build Docker images remotely (with the dind images for instance).

thanks @regis. the self-hosted runners look very promising, especial given that these can run locally on Windows and macOS. i’ll read more about these and will probably have time later this week to try to prototype something in the Cookiecutter. i’ll report back after i know more.

i’ll heed your advise on Act.

Meeting 2023-01-10

Thanks to everyone for joining! I won’t be cc’ing everyone since we had quite a large cohort today :slight_smile: . Video recording to follow.

Cliff notes:

Transcribed meeting notes

Xavier

  • Introductions
  • Catch up from Xavier on what we’ve done so far for the newcomers.
  • Are there any areas that they’re interested on collaborating, etc?
  • Requested further recap from Felipe

Felipe

  • We decided to collaborate based on Braden’s thread
  • This evolved into this working group which is working on tutor-contrib-multi
  • Obviously with the different issues that it entails

Xavier

  • The configuration repo has been deprecated and that we were all working on the same thing. Is it possible to get something nice/maintainable.
  • Last month we started the review of the current approach and with specific steps to verify that the current approach works to validate that this is a good base to be working on.

Adam

  • Joined edX 4 years ago and researched deploying to EKS clusters.
  • Took the notes app and containerised it. It’s been the only service running in k8s for 2.5 years.
  • Was deployed using Kustomize. They needed better customization especially with respect to liveness probes.
  • It’s still closed source, but it’s running about 7 new Django api endpoints. Some are public, but not the rest due to license concerns, etc.
  • Considered using an umbrella chart like tutor-contrib-multi, but migration is difficult, especially codejail.
  • They’re still considering how they’ll be able to deploy all services using k8s instead of the old configuration repo.
  • Asked who is running codejail behind Flask?

Felipe

Adam

  • 2U will consider eduNEXT’s approach to codejail
  • 2U’s has mostly figured out autoscaling and can contribute to the effort.
  • Currently using Nginx, but is interested to what the community is using as it turn out to be useful.
  • Best practices for Kubernetes comes up within the org.
    • How to do liveness probes
    • Processes that have permission to write to disk
    • Etc.
  • There’s a [fairwinds article)[Kubernetes Configuration Benchmark Report)

Xavier

  • Are there things that we’ve worked on that 2U is interested in?

Adam

  • Codejail behind a Django (not a Flask API) would be great as they appreciate the consistency of it.
  • They’ve got an internal testing environment and will be tested from there.

Xavier

  • Meeting is too short, what about next steps?
  • A good step after this meeting might be for Adam to comment on any of the open tickets, like the codejail one.
  • Then we could discuss there and have async discussion.

Braden

  • Collabaration helps us all even if not directly using the helm chart, because everyone benefits from the small changes.

Felipe

  • Working group for Devops
  • How do we move codejail to a common/shared roadmap.
  • We could tackle multiple projects at a time instead of limiting ourselves to a single goal.

Regis

  • Created a new initiative for the Devops working group.
  • There are already projects that are devops related (three)
  • It doesn’t make much sense to have a working group for all of these projects.
  • Instead Ed proposed a Devops working group, where all the Devops related projects will live.
  • Each project can have each own Slack channel, leaderships, governance, rules.
  • Otherwise it can be handled within the Devops working group.
  • Github project GitHub - openedx/wg-devops: Issue repository for the DevOps Working Group

Adam

  • How does this differ from BTR?

Regis

  • BTR is more concerned with creating code releases and as such is distinct from Devops.

Xavier

  • What’s the approach to communicating with the Devops WG?

Regis

Adam

  • How to decide something fits within Devops or somewhere else?
  • How would we spin up/down working groups depending on the project?

Xavier

  • We try to take care of the issues for deploying larger instances.
  • Helm was a good starting point, but there could be changes in future especially with differences between small/large providers.
  • Is 2U interested in continuing the discussion?

Adam

  • Not sure.

Xavier

  • Adam mentioned some of the current issues that 2U is also phasing. Could be helpful to comment on the tickets.
  • Or on the forum?

Regis

  • Trying to push forward refined/groomed issues. Good first issues to engage folks to start working on them.
  • Can this project define important work to attract newcomers?

Adam

  • 2U is trying to figure out the best way to deploy Kubernetes. In terms of stability, etc.
  • There’s nothing on scaling yet. Expects that 2U can start there.
  • 2U can contribute, but will very likely not use the project for a while.
  • Internal helm chart is fully featured. Enterprise ready at this stage.

Daniel

  • Discussed with legal on open sourcing the helm chart.
  • Following up with them again this week.
  • Expects no opposition, legal just doing their due diligence in terms of license/contributor guidelines.
  • Should hopefully be done by the end of this month.
  • Best practices is really important.
  • Scaling/Liveness Probes/Metrics used for Scaling/Observability/Monitoring
  • All of the above are good areas for collaboration

Jeremy

  • Trying to think about how to smooth the learning curve to how to deploy in production.
  • We’re deploying to k8s, but not developing with k8s.
  • Hoping to find a way to have more consistency across environments.

Adam

  • Autoscaling

Braden

  • Lawrence would be the best person to speak to at the moment.

Lawrence’s mic wasn’t working.

Xavier

  • Go over issues. Discussed the Nginx + Cert manager task with Moises.
  • HPA with Jhony.

My sound dropped out here (low headphone battery), so I didn’t get the full conversation

Jhony

  • Talked a bit about Karpenter and approaches to autoscaling.

Xavier

  • Objections to next meeting at same time in 2 weeks.
  • Quickly went over the issues in the Git repo.
  • Further discussion to happen async.
3 Likes

@keithgg Thank you for the meeting recap! And here is the video recording (chat log):

Next meeting

I’ve sent a calendar invite for the the next meeting, which will be in 2 weeks, on 2023-01-24T17:00:00Z in this Zoom room. I’ve made it recurring to simplify the planning.

My apologies for not having properly followed the agenda for the last meeting btw, but I felt it was useful to hear from the 2U participants who have just joined us. This has delayed a bit the review of the status of the work we had scheduled for yesterday, as we didn’t have enough time to discuss it, but we’ll try to cover this asynchronously until next meeting, and dedicate more time to it during the next meeting.

Here is the proposed agenda for that next meeting - don’t hesitate if you see any changes or additions:

  1. Assign scribe role, greetings & introductions as needed. (5 minutes)

  2. Kubernetes, Tutor & Helm (40 minutes) - Debrief of the work from the tasks list & formal review

  3. DevOps Working Group (10 minutes) - Continue discussing the coordination with the parent DevOps working group, and the formalization of our group (as a “big/multi instances” subgroup of devops?)

  4. Next steps & conclusion (5 minutes)

Current tasks

Since we had very little time to discuss the tasks, the follow-up is happening async on the tickets:

CC @adzuci

1 Like

Hey folks, we need to figure out a new name for the project since it’s more about Open edX + Kubernetes + Multiple Instances than it is about Tutor, though it of course assumes use of Tutor for building container images.

Thoughts on any of these names? Vote for 1-2 that you like :slight_smile:

  • Baseline (openedx-k8s-baseline)
  • Catalyst (openedx-k8s-catalyst)
  • Harmony (openedx-k8s-harmony)
  • Ensemble (openedx-k8s-ensemble)
  • Common Hosting Environment for Containers on Kubernetes (Open edX CHECK)

0 voters

2 Likes

Hi ya’ll! Here is the recap from the meeting we held on 2023-01-24.

This was a short meeting going over the current list of tasks in the Github repo with some focus on
cloud-hosted development environments.

First PRs are ready for review, with discussion continuing on the created tasks.

Meeting notes

Braden

  • Thanks to the creators of PRs. Nginx has been reviewed, just shared Elasticsearch needs to be checked.
  • He will check it with eduNEXT
  • Please vote on the name changes for the project on the forum.
  • We will check up on the tasks.
  • NGINX work is basically done.

Jhony

Lawrence

Felipe

Braden

  • It only works with AWS. No reason to not support for the folks that want to use it.
  • Mentioned, OEP-45 and what the current status is.

Felipe

  • He is currently arbiter, but is also author. How does that work?
  • There hasn’t been proper support from outside the working group yet, so there’s an assumption of approval.

Braden

  • Is there a way to say it’s provisionally approved?

Felipe

  • Lets keep it a draft. Will change status and add comments.

Braden

Jeremy

Braden

  • Does it require less resources than on local machine?

Jeremy

  • Can we run some services on it or do you have to have to whole thing running is a question not yet answered?

Felipe

  • They had the issue of wanting to run other things on tutor, but not the LMS.

Jeremy

  • In Devstack it’s possible, but wonky since LMS is the source of auth.
  • With tutor it’s not yet ready.

Braden

  • Looks like a nice comprehensive review.

Felipe

  • Codespaces + Github is great.

Jeremy

  • We don’t want to be using a custom orchestration system, then using something else to deploy.
  • Eg. Using Docker/Python on the dev machine, but then deploying using K8s.
  • Preferably devs should be using the same dev environment to deploy.

Braden

  • Anything else that we want to chat about?

Felipe

  • They want to present at the conference a talk about crunching the Kubernetes numbers.
  • And some sort analysis of what they’ve found so far running large instances on Kubernetes.

Braden

  • OpenCraft can definitely collaborate and share some numbers. Sounds great.
  • Anything else?

Felipe

  • Regarding the monitoring, or are we interested in doing something that monitors the platform in a more useful mannner.
  • Most of us are using Prometheus + Grafana.
  • Which results in too much information sometimes that he doesn’t care about.
  • Wants to know Pod utilization vs heartbeat of LMS.
  • Is anyone else interested.

Lawrence raises his hand

Lawrence

  • Some guys at edX have made him aware of a product called Kubecost.
  • Had a convo with them last week, will be able to break down the cost service by service.

Felipe

  • Was thinking more of the values that he wants to monitor.
  • Database values/MySQL/Ingress values.
  • But going via cost is a worthwhile avenue.

Braden

  • We haven’t really looked at the cost, but it would be very useful.
  • Monitoring would be a really great thing to collaborate on.

Nothing else to discus. Meeting ends.

3 Likes

@keithgg Thanks for the notes and the recap! :+1:

@braden Is there a recording of the meeting? I didn’t receive one from Zoom, but it looks like the calendar was the host?

@lpm0073 @keithgg Did that discussion happen? It would be useful to post an update on the ticket - it looks like some pings don’t get through to you @lpm0073 ?

I also see further down the notes that @Felipe and @braden are also interested in collaborating on this. What would be the next steps on this?

@Felipe @braden To clarify here, was the decision that we are ok to move forward with Karpenter? Could this be mentioned explicitly on the ticket at Karpenter? · Issue #7 · openedx/openedx-k8s-harmony · GitHub ? And did someone volunteer to work on it?

Btw, there are a few questions on the topic of auto-scaling in Add autoscaling · Issue #2 · openedx/openedx-k8s-harmony · GitHub that could use a few eyes maybe? Were those topics discussed during the meeting? To be able to move forward, a decision would be useful? CC @jhony_avella

@Felipe Since the formal review period is over, it would be better to merge it provisionally - this way it becomes accessible at https://open-edx-proposals.readthedocs.io/en/latest/index.html and it’s clear that it’s the way we have agreed on for now. Then when refinements are done, we can always update the OEP. It makes it easier for others to open PRs against the document, too.

Yes, both are moving forward. I’ve commented on the first one to follow-up with @regis . For the second one, we need those who want to have access to the repo to mention it on the ticket.

@jmbowman +100 to this! I have created this ticket on the repo to track the collaboration/support work on that end: Support for the Cloud-based developer environment · Issue #14 · openedx/openedx-k8s-harmony · GitHub

@Felipe @braden Good idea to discuss this at the conference! It could be an occasion to advertise our work, to attract more contributors from the rest of the community?

I have created a ticket to track this: Conference presentation about Kubernetes on large instances · Issue #15 · openedx/openedx-k8s-harmony · GitHub - I’ve also asked there about who is going to be presenting?

@antoviaque Unfortunately none of us could figure out how to do a recording. It said we didn’t have permission. I’ll have to learn what it means if the calendar is the host; I assumed it was you.

That’s what we actually decided on the call, just used the wrong word. See update from @Felipe here: OEP-45 :: ADR-002: Deploying Open edX on Kubernetes Using Helm by bradenmacdonald · Pull Request #372 · openedx/open-edx-proposals · GitHub

We looked at the ticket very briefly during the meeting. I posted an update on the ticket just now. IMHO this one is not urgent so there is no need to decide now while some of the questions of scope are still a bit fuzzy. I’m hoping we’ll get more context to make such a decision in the future after other pieces have fallen into place.

Yes we discussed it and @lpm0073 is still planning to do it.

We looked at it briefly and decided to follow up async so we have more time to consider it.

I’ll reply there.

2 Likes

FYI I have started drafting a page to describe our group at:

https://openedx.atlassian.net/wiki/spaces/COMM/pages/3655008783/Large+Instances+-+DevOps+Sub-Working+Group

It’s a draft, so comments/edits are welcomed!

See also the next steps after that, to make it a formal working group.

1 Like

Hey everyone. Here’s the notes for Devops WG meeting held at 17:00 UTC on 2023-02-07.

Thank you to all the attendees. Recording to follow.

Cliff notes:

  • Reviewed the outstanding tasks and worked on next steps.
  • Confirmed that the ADR is complete, just needs final approval from a CC.
  • Wiki page for the group has been created, feedback welcome.
Meeting notes

Meeting start

  • Xavier nominates, Felipe as the meeting leader in Braden’s absence.

Going through list of issues.

Moises

  • Liland knight commented on the Nginx + Cert manager.
  • He was outside the community and was exploring deploying Open edX through the Helm chart.
  • Left a comment about their deployment process and ideas on deploying the subcharts.

Felipe

  • Can the work still continue? Is there anything left.

Moises

  • Yes, just need to work through Braden’s comments.

Xavier

  • So next Braden needs to look at the PR?

Moises

  • Yes, or someone else can take a look.

Xavier

  • Are there blockers, cause there are a lot of comments?

Moises

  • Not really, it can go through after the next review.

Felipe

Jhony

  • Last meeting the comments on the task were checked.
  • Decided to step forward, there are two ways to scale. Pods and Nodes.
  • PR was created in the repo, although couple of dependencies are required.
  • Still a work in progress though.
  • Hopes to add a couple more details in the next few days.
  • Will add HPA, but not BPA based on existing plugin.
  • Although, we can merge both in one plugin, it will need to be configured for each instance deployed with tutor.
  • Node autoscaling hasn’t started yet, but hope to work on cluster autoscaling solution.
  • From that point, we can then implement Karpenter.
  • But will start with tool that works across cloud providers.

Felipe

  • Next steps?

Jhony

  • PR needs to be moved from Draft to Review and then prroceed with the cluster autoscaling implementation.

Felipe

Lawrence

  • No work done yet, it only makes sense right now to dive in.
  • He volunteered to add Karpenter, but Jhony has done most of the work required.
  • The big one was metrics-server, which he added.
  • Now he can get started with Karpenter.

Felipe

  • Can it be turned on and off?

Lawrence

  • Yep. Can be turned off using Helm.

Felipe

  • It only supports AWS, so as long as it has a toggle it’s fine.

Lawrence

  • The long term vision of Karpenter is that it’s multi-platform.
  • It was started by AWS, so is AWS-centric at the moment.
  • Eventually it might grow to work on GCP and Azure as well.

Felipe

  • Isn’t there a competing project to it that supports other providers?

Lawrence

  • No idea.

Felipe

  • It’s fine, but we should look to have wider cloud provider support.

Jhony

  • Cluster Autoscaler is the only alternative. AFAIK Azure has a built-in autoscaling solution.

Lawrence

  • Pretty much everyone has a solution that scales compute nodes.
  • Karpenter is interesting because it’s able to acquire compute nodes at spot pricing.
  • They cost 30% less than on-demand prices.
  • To take advantage of it in AWS, you need Karpenter.
  • Does other platforms even have spot pricing? Not sure.

Xavier

  • How is Karpenter related to Prometheus and Grafana? Is Karpenter required for any part of it?

Lawrence

  • It’s an incremental improvement for scaling.
  • There are three types, horizontal, vertical and compute nodes.
  • In Kubernetes itself, there are resources for autoscaling of nodes.
  • If you do all that, then Karpenter helps with costs in getting you the same compute for less money.

Xavier

  • How does it relate to Prometheus + Grafana. Is it needed for that?

Lawrence

  • Not really.

Felipe

  • This is good, because if you’re not on AWS, you can just disable Karpenter.

Xavier

  • Side note, how we can link the work right to things that are relevant to 2U?
  • At Jeremy Anything that would be interesting to them?
  • Would be useful to identify concrete work items.

Jeremy

  • So far the work here is relatively tangential to the work that 2U’s doing.

Xavier

  • Anything that would be directly relevant?

Jeremy

  • Doesn’t quite have insight into SRE teams.
  • Would like to figure out how orchestration can work between prod and dev.
  • But not 100%, depends on Adam.

Moises

  • The current work with DevX. Is there anything to test at the moment?

Jeremy

  • Discovery has been completed, Coder, Devspace and Tilt.
  • Work is currently stuck with a rubric comparing those against each other.
  • Trying to decompose features that they want from dev environment, to which ones are just conventions.
  • What is linked to service orchestration, docker-compose or k8s, deploying. And how they would integrate with devstack and tutor.
  • Still working on it, haven’t had the time yet.

Moises

  • Not too clear.

Jeremy

  • Will share the notes and the spreadsheet again.

Felipe

Keith

  • Discussed implementation. Work that’s left is to address Braden’s comment and to fix the issue with patchStrategicMerge.
  • Once that’s finalised, this can be merged.

Andres

  • How is the issue with PV’s handled? Once a node is destroyed, the PV is destroyed with it.

Keith

  • Not handled yet, but plan to do some load testing to see the implications this sprint.
  • Hopefully it’ll not be an issue as ES is deployed as a StatefulSet, which means that each node has a copy of the index.

Moises

  • Question, do we need any more changes? Will the solution work for the past releases?

Keith

  • Deploying ES v7, with the intention of only targeting Olive for the moment. Once that works, can work on previous versions.

Felipe

  • Results of the poll?

Xavier

  • K8s-harmony is the chose name. How does that work for everyone?

Felipe

  • Went to Karpenter issue, but has already been mentioned.
  • Waiting on approval for the ADR to be merged.
  • Status has been provisional, feedback has been positive. Should be ready to merge this.

Andres

  • The title is not representative of the work. We’re not creating a Helm chart for Open edX, but are installing the supporting components.

Felipe

  • We haven’t yet decided what the limit for this project is. We might eventually having Helm charts for the Open edX space.
  • It hasn’t been necessarily specified and depends on our success with this project.

Andres

  • Most things can work through tutor. Tutor gets in the way, because it’s concerned with only a single instance.
  • We should be clear if we’re replacing Tutor or using Helm to complement what tutor does.

Felipe

  • We’re not replacing tutor. Will still manage to use Tutor and it’s plugin ecosystem.

Andres

  • Cool, then the title is just confusing.

Xavier

  • Andres to comment so that ADR can be approved and merged.
  • To Felipe: Are you just waiting for someone to review. Can you not merge as Arbiter?

Felipe

  • Has to wait for someone’s approval. Waiting on anyone with CC status to approve so that he can merge.

Xavier

  • You’re well-placed to give that approval.

Felipe

  • Would just like someone to sign off of this. To get the sense that there’s some consensus in the community.

Xavier

  • Do we have someone that’s a CC that could possible do it?

Felipe

  • Will just ping someone to do it. Just needs approval.

Xavier

  • Individual pings are best, will check with other CC’s if they can approve.
  • Does anyone from the group, want to be a CC on this? Keith/Lawrence?

Keith

  • Not really, but will consider it in future.

Lawrence

  • Excited to help, with some guidance to get started.

Xavier

  • All we need is the willingness required to work on it.
  • There is a nomination process showing contributions to the community.
  • Work on this project is a great angle to get started as it’s useful to the community.
  • Extends to others as well, if they want to be CC.

Felipe

  • Who else is contributors to this repo besides Braden and Him?

Xavier

  • Gabor is also part of it.

Felipe

  • Great, will see if anyone else can be added.
  • Andres to comment on title change, before merging.
  • #12, registering as formal working group.

Xavier

  • There’s been some progress on this. Opened JIRA, followed the usual format.
  • Went through the wiki page in JIRA
  • There are a couple things related to project leadership.
  • Happy to help with blockers and project management stuff.

Felipe

  • Prefers work to PM stuff. If this is what needs to be done, happy to take role.
  • Both Braden and him finding less and less times to get into details of PR.
  • Might not be the best reviewer, but will be able to point to folks that are best placed to make the review.

Jhony

  • Conference talk was accepted. Will talk with OC with regards to Kubernetes information.

Xavier

  • Are you gonna focus on a specific use-case or what the group has been doing? Anything else?

Jhony

  • Included details on installations they’re supporting right now.
  • They’ve got a large one, medium one, small one.
  • Will share summary of talk.

Felipe

  • Will share some actual number regarding the number of nodes/number of users.
  • Removing the uncertainty for folks that haven’t yet jumped into the lake.

Xavier

  • Braden gave some details on the numbers already yeah?

Felipe

  • Would be great for Jhony and Braden to collab not just on the numbers, but the impact of k8s.

Xavier

  • Will check with Braden on it.

Jhony

  • Will ask for co-speaker and for someone to expose some numbers and stuff.

Lawrence

  • Happy to volunteer to help.

Jhony

  • Great, will find presentation summary and then they can collaborate.

Xavier

  • Is there anything specific in #14 that needs to be planned for?

Jeremy

  • Go through the feature list and see what can be done wherever.
  • Just mapping out the features at the moment.

Felipe

  • Raised the issue that he doesn’t have the rights to close tickets in the repo.

Xavier

  • Maybe check on the ticket regarding rights to see if it can be restored?

Moises

  • Which place would be the best place to pose questions regarding deployments?

Xavier

  • Create a ticket for the project is the best way?

Andres

  • Maybe a category in the Open edX forums?

Xavier

  • There is a devops section. If we need a new category, please comment on the wiki page.

Good byes are said and meeting ends

1 Like

@keithgg Thank you for the recap!

Here is the recording:

And the chat logs:

Also note that I have created a new project board view for the project tickets:

So, here we go with the WhenIsGood link for the proposal the reschedule of the meeting:

1 Like

Hi everyone. These are the meeting notes for the working group meeting held on 21 Feb 2023 at 17:00 UTC.

Highlights and action items:

  • A project board has been created for tracking tasks.
  • There’s a poll on moving the meeting to a time more suitable that works for everyone.
  • @felipe to create issue on the conference topics and the best place to get feedback.

Attendees: @keithgg @Felipe @gabor @jmbowman @braden @mtyaka @sambapete @antoviaque @Maksim_Sokolskiy @jhony_avella gotcha.biniashvili

Full meeting notes

We had a new attendee, Gotcha Biniashvili from Midlink. Introductions from everyone.

Xavier added two notes to the agenda:

  • Gabor would like an earlier meeting
  • Discuss the collab at the conference

Felipe going through the list of issues:

  • #1 - Nginx cert manager. PR is ready to be merged but no rights.
  • #2 - Autoscaling - Jhony not available yet, but might join later.
  • #3 - Monitoring with Prometheus and Grafana. No movement yet. Lawrence was pinged.
  • #4 - Shared Elasticsearch is now ready for review. To ping Moises regarding it.
  • #5 - Karpenter. Should be low priority.
  • #8 - Formal review of proposal. Merged in provision status, but issue cannot be closed due to permissions.
    • Once something is implemented by external party then it can be merged.
  • #9 - Getting permissions for the repo. Someone is working on it seems.
  • #12 - Register as formal working group. Xavier was blocked on permissions.
    • Most everything is done.
    • Just need to do admin like assign good first issues, etc.
    • See comments on ticket.
  • #14 - Support for the cloud-based developer environment.
    • Still need to fix some minor issues on devstack
    • Work is progressing, but a bit slowly.
    • There are several things in the planning stages getting worked on at 2U
  • #15 - Conference presentation about Kubernetes on large instances.
    • Will reach out to Jhony if there’s some notes.
    • Braden will compile some numbers and send it off to Jhony.
    • Felipe will chat with Jhony to get it done.
  • #18 - Adjusting the references to tutor-multi

Jeremy

  • 2U is trying to move away from ElasticSearch.
  • Could be using MySQL text search, OpenSearch, etc.
  • There’s problems with the ES license, because some libraries don’t work with it if you’re not using proper Elasticsearch.

Xavier

Felipe

  • Next steps, should we move the meeting?

Gabor

  • Can the meeting be moved one hour earlier as it’s a bit late in his timezone?

Felipe

  • That would make this right after the CC meeting?

Pierre

  • Right in the middle for the meeting for the DevExp working group…

Xavier

  • Maybe we can find another time, putting it on another day?

Felipe

  • We can follow up with a poll to find a time that works for everyone.
  • Will do a whenisgood poll.

Xavier

  • Does one hour earlier work for you?

Braden

  • Yep.

Xavier

  • When creating poll focus on one hour earlier for Gabor and Braden so that it works for them.
  • Gabor to create poll.

Felipe

  • Group has been asked to propose some topics to discuss during the conference.
  • Need to propose someone for some sort of awards?

Xavier

  • It’s probably different from the topics we are discussing here.
  • It’s a meeting just between people in the DevOps working group.
  • Not sure on the agenda.

Felipe

  • The meeting pertains to the larger DevOps working group.
  • Since we are a sub-working group, we’re not sure if it applies to this group as well.
  • What’s the best place for Conference topics and get some async feedback?
  • Will create issue to discuss.

Xavier

  • Is there anything we want to achieve while face-to-face?
  • Is there anything we want to discuss with other groups that aren’t part of this.
  • Maybe other outcomes?

Felipe

  • Would like to discuss how this effort can tie in with work the other groups are doing?
  • Can we join into the larger effort of making this easier to deploy.
  • Once the issue is posted will add comment on the above.

Keith

  • Is leaving the working group to focus more on client work.
  • Will close out current tickets. Gabor as replacement.

Jhony

  • Is there comments on the autoscaling PR?
  • There are couple things.
    • tutor-contrib-autoscaler plugin created.
    • Gathers contributions made from different orgs.
    • Includes HPA and VPA. Needs some documentation improvements.
    • PR contains installation of metrics-server and VPA which are required for plugin.
    • It’s still a WIP at the moment. Got some feedback from Braden that’s been answered.

Braden

  • Can we have some default values for the plugin?

Jhony

  • We can maybe provide some suggestions based on installation size?

Felipe

  • We need to think about the cost as there’ll be presets on scaling up and down that users would like to know.

Gabor

  • We worked on the numbers that could be generally good based on the number of users etc.
  • It’s easy to come up with defaults, but it’s very dependent on usage.
  • We have the data, but not sure if it will be a good match for everyone.

Felipe

  • We will share data and see the costs.
  • They don’t care as long as there is an upper bound on what they can spend.

Gabor

  • Was eduNEXT able to tie the usage to the number of usage?

Felipe

  • They wrote a test script that simulates a user using https://k6.io/.
  • Then do simultaneous test and see how the instance’s behaves.
  • That way they’re able to get some number to get compute per user.
  • There’s too much variance otherwise.

Gabor

  • OpenCraft wasn’t able to do the same. Very curious about their approach.

Felipe

  • Doesn’t know if what they did was necessarily good, but works for them.
  • Could make the code public if there’s interest for someone else.

Jhony

  • It’s part of the roadmap to do this. Might come soon.

Maksim

  • Does anyone use load testing from repo or something else.
  • Did anyone use the legacy open edx repo.

Jhony

  • They’re using k6 which works very well.

Felipe

  • Anything else before closing off the meeting?

No real responses and meeting ends

1 Like

@gabor @keithgg Thanks! I’ve voted on the poll.

And here is the meeting recording:

Meeting chat logs:

Here are the results of the voting. It is a bit funny as it turns out that even the current time wouldn’t be good :smiley:

Thursday and Friday have times where only one person couldn’t make it. @jmbowman do you have any suggestions when would be the best for you?

Personally, if possible, I wouldn’t put the meeting on Friday, hence it’s the last day of the week and holidays can have impact on Friday meetings (ie. extended weekends).

That Thursday time slot is already double or triple booked for me: Deprecation WG, Frontend WG, FedX (2U frontend developers), and/or Arch-BOM daily meeting depending on the week. I’d routinely miss the beginning of the Friday time slot, but could join after the Arch-BOM sprint retro/planning meeting ends around 15 minutes in (if I reschedule another biweekly meeting that usually starts 15 minutes after that).

All of the “morning in Eastern time zone” time slots are in high contention for 2U folk both for the reason here (it’s the only good overlap with folk in Europe) and because 2U has a large South Africa office with essentially the same scheduling constraints. Most meetings scoped larger than a single team are in those same hours.

The time slot we’ve been using “works” in the sense that I can attend while eating lunch…

Today’s call notes: openedx-k8s-harmony-notes.txt (2.8 KB)