Deploying Open edX on Kubernetes Using Helm

[jmbowman (Jeremy Bowman)]

The Thursday 11am Eastern time slot is currently triple-booked for me every week even without moving this meeting there. And I have a weekly Friday 10-11:15 meeting also (that sometimes runs over), so I’d miss the beginning of the Friday 11am Eastern time slot also. But if I’m the only one those times don’t work for, maybe you should just move it and I can try to get someone else from 2U to attend?

[antoviaque (Xavier Antoviaque (OpenCraft) - opencraft.com/help)]

@lpm0073

Today’s call notes: openedx-k8s-harmony-notes.txt (2.8 KB)

Thanks for taking the notes! And here is the partial recording:

kubernetes_collaboration_20230308.mp4

Unfortunately I only thought of starting the recording at 2/3rd of the meeting, so the beginning is missing.

[antoviaque (Xavier Antoviaque (OpenCraft) - opencraft.com/help)]

Here is the recording for last week’s meeting:

kubernetes_collaboration_20230321.mp4

Chat log

00:02:53 Felipe Montoya: Notas compartidas - Google Docs
00:21:16 Adam Blackwell: I might be interested Adjusting the references to tutor-multi · Issue #18 · openedx/openedx-k8s-harmony · GitHub
00:23:17 Adam Blackwell: Keep it for now and if I dig in I’ll message Moises
00:25:49 Maksim Sokolskiy: The same for me - I will just open PR for #18 for some reference changes in case I’ll have some time.
00:30:35 Adam Blackwell: Lawrence, your audio is a bit quiet in case it’s easy for you to turn it up a bit.
00:32:33 Adam Blackwell: prometheus-metrics.installPrometheusServer → charts/stable/insights-agent at master · FairwindsOps/charts · GitHub
00:37:39 Braden MacDonald: DevOps Working Group · GitHub
00:37:43 Felipe Montoya: DevOps Working Group · GitHub
00:45:53 Adam Blackwell: Those services are GitHub - openedx/credentials: Service hosting course and program certificates, GitHub - openedx/registrar: A service to facilitate learner-program enrollments. and GitHub - edx/portal-designer: A place to create and design new learner-portal instances. which were pretty easy to containerize, some key issues were our Jenkins jobs.
00:46:28 Adam Blackwell: My team may not agree with the “pretty easy” statement.
00:54:32 Adam Blackwell: app of apps vs applicationset
01:01:50 Adam Blackwell: Thank you for the context! I have to drop off for another call, but if anyone has 2U SRE questions, I’m available in the Open edX Slack.

Notes

@braden @Felipe Was there a scribe taking notes for that one?

[Felipe (Felipe Montoya)]

Yes. I was taking notes for the session. The published notes are in confluence: https://openedx.atlassian.net/wiki/spaces/COMM/pages/3708813313/Large+Instance+Meeting+Notes+21.03.2023

[antoviaque (Xavier Antoviaque (OpenCraft) - opencraft.com/help)]

Here are the notes for the working group meeting today: https://openedx.atlassian.net/wiki/spaces/COMM/pages/3720642563/Meeting+notes+-+2023-04-04

Video recording:

kubernetes_collaboration_202211.mp4

[braden (Braden MacDonald (OpenCraft) - opencraft.com/help)]

Here are the notes from today’s meeting: https://openedx.atlassian.net/wiki/spaces/COMM/pages/3734437889/Large+Instance+Meeting+Notes+18.04.2023

[antoviaque (Xavier Antoviaque (OpenCraft) - opencraft.com/help)]

Thanks @braden !

Meeting recording

kubernetes_collaboration_20230418.mp4

Chat log

00:03:27 Felipe Montoya: https://openedx.atlassian.net/wiki/spaces/COMM/pages/3734437889/Large+Instance+Meeting+Notes+18.04.2023
00:04:16 Felipe Montoya: DevOps Working Group · GitHub
00:08:52 Felipe Montoya: Open edX Community Wiki

[braden (Braden MacDonald (OpenCraft) - opencraft.com/help)]

Meeting notes from today: https://openedx.atlassian.net/wiki/spaces/COMM/pages/3745808401/Large+Instances+Meeting+Notes+2023-05-02

[antoviaque (Xavier Antoviaque (OpenCraft) - opencraft.com/help)]

The video recording of the meeting:

kubernetes_collaboration_20230502.mp4

Chat logs:

00:04:34	Braden MacDonald:	Open edX Community Wiki - notes
00:21:34	jhony:	edx-search PR: feat: updating edx-search version to 3.5.0 by jfavellar90 · Pull Request #32152 · openedx/edx-platform · GitHub
00:21:44	Maksim Sokolskiy:	thanks
00:27:09	Braden MacDonald:	Helm | Chart Tests

[regis (Régis Behmo)]

I hope to be able to join the meeting tonight. If I make it, I’d like to showcase some of the improvements to Docker image build times that will land in Palm, next month: feat: upgrade to Palm by regisb · Pull Request #827 · overhangio/tutor · GitHub This is only tangentially related to the day-to-day issues of this working group, but the matter was initially raised in the early conversations that were held here.

[antoviaque (Xavier Antoviaque (OpenCraft) - opencraft.com/help)]

Video recording of yesterday’s meeting:

kubernetes_collaboration_20230516.mp4

Chat logs

00:03:41 Braden MacDonald: Open edX Community Wiki
00:12:36 Régis Behmo: feat: upgrade to Palm by regisb · Pull Request #827 · overhangio/tutor · GitHub
00:44:54 Régis Behmo: I need to go. Good day ya’ll!

[braden (Braden MacDonald (OpenCraft) - opencraft.com/help)]

Meeting notes from today: https://openedx.atlassian.net/wiki/spaces/COMM/pages/3771203649/Large+Instances+Meeting+Notes+2023-05-30

Note: In general, all meeting notes can be found from the Confluence page: https://openedx.atlassian.net/wiki/spaces/COMM/pages/3655008783/Large+Instances+-+DevOps+Sub-Working+Group#Pages

[antoviaque (Xavier Antoviaque (OpenCraft) - opencraft.com/help)]

And the video recording: kubernetes_collaboration_20230531.mp4 - Google Drive

[antoviaque (Xavier Antoviaque (OpenCraft) - opencraft.com/help)]

Meeting notes from yesterday’s meeting:

https://openedx.atlassian.net/wiki/spaces/COMM/pages/3787292690/Large+Instances+Meeting+Notes+2023-06-13

Video recording:

kubernetes_collaboration_20230614.mp4

And the chat logs:

00:06:33 Adam Blackwell (he/him/his): GitHub - argoproj-labs/argocd-vault-plugin: An Argo CD plugin to retrieve secrets from Secret Management tools and inject them into Kubernetes secrets
00:06:40 Adam Blackwell (he/him/his): Argo CD Vault Plugin
00:08:02 Felipe Montoya: @moises you are refering to Introduction - External Secrets Operator ?
00:09:10 Moisés González: Yes that one
00:11:55 Adam Blackwell (he/him/his): Kubernetes logs | Vector documentation ?
00:15:10 Moisés González: Sorry I got a spotty connection. Can you write the last thing you said Adam?
00:15:20 Adam Blackwell (he/him/his): I appreciate your appreciation
00:15:37 Adam Blackwell (he/him/his): Axim if hoping to put their eggs in the harmony chart(s)
00:17:49 Jeremy Bowman: Is this relevant to the topic at hand? GitHub - openobserve/openobserve: 🚀 10x easier, 🚀 140x lower storage cost, 🚀 high performance, 🚀 petabyte scale - Elasticsearch/Splunk/Datadog alternative for 🚀 (logs, metrics, traces, RUM, Error tracking, Session replay).
00:27:12 Adam Blackwell (he/him/his): Minimum 3, but the 1.25 EKS upgrades did break a bunch of v1beta things.
00:27:27 Adam Blackwell (he/him/his): I believe only for scheduled cronjobs though, so it didn’t impact learners.
00:28:29 Maksim Sokolskiy: GitHub - aulasneo/tutor-contrib-hpa
Did you use this?
00:30:04 jhony: or GitHub - eduNEXT/tutor-contrib-pod-autoscaling: This repository aims to provide support for HPA and VPA in tutor
00:30:49 Adam Blackwell (he/him/his): We use ls :facepalm:
00:31:24 Adam Blackwell (he/him/his): We want to use helm hooks for mysql migrations
00:32:42 Jeremy Bowman: Need to drop for another meeting
00:34:47 Adam Blackwell (he/him/his): +1
00:36:12 Adam Blackwell (he/him/his): We default our Django services to: resources:
limits:
cpu: 100m
memory: 512Mi
requests:
cpu: 25m
memory: 512Mi
00:38:32 Adam Blackwell (he/him/his): Bad liveness probes can hurt people*
00:38:59 Moisés González: We can atleast have the discussion
00:41:31 Adam Blackwell (he/him/his): For us, notes uses /heartbeat, most other services use /health, we have a `liveness_probe_initial_delay_seconds and liveness_probe_initial_delay_seconds var exposed in the currrrent django helm chart.
00:42:19 Adam Blackwell (he/him/his): (For the curious, GitHub - edx/portal-designer: A place to create and design new learner-portal instances. is public but not part of Open edX)
00:46:46 Adam Blackwell (he/him/his): I would love a better way to do performance testing on a Django service before we move things from EC2 to EKS, right now we have teams write smoke test runbooks, but they are incomplete.
00:48:42 Adam Blackwell (he/him/his): latency under load
00:49:09 Adam Blackwell (he/him/his): We at once point would scale up manually before major news announcements.
00:49:27 Maksim Sokolskiy: ++
00:49:40 Adam Blackwell (he/him/his): That also relates to bad liveness probes which slow down scaling
00:49:57 Felipe Montoya: We ask customers to let us know if they will do big announments to be able to judge if a manual scale up is necessary
00:51:52 Felipe Montoya: k6
00:54:35 Moisés González: https://k6.io/
00:54:42 Lawrence McDaniel: I have a hard stop in 5 minutes
00:54:56 Adam Blackwell (he/him/his): Sorry for derailing the conversation.
00:55:18 Gábor Boros: Same here about the 5mins
01:03:31 Adam Blackwell (he/him/his): Have to drop off for another meeting, but wanted to thank all of you for all of your inspiring collaboration!
01:03:53 Braden MacDonald: Reacted to “Have to drop off for…” with

[antoviaque (Xavier Antoviaque (OpenCraft) - opencraft.com/help)]

Meeting notes: https://openedx.atlassian.net/wiki/spaces/COMM/pages/3801448449/Large+Instances+Meeting+Notes+2023-06-27

Video recording: kubernetes_collaboration_20230627.mp4 - Google Drive

Time	Name	Message
00:14:13	Braden MacDonald:	https://openedx.atlassian.net/wiki/spaces/COMM/pages/3801448449/Large+Instances+Meeting+Notes+2023-06-27
00:39:51	Braden MacDonald:	https://discuss.openedx.org/
00:51:28	Felipe Montoya:	Adding Celery Beat as a scheduler to the core offering - #9 by Alecar?

[antoviaque (Xavier Antoviaque (OpenCraft) - opencraft.com/help)]

Meeting notes: https://openedx.atlassian.net/wiki/spaces/COMM/pages/3814457347/Large+Instances+Meeting+Notes+2023-07-11

Video recording: kubernetes_collaboration_20230711.mp4 - Google Drive