Notes for the BTR portion of the meeting
Interests
- Common tool for deploying in k8s +5
- Automating deployments
- Testing blocker for Nutmeg
Nutmeg
- June 9th releaese date
- Need to upgrade every official tutor plugin
- mfe, notes, xqueue, indigo, minio, discovery, ecommerce (increasing difficulty order)
- Why? Plugin v1 API?
- v1 upgrade is not the reason we need to upgrade
- every Open edX release has breaking changes, and plugins need to upgraded thusly
- Topic on the tutor forum with instructions (LINK)
- roughly, for every plugin:
- bump the version, build the images, see if it works
- Regis could use help!
- Issues anywhere for these upgrades?
- roughly, for every plugin:
- Folks with unofficial plugins also need to upgarde
- Felipe: As maintainers of a tutor plugin, when/how should we upgrade?
- if you test with the new release and it works, bump the tutor version that the plugin depends on!
- Is tutor’s release already out?
- The is a WIP pull request – if you are upgrading a plugin, work off of that branch
- Testing for Nutmeg
- Dean is coordinating this- he has a comprehensive testing plan and leaderboard (LINK)
- Plugin testing in test plan?
- Testing blocker for Nutmeg
- Files and uploads interface - broken in Nutmeg branch
- studio-frontend
- Context: is react in studio, but not an MFE
- Felipe: EduNext has a fix for this on Lilac
- We should land this in master and backport it to Nutmeg ,
- Why isn’t this broken on edx.org?
- (Confirmation: it isn’t broken on edx.org)
- Phil will reach out
- First-time contributor found the solution
Kubernetes
- How are people doing it?
- OpenCraft built Grove
- Currently in testing
- Lawrence’s cookiecutter
- In use for ~6 large production platforms
- What could be done other than Jinja templating to standardize things across instances/repos
- A lot of maintenance to keep terraform working
- AWS is always changing… terraform is always changing
- Lot of work to keep these things up to date
- OC has terraform code that works for digitalocean. Not being bound to AWS be very nice.
- Some sort of infra-as-code that is not vendor-specific would be a powerful thing for the BTR to provide
- Contribution of manifests for different environments
- Tutor has a templating engine. Why have different tools with their own templating engines?
- Tutor uses Jinja2 to render its config environment
- Shared mainteancne of Tutor plugins for different vendors.
- Or: tutor-terraform plugin
- As a Terraform APIs changes, it can take a lot of time to get Terraform to plan & apply again
- couple different pieces: (1) underlying terraform to specify manifest files for different vendors, and (2) the terraform data sources on top of them.
- Sometimes spending hours of time to do something in Terraform that would take ~40mins to do by hand.
- Upside: An hour to build an environment from scratch.
- ^^ ends up being a wash when one person is doing it. Could be valuable if we are cooperating, but as an indvidual it’s not worth it
- DigialOcean is lighter weight and more stable, more amenable for terraform maintenance
- If enough of us were using AWS, would AWS be willing to reach out and help keep an AWS deployment of Open edX stable?
- This is happening / has happened - there is a reference deployment on AWS for Open edX that they’d like to have available.
- OpenCraft has not found terraform for AWS to be a huge maintenance burden. They keep as much of their Terraform in common between providers as possible, which helps.
- EduNext is building a k8s pipeline (Shipyard)
- In prodution for one site, in use for multiple staging sites
- How does it work?
- Uses Tutor build and test image
- Using Tutor to build image, but not using the
overhangioimages.
- Using Tutor to build image, but not using the
- Terraform to create clusters
- in-house templating engine to make manifest files to publish to GH, whatever
- ArgoCD syncs between manifests and infra
- Uses Tutor build and test image
- Felipe thanks folks for sharing their different pipelines
- Differences wrt other piplines: Needing to add boilerplate to different repositories vs. central management
- Grove
-
@fghaas 's tox pipeline
- Using for production pipelines
- OpenCraft built Grove
- We could keep doing our own different solutions, or we could collaborate. Or both! Common solution + everyone’s own spices and flavours
- Does a shared solution end up in the openedx organization, or another home? Who maintains it, decides how it evolves?
- Is there a common core that we can agree upon?
- Building support into Tutor itself?
- Regis is very excited about seeing all these projects on top of tutor
- but does not want to maintain a central solution himself
Grove & Tutor
-
Grove is trying to move as much of it’s work on top of Tutor as possible.
-
Move as much of Grove’s work into Tutor plugins as possible.
-
Moving instance-specific code to Tutor plugins as possible
-
OC & EduNext deploys multiple instances to the same cluster
- OC uses repository per-cluster.
- Grove has a UI for mapping instances ot clusters.
Codejail
- What is it?
- Safe execution sandbox
- Limits the libraries that Python can use
- Uses AppArmor for resource limiting
- Native Insatllation would do this in-process with edx-platform
- This is not amenable with k8s
- So, in Tutor, you need a micro-service. edx-platform calls out to the service, the service grades the code, and sends it back.
- tutor-codejail plugin handles this for Tutor
- App Armor needs to be added to kernels of servers separately, though.
- This can be done at the Docker level?
- Kinda, but the profile is loaded from the host.
- This can be done at the Docker level?
- App Armor needs to be added to kernels of servers separately, though.
How can tutor help you share, publicize you work?
- Regis’s current thinking:
- List of plugins, with links out to maintainers of those plugins
- There is a balance between individual provider’s differentiation vs. what we want to share as a community
- Grove is free and open
- Public and collaborative are not the same thing (source available vs. open source)
- Some folks just learned about Grove, Shipyard
- Lawrence is sharing his tool for collaboration, but would rather be contributing to a common project
Commonalities between these solutions
- K8s
- Terraform
- Different vendors, but lots of terraform logic is common between the different vendors
- What are the things everybody needs to do
- first it was dockerfiles,
- now k8s, and terraform
- these are different layers
- what layer do we want open vs not?
- We have Tutor for generating templated Dockerfiles, et al
- And K8s
- Let’s focus on making those good using this engine we have, before we go to far into other layers
- Build the map with all the holes, so folks can easily hook into those holes
- Cooking
- This might be its own space / group / meeting series
- And/or just write things down
- Or just opening PRs
- Lawrence would love to put his code in someone else’s project!
- Lists of plugins
- Demos of plugins
- At BTR
- Posted videos
- Public vs BTR?
- public
- Livestreams
- Sync questions can be posted
- Async questions can be on the forum or whatever
- Less work than videos sometimes - no editing, OK if things go wrong