Hi community! First time posting
We wanted to share our deployment runbook for our recent upgrade from Redwood → Ulmo in a k8s cluster, hopefully it can be of help to others 
# Ulmo Upgrade: Production Deployment Runbook
**Applies to:** Open edX operators deploying the [Ulmo](https://docs.openedx.org/projects/openedx-releases/) named release
**Audience:** Platform engineers, site reliability engineers, and Open edX operators
**Estimated total window:** ~6–8 hours active work + up to 5 hours background reindexing
---
## Goals
- Minimize student-facing downtime
- Ensure a safe rollback path before any destructive changes
- Avoid data loss across all datastores
- Clearly communicate maintenance to users to reduce unnecessary support tickets
- Coordinate with any downstream teams that consume platform data (e.g. analytics pipelines)
---
## Prerequisites
Before scheduling the deployment window, confirm:
- [ ] You have access to create database snapshots / backups in your environment
- [ ] You have `kubectl` access to your production Kubernetes cluster
- [ ] You can run `tutor k8s apply` and `tutor k8s init -l <label>` commands selectively against your production cluster
- [ ] Your load balancer (ALB, nginx, etc.) supports returning a custom 503 maintenance page
- [ ] Any downstream data consumers (analytics, data warehouse) are notified and have a plan to pause/resume their pipelines
- [ ] A rollback tag or snapshot has been created from the current production state
---
## Roles
| Role | Responsibility |
|------|---------------|
| **Deployment lead** | Coordinates the window, runs deployments, tracks progress |
| **DBA / infra engineer** | Creates DB snapshots, runs truncate commands |
| **Platform engineer** | Manages load balancer rules, validates smoke tests |
| **Data consumer contact** | Pauses/resumes any downstream data pulls |
> Many steps require elevated permissions (snapshot creation, load balancer rule changes). Ensure the right people are on-call before starting.
---
## Step-by-Step Deployment
### Phase 1 — Preparation
#### 1.1 Create a rollback tag
Before making any changes, tag your current production deployment ref so you can roll back cleanly.
```bash
git tag rollback-pre-ulmo
git push origin rollback-pre-ulmo
```
---
#### 1.2 Open a change record (if applicable)
If your organization uses a change management process, open and start your change record now.
---
#### 1.3 Open a monitoring maintenance window (if applicable)
If your organization uses an observability platform (Datadog, Dynatrace, Grafana, etc.), create a maintenance window to suppress expected alerts during the deployment window.
---
#### 1.4 Disallow student access (maintenance mode)
> **Do this before starting any backups** to ensure a consistent snapshot.
Add a rule to your load balancer (ALB, nginx, etc.) that returns a `503` with a user-friendly maintenance message for all student-facing traffic.
**Example nginx snippet:**
```nginx
location / {
return 503;
}
error_page 503 /maintenance.html;
location = /maintenance.html {
root /var/www/html;
internal;
}
```
**Example AWS ALB listener rule:**
Add a fixed-response rule with HTTP 503 and a plain-text or HTML message body, positioned above all other rules.
Your maintenance message should clearly explain that the platform is undergoing scheduled maintenance and provide an expected return time if known.
> **Estimated time:** ~10 minutes
> **Permission level:** Requires load balancer edit access
---
#### 1.5 Backup MySQL
Create a snapshot of your MySQL / Aurora MySQL cluster **before** running any migrations.
- For AWS RDS/Aurora: create a manual cluster snapshot via the console or CLI.
- For self-hosted MySQL: use `mysqldump` or your standard backup tooling.
```bash
# AWS CLI example
aws rds create-db-cluster-snapshot \
--db-cluster-identifier <your-mysql-cluster-id> \
--db-cluster-snapshot-identifier ulmo-pre-upgrade-$(date +%Y%m%d)
```
> **Estimated time:** ~1 hour
> **Permission level:** Requires RDS snapshot creation access
> **Final backup size:** 1416 GB
Tip: do incremental backups by running a backup early, next backups will be way faster!
---
#### 1.6 Backup PostgreSQL (Superset / Cairn)
If your deployment includes Superset or Cairn, dump the PostgreSQL database before proceeding.
```bash
pg_dump --verbose \
-h <your-postgres-host> \
-U <db-user> \
-d <db-name> \
-C \
-f backup-superset-$(date +%Y%m%d).sql
```
Retrieve credentials from your secrets manager (AWS Secrets Manager, Vault, environment config, etc.).
> **Estimated time:** ~15 minutes
> **Permission level:** Requires read access to the PostgreSQL host from your local machine or a bastion
> **Final backup size:** 270 MB
---
#### 1.7 Backup MongoDB
Run `mongodump` against a secondary read replica to avoid impacting the primary. The recommended approach streams the archive directly to object storage in a single command, avoiding local disk usage.
**Access the replica node** using your preferred method — AWS Session Manager, SSH bastion, or equivalent. Using a terminal multiplexer like `screen` or `tmux` is strongly recommended so the backup continues if your session is interrupted.
```bash
# 1. Switch to the service user
sudo su - ec2-user
# 2. Start a screen session as a safeguard against session disconnection
screen
# 3. Retrieve the MongoDB password from Secrets Manager, run mongodump,
# and stream the archive directly to S3 in one pipeline.
#
# Adjust: secret ARN, host, database name, auth user/db, S3 bucket/path,
# and --expected-size (see note below).
AWS_DEFAULT_REGION=<your-aws-region> \
aws secretsmanager get-secret-value \
--secret-id <your-mongo-secret-arn> \
| jq --raw-output .SecretString \
| jq -r .password \
| mongodump \
-h localhost \
-d <your-database-name> \
-u <your-mongo-admin-user> \
--authenticationDatabase admin \
--archive \
| aws s3 cp - \
s3://<your-backup-bucket>/mongobackups/mongo-prod-dump-$(date +"%Y%m%d%H%M").archive \
--expected-size <expected-size-in-bytes>
# 4. After the dump completes, restart the mongod service.
# MongoDB 4.x has a known memory leak after running mongodump on large
# databases. Restarting prevents a slow instance or failures on future runs.
# We haven't tested this in newer versions, but we wanted to play safe :)
sudo service mongod restart
```
**Setting `--expected-size`:**
This flag tells the AWS CLI the approximate upload size so it can calculate multipart chunk sizes correctly. If the dump is larger than expected you will get an error like:
```
An error occurred (InvalidArgument) when calling the UploadPart operation:
Part number must be an integer between 1 and 10000, inclusive
```
If you see this, increase `--expected-size`. Calculate the value in bytes:
```
# Example: 800 GB
1024 * 1024 * 1024 * 800 = 858993459200
```
Monitor your actual archive sizes after each run and update the value if your database grows significantly.
> **Estimated time:** ~80 minutes
> **Permission level:** Requires shell access to a MongoDB secondary replica node (e.g. via AWS Session Manager or SSH)
> **Final backup size:** 747.4 GB
---
#### 1.8 Backup ClickHouse (if using Cairn/Aspects)
If your deployment includes ClickHouse (via Cairn or Aspects):
1. **Before backing up**, verify that the event ingestion queue (e.g. SQS or equivalent) has fully drained into ClickHouse. Check that:
- The age of the oldest message in the queue is `0`
- The number of messages received by the ClickHouse subscriber is `0`
- This typically takes only a few minutes; wait and re-check if needed.
2. Run the ClickHouse backup job
Notify any downstream analytics consumers that a backup is in progress.
> **Estimated time:** ~45 minutes
> **Permission level:** Requires `kubectl` access to the production cluster
> **Final backup size:** ~900 GB
---
#### 1.9 Truncate social auth tables in MySQL
These tables contain transient session/auth data that can cause migration to delay significantly. They are safe to truncate before the upgrade.
```sql
-- Connect to your MySQL openedx database
mysql -A -h <your-mysql-host> -u <db-user> -p
USE openedx;
TRUNCATE TABLE social_auth_usersocialauth;
TRUNCATE TABLE social_auth_partial;
TRUNCATE TABLE support_historicalusersocialauth;
```
Retrieve credentials from your secrets manager.
> **Estimated time:** ~5 minutes
> **Permission level:** Requires MySQL write access
---
### Phase 2 — Deployment
The deployment is broken into multiple passes to handle init scripts safely. Each pass runs a specific subset of `tutor k8s init` jobs so that migrations, schema changes, and reindexing happen in a controlled sequence rather than all at once.
> **Note on CMS reindexing:** By default, `tutor k8s init -l cms` triggers a full course reindex which can be very time-consuming. During the upgrade passes we skip the CMS reindex (via a Tutor plugin) and defer it to a dedicated background pass at the end.
> **Note on MongoDB:** The first three passes should use a temporary deployment branch/config that sets MongoDB to connect to a **single instance** (not a cluster). The final passes switch back to the normal replica-set configuration.
---
#### 2.1 First deployment — no init scripts (temporary branch)
> ⚠️ **Warning:** If your load balancer (e.g. ALB) is managed by Kubernetes (via Ingress resources or a controller) - which was our case, this deployment will reconcile the Ingress and restore the original listener rules — effectively removing your maintenance page and re-exposing the platform to students. Make sure your maintenance-mode strategy accounts for this.
Deploy the Ulmo release images. **Do not run any `tutor k8s init` commands on this pass.** Only apply the Kubernetes manifests:
```bash
tutor k8s apply --selector "app.kubernetes.io/component notin (job,namespace)"
```
This pass pulls the new images and runs container/service startup without executing database migrations or reindexing.
We wait for deployment to stabilize (no new pods are constantly created / terminated).
> **Estimated time:** ~15 minutes
---
#### 2.2 Second deployment — Open edX init only (temporary branch)
Run the Open edX initialization jobs. This executes Django management commands and database migrations.
```bash
tutor k8s init -l mysql
tutor k8s init -l lms
tutor k8s init -l cms # with CMS reindex disabled — see note above
tutor k8s init -l mfe
```
> **Estimated time:** ~20 minutes
---
#### 2.3 Third deployment — Cairn init only (temporary branch)
Run the Cairn/ClickHouse initialization jobs. This executes ClickHouse schema migrations.
```bash
tutor k8s init -l cairn
```
> **Estimated time:** ~20 minutes
---
#### 2.4 Fourth deployment — production branch, no init scripts
Switch back to your main production branch (with MongoDB replica-set config restored). Apply manifests only — **do not run any `tutor k8s init` commands.** This re-enables the MongoDB cluster connection and stabilizes the environment.
```bash
tutor k8s apply --selector "app.kubernetes.io/component notin (job,namespace)"
```
> **Estimated time:** ~5 minutes
---
#### 2.5 Fifth deployment — Meilisearch reindex (background, production branch)
Run the Meilisearch reindexing job. This can run in the background — student access can be restored before this completes.
```bash
tutor k8s init -l meilisearch
tutor k8s init -l cms # with CMS reindex enabled — see note above
```
> **Estimated time:** Up to 5 hours (background)
> Search functionality may be degraded or unavailable until reindexing completes.
---
#### 2.6 Update feature flags and platform settings
After deployments complete, apply any required configuration changes, feature flag updates, or waffle flag toggles documented in the Ulmo release notes.
> **Estimated time:** ~10 minutes
---
### Phase 3 — Post-Deployment
#### 3.1 Smoke testing and validation
Run your standard smoke test suite against production:
- [ ] LMS homepage loads
- [ ] Student login works
- [ ] Course catalog is accessible
- [ ] Course enrollment flow works
- [ ] Instructor dashboard accessible
- [ ] Learner dashboard accessible
- [ ] Any release-specific features from the Ulmo changelog are verified
> **Estimated time:** ~30 minutes
---
#### 3.2 Restore student access
Remove the maintenance rule from your load balancer to restore normal traffic routing. Verify that the maintenance page is no longer being served.
> **Estimated time:** ~10 minutes
> **Permission level:** Requires load balancer edit access
---
#### 3.3 Resume downstream data pipelines (if applicable)
If any downstream data consumers (analytics pipelines, data warehouses) paused for the deployment, coordinate with them to run a manual data pull and resume their normal scheduled process.
> **Estimated time:** ~90 minutes (coordinate with the relevant team)
---
#### 3.4 Close monitoring maintenance window
Disable the maintenance window in your observability platform so that alerts resume normally.
> **Estimated time:** ~1 minute
---
#### 3.5 Close change record
Mark your change record as complete with the outcome and any relevant notes.
---
## Rollback Procedure
If a critical issue is found during or after deployment:
1. Restore student-blocking rule on the load balancer (if access was already restored)
2. Roll back your deployment to the `rollback-pre-ulmo` tag
3. Restore MySQL from the pre-upgrade snapshot
4. Restore MongoDB from the pre-upgrade backup
5. Restore PostgreSQL from the pre-upgrade dump (if applicable)
6. Restore ClickHouse from the backup job output (if applicable)
7. Validate the rollback with smoke tests
8. Restore student access
9. Document the incident
---
## Reference Timings
| Phase | Step | Estimated Duration |
|---|---|---|
| Preparation | Create rollback tag | 5 min |
| Preparation | Enable maintenance mode | 10 min |
| Preparation | Backup MySQL | 1 hour |
| Preparation | Backup PostgreSQL | 15 min |
| Preparation | Backup MongoDB | ~80 min |
| Preparation | Backup ClickHouse | 45 min |
| Preparation | Truncate social auth tables | 5 min |
| Deployment | Pass 1 — no inits | 15 min |
| Deployment | Pass 2 — OEX init | 20 minutes |
| Deployment | Pass 3 — Cairn init | 20 minutes |
| Deployment | Pass 4 — production branch | 5 minutes |
| Deployment | Pass 5 — Meilisearch reindex | up to 5 hours (background) |
| Post-deployment | Smoke testing | 30 min |
| Post-deployment | Restore student access | 10 min |
| Post-deployment | Resume data pipelines | 90 min |
**Total active window (excluding background reindex):** ~7 hours
---
## Resources
- [Open edX named releases](https://docs.openedx.org/en/latest/community/release_notes/index.html)
- [Tutor documentation](https://docs.tutor.edly.io/)
- [Open edX community forum](https://discuss.openedx.org/)
- [Cairn / Aspects analytics plugin](https://github.com/openedx/openedx-aspects)
- [tutor-cairn plugin](https://github.com/overhangio/tutor-cairn)