Hi everyone,
We are migrating from Cairn to Aspects and would appreciate feedback on our approach before proceeding with production.
Our setup
- Open edX was previously running Quince with Cairn and now runs Ulmo with Aspects.
- We are using tutor-contrib-aspects 3.0.3.
- The deployment runs on OpenShift through Helm and ArgoCD.
- Historical data exists in Cairn’s ClickHouse and in tracking logs stored on persistent volumes (single tracking.log file).
- Single log file size is approx. 1 TB.
We understand that newer Aspects versions exist. Are there relevant fixes or migration tools in those versions, and what version would you recommend for Ulmo?
Migration approach
Our approach has been to:
- Enable live Aspects ingestion.
- Backfill current LMS data, such as course enrollments, course information and user information, using
dump_data_to_clickhouse. - Extract supported historical events from Cairn’s ClickHouse and transform them into the format expected by Aspects.
- Process tracking logs for periods not covered by Cairn.
- Import in bounded batches, checking event identifiers for duplicates and recording progress so interrupted imports can resume.
Issues we have encountered
Enrollment counts differ between current-state data and dashboards
For many courses, Django/MySQL and the latest enrollment snapshots in event_sink.course_enrollment showed 100 active enrollments, while the Course Dashboard’s “Current Enrollees” showed a lower count.
Our investigation suggests that the chart uses enrollment events rather than simply counting the latest active enrollment snapshots. Some historical activation events were absent from the sources we checked. (Present in logs but not in clickhouse or vice versa.)
For another course, we also found that comparing enrollments created after our migration cutoff with the dashboard was not a like-for-like comparison: the dashboard included some learners enrolled before that cutoff.
tracking.log records are malformed or missing
We found:
- Malformed or truncated JSON.
- Interleaved log records.
- Large NUL-containing spans in historical files.
- Enrollment changes present in the LMS database/audit history without a recoverable activation event in the log records inspected.
- Some intact events in application logs that were damaged or absent in
tracking.log.
Resource usage during backfill
We encountered ClickHouse memory pressure during ingestion, including significant memory consumption from diagnostic-table merges. This made us concerned about the operational requirements for processing production history while live ingestion continues.
Questions
-
Is this a recommended migration strategy?
Is extracting and transforming events from Cairn’s ClickHouse a supported approach, or is replaying the original tracking logs generally preferable? Is there an existing Cairn-to-Aspects migration tool? If not, will community benefit from such a tool if it existed ? -
What is the recommended workflow for approximately 1 TB of historical logs?
Should we usetransform_tracking_logs, transform to intermediate files/object storage, or use another supported ingestion path? How do you handle checkpoints, retries and resumption after a failure? -
How should overlap and duplicates be handled?
We potentially have the same activity in Cairn, Quince logs, Ulmo logs and live ingestion. Are event identifiers stable across these paths? What deduplication guarantees can we rely on, including in derived reporting tables? -
How should missing historical enrollment events be handled?
Is there a supported way to reconcile enrollment dashboards with current LMS enrollment state? Should “Current Enrollees” agree with current active enrollments, or does it intentionally represent a different population? -
What is the recommended treatment of malformed records?
Are there supported tools for quarantining invalid events, diagnosing transformation failures and retrying recoverable records without silently losing data? -
What production validation checks do you recommend?
Beyond raw event counts, how do you validate enrollment, learner/course sinks, problem attempts, video activity and derived dashboard results after a historical import? -
How should we size and operate the backfill?
Are there useful benchmarks or guidance for ClickHouse storage, memory, batch sizes and concurrency? How should historical imports be throttled to protect live ingestion?
Our goal is a repeatable migration procedure that operators can run independently in each environment(Prod, Dev, UAT), with explicit reporting of imported, duplicate, rejected and unrecoverable records.
If anyone has migrated a similarly sized installation from Cairn to Aspects, we would appreciate hearing what worked, what you would avoid, and which supported tools or documentation we should follow.
Thank you!