Attacking the Monolith by Extracting a Core

Disclaimer: This is just a thought I had recently, and is not any kind of official plan.

The edx-platform repo has a lot of stuff in it, and most folks share the belief that it should be simplified. We have various documents and talks where we outline the need for an “extensible core” model. Every so often, we manage to peel one or two bits of edx-platform out into its own repo, and sometimes new pieces of functionality are started outside of edx-platform entirely and pulled into edx-platform as a pip-installed requirement (like edx-when).

But edx-platform is enormous, DEPRs have a long cycle time, and this process has stretched on for years with no clear end in sight. We all want a sane, simplified core set of apps to extend. The question I had in my mind was: Does that subset have to be edx-platform? For instance, could we leave edx-platform as kind of a messy place, but extract out a few key pieces into an openedx-content-core that edx-platform also installs?

Some parts of the content code have tendrils everywhere, but many of them have minimal dependencies that could be shifted. In a number of places, we use the publish step to translate some obscure and complex XBlock edge cases into mostly boring Django data models that are much easier to work with and reason about.

For instance, with the learning_sequences app (intended to give you course outlines and other sequence metadata), the app itself defines an explicit set of data types in its data.py, and its api package gives a public set of methods with type annotations. But code in contentstore is responsible for extracting XBlock data from the modulestore and translating it into the simpler models in learning_sequences–catching cases like a sequence appearing in more than one section, or scanning the units of a sequence to determine whether enrollment track partition group settings should bubble up.

So what if we could extract out a set of apps with a simplified model of the world, and leave edx-platform as a kind of messy translation layer? Say we created an openedx-content-core repo that started with one or two of these, but gradually grew to contain the whole set:

  • Course Overviews
  • Course Settings
  • User Roles (course/library student/staff/etc.)
  • Learning Sequences
  • Scheduling
  • Unit Composition (what blocks are in this unit for a given user?)
  • User Partition Groups
  • Grades?

And say we made sure that as each piece was extracted, it was given a well documented public interface with the conventions specified in OEP-49. No static assets, no UI, maybe not even any REST APIs. Just the base building blocks for some of the core pieces of our platform, so that extensions are free to interact with them via in-process APIs.

If we had such a repo, then people developing extensions could import that in the same way that edx-platform would. An extension would pin versions that it supports, and tox would hopefully find the breaking error when some API call unexpectedly goes away with the latest update. When combined with the new work happening in openedx-events and openedx-hooks, could extensions be mostly developed without having to run edx-platform as a whole?

4 Likes

I am definitely supportive of this idea. What is a core though? What you had in that list, why would that be a single core?
As I imagine, those features you listed can arguably be their own libraries or even IDAs, right? Why can’t we just extract those one by one into their own repos, instead of a core set of functionalities?
That said, I do recognize things below are content related and may make sense to be together:

  • Learning Sequences
  • Scheduling
  • Unit Composition
    The following are what construct as a course, which is a special set of content context:
  • Course Overview
  • Course Settings
  • User Partition Groups
  • User Roles

All I am saying is, instead of a single core, we could have multiple sub functioning groups of features constructing into a largely coherent repository.
This way, edx-platform is nothing but a glorified HA account service.

For grading, I say that should be it’s own IDA and service (not gonna be a micro one at that)

To add to your last bolded question, the problem in my opinion is how to emulate the user (learners or course teams) behaviors interacting with these Cores repo. That is something I believe we can use more guidance from.

I agree on both counts:

  • taking action and bringing to fruition our thought exercises to-date on monolith and system boundaries (we are garnering organizational support on this).

  • extensible core does not imply a single core

Core versus extension

On the 2nd point, extensions and core are reference points in a distributed system. Referring to our architectural vision diagram for an extensible platform, we’re designing pluggability frameworks that allow extensions to any microservice and any microfrontend.

Taking Proctoring as a concrete example, Proctoring is a type of Exam that can extend the core Learning experience. From this vantage point, Proctoring is an extension to the core Courseware. On the other hand, from the perspective of Proctoring itself, Proctoring’s core framework supports multiple 3rd party Proctoring providers as extensions to its core API. So a single Proctoring component is both an “extension” to one reference point and “core” to another reference point.

(Perhaps this was obvious, but I wanted to clarify since I myself may have obfuscated DDD’s definition of “core” with the “core” we really mean when we say “extensible core platform”.)

Avoid entity-based services

As the Content theme pursues this path of exploration, I encourage you to refresh your memories of the Architecture Manifesto’s guidance on Bounded Contexts. In particular, avoid starting from the data models as a point of extraction. Instead, separate along the lines of “jobs-to-be-done” in order to avoid the anti-pattern of entity-based services.

For example, I would put Course Overviews, Course Settings, User Partition Groups as data models and storage mechanisms that may (or may not) come in for the ride or may be duplicated across multiple services once edx-platform is broken apart. I do not classify them as 1st class named citizens of services since they are data models and entities used to serve higher-order jobs.

If it would be helpful, we can facilitate an event-storming exercise specific to the Content theme’s domain to get a better sense of the “jobs” owned by the theme.

Yeah, that’s totally fair. I throw around the word “core” a little too casually. That’s why I sort of hand-waved it with the name “openedx-content-core”. I could see edx-platform broken apart into previously sketched bounded contexts, so that it looks more like:

  • User / Account / Roles
  • Authoring
  • Learning

The list I gave above is mostly on the Learning side of this. The reason I was focusing on extracting one such repo out was because I was fumbling around for a practical first cut that would yield an actual tangible benefit (e.g. accelerated extension development).

For some of these, I think that extracting them out to individual repos imposes more overhead than it’s worth. There’s some fixed cost that we incur on a per-repo basis: permissions, PyPI, OEP compliance tracking, CI setup, etc. That’s not worth it for some things that will never be used in isolation. Also, I feel like if we’re going to lift out a sane subset for the purposes of making extensions easier to build, then it should have a very straightforward versioning story. People shouldn’t have to worry about a dozen individual repos with separate versions that may have funny interactions with other versions.

It’s a balancing act, to be sure. Individual repos might still be spun off where things are unstable or ownership boundaries are strongly needed.

Iteration on edx-platform is slowed by a bunch of things. A byzantine mess of static asset compilation slows down the builds. Modulestore makes tests take 10X longer than they should. The list goes on. But at the same time, I don’t want to overcorrect towards full granularity either. Taking out modulestore usage, all the apps I listed above would have a test suite that runs in seconds. There’d be no static asset compilation step.

But it all comes back to how useful this would actually be to developing extensions. There needs to be a user-centric driver to prioritize what gets pulled out and what doesn’t in this model, and the lens I’m thinking of is “what removes the need to spin up edx-platform?”

I listed these things because in my mind, they’re low level building blocks that need to be pulled out in order to pull out some of the other things (e.g. you can’t have something that does unit composition without user partition groups–the implementation of mapping users to groups may be pluggable, but the data models to answer “what group is this user in for this partition?” should be brought along).

Right. I haven’t forgotten the arch manifesto dictates, particularly in terms of data duplication–things like User Partition Group configuration data and Course Settings currently canonically live in the modulestore, and I’m proposing these have their own self-contained models. I’m proposing a new facility to handle Course Settings, and it’s not just a raw data model–pulling config values for a course is a really common use case, and sometimes we’ll want that to come from the content directly, sometimes it’s a derived value from multiple settings, and sometimes we want the site operators to set overrides at a system level. CourseApps would probably fall into this category as well. I think that 90% of what Course Overviews does is better handled by something that handles settings specifically. The other major jobs of Course Overviews are to give the local view of discovery-style catalog information and to just have a table to hang foreign keys off of when it comes to ensuring your course key actually exists.

I think it’s a useful exercise, but I also don’t think that’s really the question I’m looking to answer right now. Or rather, it might be better as a second pass. I’ve been rambling a lot (my apologies), but I’ll try to summarize here:

Pain point I’m trying to address: edx-platform slows development of extensions.

Hypothesis/consensus: This is because edx-platform is too big, complex, slow, and poorly documented.

My interpretation of our approach to date: Gradually remove things from edx-platform and start development of new features outside of edx-platform, until what remains inside edx-platform is “core”.

Alternative suggestion: Shift the focal point of defining the core out of edx-platform itself. Instead of winnowing non-core elements from edx-platform, gradually extract the core elements from edx-platform into a new repo. Have extension developers work with that repo’s smaller subset instead. Have edx-platform import and make calls to the new thing.

Tactics: Pull out apps one by one. Make their data models independent of modulestore (e.g. shift to push data in during publish in certain places where it currently lazily loads). Combine or refactor apps as necessary (e.g. separation of settings management from catalog info in the Course Overviews app). Adapt them to fit our Python internal app and documentation conventions, but don’t radically change the interfaces if it makes adoption difficult. The new repo’s apps should have no runtime dependencies on anything in edx-platform for basic operation (though edx-platform would start defining some things as plugins for new-repo extension points, like extensions do).

Hypothetical Example: If the set of apps that I hand-waved were pulled out, we could have a management command that would just create a course locally, with no XBlock or studio dependencies, basically instantly (based on some template YAML file or something), and populate it with “lorem ipsum”-style dummy HTML for Unit rendering. You could start up a Django server that would offer all the APIs that the baseline version of the Courseware MFE would need to do its work. This would all be done from a Django project that imports this new openedx-learning-core lib, and not require edx-platform as a whole. I think that should be enough of a framing to develop CourseApps like inline discussions that don’t really care that much about all the crazy business rules of edx-platform.

What I’m looking for: I’m trying to figure out how to get to that accelerated extension development as quickly as possible, and test my assumptions. If we do all this and people still have to spin up edx-platform for day to day iteration on their plugin, then this kind of separation only slows development by adding one more thing. That’s why I want to start from the extension developer point of view and figure out what jobs need doing, by examining what’s already being done. Once we have a mapping of what’s being used and for what purpose, I feel like having a storming session where we categorize these into buckets makes a lot of sense. If that means this is three repos instead of one, that’s fine by me.

What I’m wary of: We’ve done large scale mapping exercises in the past, and they’ve been valuable to etch out the broad boundaries. But I want to stay tightly focused on the user needs, and I expect that our potential extension points follow some power law distribution of usefulness. Having a separate space to pull these core pieces into means that we don’t have to make the decision up front of sifting through all the things–we can go through the most important functionality needed by our developer users and elevate them one by one.

1 Like