# Attacking the Monolith by Extracting a Core **URL:** \ **Category:** Architecture\ **Created:** [July 12, 2021, 5:51am UTC](https://discuss.openedx.org/t/attacking-the-monolith-by-extracting-a-core/5383 "2021-07-12T05:51:05Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![dave](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/dave/32/263_2.png) [@dave](https://discuss.openedx.org/u/dave)\ **Post date:** [July 12, 2021, 5:51am UTC](https://discuss.openedx.org/t/attacking-the-monolith-by-extracting-a-core/5383/1 "2021-07-12T05:51:05Z")
Disclaimer: This is just a thought I had recently, and is not any kind of official plan. The edx-platform repo has a lot of stuff in it, and most folks share the belief that it should be simplified. We have various documents and talks where we outline the need for an “extensible core” model. Every so often, we manage to peel one or two bits of edx-platform out into its own repo, and sometimes new pieces of functionality are started outside of edx-platform entirely and pulled into edx-platform as a pip-installed requirement (like edx-when). But edx-platform is enormous, DEPRs have a long cycle time, and this process has stretched on for years with no clear end in sight. We all want a sane, simplified core set of apps to extend. The question I had in my mind was: Does that subset have to be edx-platform? For instance, could we leave edx-platform as kind of a messy place, but extract out a few key pieces into an openedx-content-core that edx-platform also installs? Some parts of the content code have tendrils everywhere, but many of them have minimal dependencies that could be shifted. In a number of places, we use the publish step to translate some obscure and complex XBlock edge cases into mostly boring Django data models that are much easier to work with and reason about. For instance, with the learning\_sequences app (intended to give you course outlines and other sequence metadata), the app itself defines an explicit set of data types in its [data.py](https://github.com/edx/edx-platform/blob/master/openedx/core/djangoapps/content/learning_sequences/data.py), and its [api](https://github.com/edx/edx-platform/blob/master/openedx/core/djangoapps/content/learning_sequences/api/ __init__.py) package gives a public set of methods with type annotations. But code in contentstore is responsible for extracting XBlock data from the modulestore and translating it into the simpler models in learning\_sequences–catching cases like a sequence [appearing in more than one section](https://github.com/edx/edx-platform/blob/master/cms/djangoapps/contentstore/outlines.py#L257-L269), or scanning the units of a sequence to determine whether enrollment track partition group settings should [bubble up](https://github.com/edx/edx-platform/blob/master/cms/djangoapps/contentstore/outlines.py#L106-L153). So what if we could extract out a set of apps with a simplified model of the world, and leave edx-platform as a kind of messy translation layer? Say we created an openedx-content-core repo that started with one or two of these, but gradually grew to contain the whole set: - Course Overviews - Course Settings - User Roles (course/library student/staff/etc.) - Learning Sequences - Scheduling - Unit Composition (what blocks are in this unit for a given user?) - User Partition Groups - Grades? And say we made sure that as each piece was extracted, it was given a well documented public interface with the conventions specified in [OEP-49](https://open-edx-proposals.readthedocs.io/en/latest/oep-0049-django-app-patterns.html). No static assets, no UI, maybe not even any REST APIs. Just the base building blocks for some of the core pieces of our platform, so that extensions are free to interact with them via in-process APIs. If we had such a repo, then people developing extensions could import that in the same way that edx-platform would. An extension would pin versions that it supports, and tox would hopefully find the breaking error when some API call unexpectedly goes away with the latest update. **When combined with the new work happening in openedx-events and openedx-hooks, could extensions be mostly developed without having to run edx-platform as a whole?** ---
**Author:** ![schenedx](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/schenedx/32/72_2.png) [@schenedx](https://discuss.openedx.org/u/schenedx)\ **Post date:** [July 12, 2021, 11:03pm UTC](https://discuss.openedx.org/t/attacking-the-monolith-by-extracting-a-core/5383/2 "2021-07-12T23:03:32Z")
I am definitely supportive of this idea. What is a core though? What you had in that list, why would that be a single core? As I imagine, those features you listed can arguably be their own libraries or even IDAs, right? Why can’t we just extract those one by one into their own repos, instead of a core set of functionalities? That said, I do recognize things below are content related and may make sense to be together: - Learning Sequences - Scheduling - Unit Composition The following are what construct as a course, which is a special set of content context: - Course Overview - Course Settings - User Partition Groups - User Roles All I am saying is, instead of a single core, we could have multiple sub functioning groups of features constructing into a largely coherent repository. This way, edx-platform is nothing but a glorified HA account service. For **grading** , I say that should be it’s own IDA and service (not gonna be a micro one at that) To add to your last bolded question, the problem in my opinion is how to emulate the user (learners or course teams) behaviors interacting with these Cores repo. That is something I believe we can use more guidance from. ---
**Author:** ![nimisha](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/nimisha/32/1511_2.png) [@nimisha](https://discuss.openedx.org/u/nimisha)\ **Post date:** [July 13, 2021, 12:44pm UTC](https://discuss.openedx.org/t/attacking-the-monolith-by-extracting-a-core/5383/3 "2021-07-13T12:44:07Z")
I agree on both counts: - taking action and bringing to fruition our thought exercises to-date on monolith and system boundaries (we are garnering organizational support on this). - extensible core does not imply a single core **Core versus extension** On the 2nd point, extensions and core are reference points in a distributed system. Referring to our [architectural vision diagram for an extensible platform](https://openedx.atlassian.net/wiki/spaces/AC/pages/922222913/Extensible+and+Pluggable+Platform), we’re designing pluggability frameworks that allow extensions to _any_ microservice and _any_ microfrontend. Taking Proctoring as a concrete example, Proctoring is a type of Exam that can extend the core Learning experience. From this vantage point, Proctoring is an extension to the core Courseware. On the other hand, from the perspective of Proctoring itself, Proctoring’s core framework supports multiple 3rd party Proctoring providers as extensions to its core API. So a single Proctoring component is both an “extension” to one reference point and “core” to another reference point. _(Perhaps this was obvious, but I wanted to clarify since I myself may have obfuscated DDD’s definition of “core” with the “core” we really mean when we say “extensible core platform”.)_ **Avoid entity-based services** As the Content theme pursues this path of exploration, I encourage you to refresh your memories of the [Architecture Manifesto’s guidance on Bounded Contexts](https://openedx.atlassian.net/wiki/spaces/AC/pages/1074397222/Architecture+Manifesto+WIP#Clear-Bounded-Contexts). In particular, avoid starting from the data models as a point of extraction. Instead, separate along the lines of “jobs-to-be-done” in order to avoid the anti-pattern of entity-based services. For example, I would put Course Overviews, Course Settings, User Partition Groups as data models and storage mechanisms that may (or may not) come in for the ride or may be duplicated across multiple services once edx-platform is broken apart. I do not classify them as 1st class named citizens of services since they are data models and entities used to serve higher-order jobs. If it would be helpful, we can facilitate an event-storming exercise specific to the Content theme’s domain to get a better sense of the “jobs” owned by the theme. ---
**Author:** ![dave](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/dave/32/263_2.png) [@dave](https://discuss.openedx.org/u/dave)\ **Post date:** [July 13, 2021, 4:53pm UTC](https://discuss.openedx.org/t/attacking-the-monolith-by-extracting-a-core/5383/4 "2021-07-13T16:53:35Z")
> [@schenedx](#): > > What is a core though? What you had in that list, why would that be a single core? Yeah, that’s totally fair. I throw around the word “core” a little too casually. That’s why I sort of hand-waved it with the name “openedx-_content_-core”. I could see edx-platform broken apart into [previously sketched bounded contexts](https://openedx.atlassian.net/wiki/spaces/AC/pages/663224968/edX+DDD+Bounded+Contexts), so that it looks more like: - User / Account / Roles - Authoring - Learning The list I gave above is mostly on the Learning side of this. The reason I was focusing on extracting one such repo out was because I was fumbling around for a practical first cut that would yield an actual tangible benefit (e.g. accelerated extension development). > [@schenedx](#): > > As I imagine, those features you listed can arguably be their own libraries or even IDAs, right? Why can’t we just extract those one by one into their own repos, instead of a core set of functionalities? For some of these, I think that extracting them out to individual repos imposes more overhead than it’s worth. There’s some fixed cost that we incur on a per-repo basis: permissions, PyPI, OEP compliance tracking, CI setup, etc. That’s not worth it for some things that will never be used in isolation. Also, I feel like if we’re going to lift out a sane subset for the purposes of making extensions easier to build, then it should have a very straightforward versioning story. People shouldn’t have to worry about a dozen individual repos with separate versions that may have funny interactions with other versions. It’s a balancing act, to be sure. Individual repos might still be spun off where things are unstable or ownership boundaries are strongly needed. Iteration on edx-platform is slowed by a bunch of things. A byzantine mess of static asset compilation slows down the builds. Modulestore makes tests take 10X longer than they should. The list goes on. But at the same time, I don’t want to overcorrect towards full granularity either. Taking out modulestore usage, all the apps I listed above would have a test suite that runs in seconds. There’d be no static asset compilation step. But it all comes back to how useful this would actually be to developing extensions. There needs to be a user-centric driver to prioritize what gets pulled out and what doesn’t in this model, and the lens I’m thinking of is “what removes the need to spin up edx-platform?” > [@nimisha](#): > > As the Content theme pursues this path of exploration, I encourage you to refresh your memories of the [Architecture Manifesto’s guidance on Bounded Contexts](https://openedx.atlassian.net/wiki/spaces/AC/pages/1074397222/Architecture+Manifesto+WIP#Clear-Bounded-Contexts). In particular, avoid starting from the data models as a point of extraction. Instead, separate along the lines of “jobs-to-be-done” in order to avoid the anti-pattern of entity-based services. I listed these things because in my mind, they’re low level building blocks that need to be pulled out in order to pull out some of the other things (e.g. you can’t have something that does unit composition without user partition groups–the implementation of mapping users to groups may be pluggable, but the data models to answer “what group is this user in for this partition?” should be brought along). > [@nimisha](#): > > For example, I would put Course Overviews, Course Settings, User Partition Groups as data models and storage mechanisms that may (or may not) come in for the ride or may be duplicated across multiple services once edx-platform is broken apart. I do not classify them as 1st class named citizens of services since they are data models and entities used to serve higher-order jobs. Right. I haven’t forgotten the arch manifesto dictates, particularly in terms of data duplication–things like User Partition Group configuration data and Course Settings currently canonically live in the modulestore, and I’m proposing these have their own self-contained models. I’m proposing a new facility to handle Course Settings, and it’s not just a raw data model–pulling config values for a course is a really common use case, and sometimes we’ll want that to come from the content directly, sometimes it’s a derived value from multiple settings, and sometimes we want the site operators to set overrides at a system level. CourseApps would probably fall into this category as well. I think that 90% of what Course Overviews does is better handled by something that handles settings specifically. The other major jobs of Course Overviews are to give the local view of discovery-style catalog information and to just have a table to hang foreign keys off of when it comes to ensuring your course key actually exists. > [@nimisha](#): > > If it would be helpful, we can facilitate an event-storming exercise specific to the Content theme’s domain to get a better sense of the “jobs” owned by the theme. I think it’s a useful exercise, but I also don’t think that’s really the question I’m looking to answer right now. Or rather, it might be better as a second pass. I’ve been rambling a lot (my apologies), but I’ll try to summarize here: **Pain point I’m trying to address:** edx-platform slows development of extensions. **Hypothesis/consensus:** This is because edx-platform is too big, complex, slow, and poorly documented. **My interpretation of our approach to date:** Gradually remove things from edx-platform and start development of new features outside of edx-platform, until what remains inside edx-platform is “core”. **Alternative suggestion:** Shift the focal point of defining the core out of edx-platform itself. Instead of winnowing non-core elements from edx-platform, gradually extract the core elements from edx-platform into a new repo. Have extension developers work with that repo’s smaller subset instead. Have edx-platform import and make calls to the new thing. **Tactics:** Pull out apps one by one. Make their data models independent of modulestore (e.g. shift to push data in during publish in certain places where it currently lazily loads). Combine or refactor apps as necessary (e.g. separation of settings management from catalog info in the Course Overviews app). Adapt them to fit our Python internal app and documentation conventions, but don’t radically change the interfaces if it makes adoption difficult. The new repo’s apps should have no runtime dependencies on anything in edx-platform for basic operation (though edx-platform would start defining some things as plugins for new-repo extension points, like extensions do). **Hypothetical Example:** If the set of apps that I hand-waved were pulled out, we could have a management command that would just create a course locally, with no XBlock or studio dependencies, basically instantly (based on some template YAML file or something), and populate it with “lorem ipsum”-style dummy HTML for Unit rendering. You could start up a Django server that would offer all the APIs that the baseline version of the Courseware MFE would need to do its work. This would all be done from a Django project that imports this new openedx-learning-core lib, and not require edx-platform as a whole. I think that should be enough of a framing to develop CourseApps like inline discussions that don’t really care that much about all the crazy business rules of edx-platform. **What I’m looking for:** I’m trying to figure out how to get to that accelerated extension development as quickly as possible, and test my assumptions. If we do all this and people still have to spin up edx-platform for day to day iteration on their plugin, then this kind of separation only slows development by adding one more thing. That’s why I want to start from the extension developer point of view and figure out what jobs need doing, by examining what’s already being done. Once we have a mapping of what’s being used and for what purpose, I feel like having a storming session where we categorize these into buckets makes a lot of sense. If that means this is three repos instead of one, that’s fine by me. **What I’m wary of:** We’ve done large scale mapping exercises in the past, and they’ve been valuable to etch out the broad boundaries. But I want to stay tightly focused on the user needs, and I expect that our potential extension points follow some power law distribution of usefulness. Having a separate space to pull these core pieces into means that we don’t have to make the decision up front of sifting through all the things–we can go through the most important functionality needed by our developer users and elevate them one by one. ---
**Author:** ![dave](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/dave/32/263_2.png) [@dave](https://discuss.openedx.org/u/dave)\ **Post date:** [August 9, 2021, 4:45am UTC](https://discuss.openedx.org/t/attacking-the-monolith-by-extracting-a-core/5383/5 "2021-08-09T04:45:56Z")
I’ve been poking at this idea a bit more and trying to incorporate @nimisha’s feedback about avoiding entity-based services. My latest stab at the list of apps looks something like: `navigation`Sequence and unit metadata, outlines, etc. How do you get to the particular piece of content that you need to learn from.`policy`Content-related settings as they apply to the site as a whole, organizations, and individual courses.`publishing`Centralized list of Learning Contexts (e.g. Courses, Libraries), their published versions, and various content-related errors and warnings are associated with them.`unit_composition`What permutation of a single unit does this user see? This would handle things like A/B tests, problem banks, etc. There should be multiple backends for what would render the composed unit.`utils/partitioning`Lower level support library for user partition groups.`utils/scheduling`Lower level support library for scheduling information (fold in much of edx-when). ---
**Author:** ![braden](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/braden/32/46_2.png) [@braden](https://discuss.openedx.org/u/braden)\ **Post date:** [October 4, 2021, 9:45pm UTC](https://discuss.openedx.org/t/attacking-the-monolith-by-extracting-a-core/5383/6 "2021-10-04T21:45:09Z")
@dave I like this idea. Have you had any more thoughts on this? > [@dave](#): > > `navigation` > `policy` > `publishing` > `unit_composition` > `utils/partitioning` > `utils/scheduling` Am I right that the focus of these apps ^ is on “content”, and it would support both Studio and the LMS? i.e. both “authoring” and “learning”? Where does “enrollment” fit into this picture? Because “navigation” and “unit\_composition” (what units does a particular learner see) depend on policy (enrollment is required to view, or cohorts, etc.) which depend on knowing actual enrollment/cohort data, right? Or do these navigation/unit\_composition/etc. apps provide a more abstract data set (and events), and it’s up to some later app in the LMS to convert that to actual per-user content outline/visibility? > [@dave](#): > > I think that should be enough of a framing to develop CourseApps like inline discussions that don’t really care that much about all the crazy business rules of edx-platform. With the inline discussion example, how would user profiles/auth come into play during development? Would one still need to run edx-platform to test the frontend components? (which I assume would be a frontend plugin for the courseware MFE, which consumes the LMS’s REST API?) Do you have any examples of extensions that could be developed using only this core set of apps? I think that would really help understanding and clarifying this. Some ideas, big and small: - I want to notify instructors [or a mailing list] whenever changes to a course are published - I want to use an adaptive learning service to customize course content for each learner - I want a button next to each XBlock that will show its OLX source to instructors - I want to use H5P as an alternative to XBlocks What occurs to me when trying to think of examples is that the capability of extensions so developed is going to be very limited until there is a good solution to frontend pluggability too. ---
**Author:** ![dave](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/dave/32/263_2.png) [@dave](https://discuss.openedx.org/u/dave)\ **Post date:** [October 13, 2021, 8:36pm UTC](https://discuss.openedx.org/t/attacking-the-monolith-by-extracting-a-core/5383/7 "2021-10-13T20:36:25Z")
> [@braden](#): > > > [@dave](#): > > > > `navigation` > > > > > `policy` > > > `publishing` > > > `unit_composition` > > > `utils/partitioning` > > > `utils/scheduling` > > Am I right that the focus of these apps ^ is on “content”, and it would support both Studio and the LMS? i.e. both “authoring” and “learning”? I think this is something that’s owned by LMS but can be pushed to via Studio. As in, they’re self-contained enough where they have their own in-process API interfaces, and the thing pushing data into it could be Studio running things in process, or potentially something else picking data off a message queue. So more “learning” than “authoring”, come to think on it–`openedx-learning-core` is probably a more appropriate name. > [@braden](#): > > Where does “enrollment” fit into this picture? Because “navigation” and “unit\_composition” (what units does a particular learner see) depend on policy (enrollment is required to view, or cohorts, etc.) which depend on knowing actual enrollment/cohort data, right? Or do these navigation/unit\_composition/etc. apps provide a more abstract data set (and events), and it’s up to some later app in the LMS to convert that to actual per-user content outline/visibility? I have this whole separate rant about “enrollment” because I think that it’s an amalgamation of a number of different concepts in different systems that share the same name–and that the LMS should restrict itself to thinking about enrollments as a Student Role for a course, but that’s an entirely different thread… 😛 Enrollments are one of a number of concepts where the logic for it is both surprisingly complex and deeply ingrained in edx-platform. In the past, I feel like we’ve tried to pull some of these out, only to find it it’s actually six different things with tendrils everywhere. My thought with things like this is to have plugin interfaces that go the other way from what we usually do–where the core framework logic is in apps in this new repo, and the little plugin objects are created in edx-platform (and optionally elsewhere as well). So to use a concrete example, say we migrate the `learning_sequences` app in edx-platform today to become part of the `navigation` app in this new repo. The `navigation` app will then have the concept of OutlineProcessors–an object interface for different concerns that have to modify the set of things you can see or access in a course outline. The `navigation` app would have the logic for running OutlineProcessors, reading those values from a list defined in Django settings. The [EnrollmentOutlineProcessor](https://github.com/edx/edx-platform/blob/master/openedx/core/djangoapps/content/learning_sequences/api/processors/enrollment.py#L11) would be defined in edx-platform (and likely have imports to a bunch of things also in edx-platform), and then be specified in the settings file. By doing this, we can keep some of the crazy logic and dependencies in edx-platform–it would allow us to move things more incrementally, without taking huge risks. ## Extraction + Extension = Re-platforming? Going back to: > [@braden](#): > > Am I right that the focus of these apps ^ is on “content”, and it would support both Studio and the LMS? If this works out the way I hope it will, what we’d end up with in this new repo is _something you’d use to build an LMS on top of_. **The long term goal would be to have an extracted core learning platform that people can innovate on top of.** The migration path would be a gradual extraction of pieces that could accelerate plugin development, providing obvious with each incremental addition/extraction. Open edX has so much _stuff_ in it today, and so many assumptions about how the learning experience should be structured–course hierarchy, grading policies, enrollments, etc. As an end-user product, maybe that’s fine. But I think we need to be something more to be a platform that truly pushes things forward in terms of learning innovation. The [LabXchange](https://www.labxchange.org/) use case has really affected my thinking in this. It’s a great site that uses Open edX for its core content interactions, but the user experience is almost entirely different–no courses, no enrollments, nothing at a high level that you would expect if you were familiar with [edX](https://www.edx.org/). You folks at OpenCraft managed to pull that trick off because you have deep expertise in the platform, but what would it take to make building something like that simple and straightforward? How many of those different systems could co-exist on the same site, even? Suppose in the new repo, Courses don’t exist as a thing. We have a LearningContext model, with some identifiers and basic metadata. But something like edx-platform wants Courses, so it makes its own model and makes it a 1:1 relation with LearningContext. Something like LabXchange defines Pathways and Clusters instead. * * * > [@braden](#): > > With the inline discussion example, how would user profiles/auth come into play during development? Would one still need to run edx-platform to test the frontend components? (which I assume would be a frontend plugin for the courseware MFE, which consumes the LMS’s REST API?) I’m fuzzy on this, but my thought was that you could exercise the core logic and REST endpoints of this new repo without having to have all the plugins of edx-platform (e.g. the EnrolmentOutlineProcessor wouldn’t be installed or invoked, but the outline still renders without it). I’m not sure this will really pan though. > [@braden](#): > > Do you have any examples of extensions that could be developed using only this core set of apps? I think that would really help understanding and clarifying this. > > Some ideas, big and small: > > - I want to notify instructors [or a mailing list] whenever changes to a course are published > - I want to use an adaptive learning service to customize course content for each learner > - I want a button next to each XBlock that will show its OLX source to instructors > - I want to use H5P as an alternative to XBlocks > > What occurs to me when trying to think of examples is that the capability of extensions so developed is going to be very limited until there is a good solution to frontend pluggability too. Thank you so much for these use cases. I don’t know if I have really solid answers for any of them, but some quick thoughts: > I want to notify instructors [or a mailing list] whenever changes to a course are published This would involve listening for a publish signal (probably from openedx-events), and then querying for the appropriate Roles associated with the course’s learning context. > I want to use an adaptive learning service to customize course content for each learner This should be doable by making a new type of `navigation`. There is some mapping between learning contexts and their navigation, with some Navigation interface defined. Probably one or two are defined internally, but others are allowed for in pluggable ways. > I want a button next to each XBlock that will show its OLX source to instructors I think that’s something local to the XBlock unit rendering runtime, which would be outside the scope of this. Or it could be that the `unit_composition` piece has enough data to provide this information, in which case this would have to be surfaced as a REST API. > I want to use H5P as an alternative to XBlocks A new Unit type in `unit_composition`. Again, probably a pluggable interface. There’s a generic Unit model that defines necessary metadata, but an H5P extension could flesh out its own data model. * * * One weird thing about all of these is what the publishing process looks like. If navigation and units are pluggable interfaces, then something’s stitching these things together, ideally in the same transaction. I’m not sure what that looks like. > What occurs to me when trying to think of examples is that the capability of extensions so developed is going to be very limited until there is a good solution to frontend pluggability too. Yup. (FYI @djoy) ---
**Author:** ![braden](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/braden/32/46_2.png) [@braden](https://discuss.openedx.org/u/braden)\ **Post date:** [October 14, 2021, 4:14am UTC](https://discuss.openedx.org/t/attacking-the-monolith-by-extracting-a-core/5383/8 "2021-10-14T04:14:00Z")
Thanks @dave, I think it’s a great direction to go in. What other type of feedback or ideation would be helpful here? ---
**Author:** ![dave](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/dave/32/263_2.png) [@dave](https://discuss.openedx.org/u/dave)\ **Post date:** [October 14, 2021, 9:42pm UTC](https://discuss.openedx.org/t/attacking-the-monolith-by-extracting-a-core/5383/9 "2021-10-14T21:42:38Z")
I guess the next step for me is figuring out what piece or pieces it makes sense to try to test this out on. Something that would actually be useful to put a more stable, versioned API on top of, but wouldn’t be too difficult to extract and port over. The `publishing` app would be a precursor because it would contain the most basic content data models. The `policy` bit would be relatively straightforward to implement, and `navigation` would be a straightforward port of `learning_sequences`. I can port over things in edx-platform to point to these new apps, but it’s not clear to me that anyone would _use_ these independently. Any feedback you and others can provide on this aspect would be greatly appreciated. ---
**Author:** ![kmccormick](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/kmccormick/32/8982_2.png) [@kmccormick](https://discuss.openedx.org/u/kmccormick)\ **Post date:** [October 15, 2021, 2:17pm UTC](https://discuss.openedx.org/t/attacking-the-monolith-by-extracting-a-core/5383/10 "2021-10-15T14:17:42Z")
> I can port over things in edx-platform to point to these new apps, but it’s not clear to me that anyone would _use_ these independently. Not sure if this is what you’re going for @dave, as it’s still in the edx-platform context… but there are plenty of edx-platform-plugin-esque repos like [edx-enterprise](https://github.com/edx/edx-enterprise), which are installed into edx-platform but also have unfortunate [imports back to edx-platform](https://github.com/edx/edx-enterprise/search?q=%22from+openedx%22) features such as course\_overviews. Knowing how common it is to want course structure metadata, I wouldn’t be surprised if there existed an edx-platform-plugin repo that wants or will soon want to use learning\_sequences. If we were to pull some form of learning\_sequences into an openedx-learning-core, then perhaps we’d be preventing someone from writing `from openedx.core.djangoapps.content import learning_sequences` into a non-edx-platform repository. I know @schenedx 's team has made use of learning\_sequences already to support a piece of special exams (?) functionality. Simon, do you know if your team has edx-platform plugins that would benefit from being able to properly use a tool like learning\_sequences? ---
**Author:** ![dave](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/dave/32/263_2.png) [@dave](https://discuss.openedx.org/u/dave)\ **Post date:** [October 17, 2021, 11:52pm UTC](https://discuss.openedx.org/t/attacking-the-monolith-by-extracting-a-core/5383/11 "2021-10-17T23:52:35Z")
I’ll start with a proof-of-concept that brings over CourseOverview-like policy information. It’s a nice place to start in that it’s a useful piece with a fairly limited scope, and it doesn’t require bringing over user role information. I think the hardest thing is adding the underlying `publishing` infrastructure code—I have some ideas around conventions for stitching the overall content publishing process together, but they’re not fully baked. ---
**Author:** ![schenedx](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/schenedx/32/72_2.png) [@schenedx](https://discuss.openedx.org/u/schenedx)\ **Post date:** [October 18, 2021, 2:18pm UTC](https://discuss.openedx.org/t/attacking-the-monolith-by-extracting-a-core/5383/12 "2021-10-18T14:18:39Z")
@kmccormick I want to clarify. My team’s addition to Learning Sequence was not on special exams. We added processing of Enrollment track to the Learning Sequence API. What’s more important was, we added “bubble up” feature so if all of the units should be hidden because of the enrollment track, the whole sequence of which the units are under, should hidden. There is no other plugins I can think of at the moment that leverages this yet. I can think of possibilities, but those are not on my team’s current radar. ---
**Author:** ![dave](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/dave/32/263_2.png) [@dave](https://discuss.openedx.org/u/dave)\ **Post date:** [January 26, 2022, 3:40pm UTC](https://discuss.openedx.org/t/attacking-the-monolith-by-extracting-a-core/5383/13 "2022-01-26T15:40:50Z")
Just a quick update to let folks know that I haven’t forgotten about this. There’s a roadmap ticket and some prototyping/discovery notes: > > > Extract core learning concepts and data models into a new \`openedx-learning\` rep…ository, with goal of creating a new, scalable core platform for learning innovation. > > \# Goals > > 1. \*\*Refactor the Open edX LMS to enable more dynamic behavior in a scalable way.\*\* > \_New features like V2 Content Libraries and Effort Estimation need a more scalable implementation than what exists in courseware today.\_ > 2. \*\*Provide an easier, more reliable path for extensions developers.\*\* > \_Extensions authors should be able to build and test against a smaller repo than edx-platform.\_ > 3. \*\*Accelerate monolith breakup.\*\* > \_By delivering incremental improvements to extensibility and performance, we have more reason to work on monolith breakup than our previous "extract all the non-core things" approach, where the benefits were more back-loaded.\_ > 4. \*\*Improve separation of edX business concepts from the learning platform.\*\* > \_No edX-specific logic would come over to the new repo.\_ > 5. \*\*Advance the decoupling of Studio and the LMS.\*\* > \_Part of this work includes building a new core content data model for the LMS, which would replace the current shared model with Studio.\_ > 6. \*\*Create a lightweight foundation for entirely new learning experiences.\*\* > \_This would allow groups like LabXchange to make use of helpful infrastructure and concepts, without having to simultaneously inherit all the technical debt and weight of edx-platform.\_ > > \# Major Components > > The high level components would include: > > \## Composition > > What permutation of a single unit does a user see? This would handle things like A/B tests, randomized problem selection, disabling content by enrollment type, adding staff-only debug markup, etc. There should be multiple backends for what can render a Unit, with the XBlock runtime being one of those. > > \## Navigation > > Sequence and Unit metadata, outlines, etc. How do you get to a particular piece of content you need to learn from? This would pull in parts of the Learning Sequences API from \`edx-platform\`. > > \## Partitioning > > Low level utility that helps determine what users are in what groups for the purposes of A/B testing, enrollment tracks, etc. Used by both Composition and Navigation. > > \## Policy > > Content-related settings as they apply to the site as a whole, organizations, and individual courses. This covers a lot of what Course Overviews and course override waffle flags do today. > > \## Publishing > > Centralized list of Learning Contexts (e.g. Courses, Libraries), their published versions, and various content-related errors and warnings are associated with them. We need this to help us tackle the major issue we have today around publishing: it's an asynchronous process with many different components that sometimes take minutes to complete and may fail independently, leading to a mixed-published state. This is the most foundational component that others will be built on top of. > > \## Scheduling > > Lower level library for content scheduling information, likely pulling in most of what is \`edx-when\` today. > > \# Phases > > \- \[\] Phase 1: Publishing and Policy > \_Establish foundation and start experimenting with CourseOverview data commonly wanted by extensions.\_ > \- \[\] Phase 2: Partitioning and Composition > \_This would be useful for both upcoming V2 Content Libraries and Effort Estimation work.\_ > \- \[\] Phase 3: Scheduling > \_Much of this could be ported over from \`edx-when\`.\_ > \- \[\] Phase 4: Navigation > \_Much of this could be ported over from the \`learning\_sequences\` API.\_ > > \# Implementation Strategies > > The following are some high level approaches/considerations with this new project. > > \## Focus on content first. > > Content data that lives in Studio is easy to re-build and backfill into new apps. User data is up to five orders of magnitude larger, and involves much larger challenges in terms of data migration. > > \## Build Extensible Primitives > > LearningContexts are a generic term that applies to Courses, Content Libraries, Learning Pathways, and any number of other collections of content that we want to discretely version and publish. We can centrally define logic around these, while leaving it up to higher layers to model specific types of LearningContexts in a pluggable way. > > For instance, it makes sense for there to be a table of Courses, that have a foreign key to the LearningContexts table, which holds course-specific metadata. That table would have course-specific fields, and may even have a \`null\` learning context in the beginning (before any content is created). > > This kind of arrangement would lead to a three layered system: > > 1. Foundational primitives and logic, implemented only in \`openedx-learning\`. > 2. Specific implementations of those components to define different learning experiences, e.g. "Two-level hierarchical navigation", "randomized user partitioning scheme". These could be implemented in \`openedx-learning\` or outside. The most popular and useful ones could get folded into the repo over time. > 3. Business-level plugins that tie into systems defined in (2), such as \`EnrollmentTrackOutlineProcessor\`. These would be implemented outside the \`openedx-learning\` repo (many would live in \`edx-platform\`). > > \## Implement Plugins in \`edx-platform\` > > We once attempted to lift the ModuleStores out of \`edx-platform\` and ran into a rat's nest of dependencies that made the task extremely difficult. My thought with things like this is to have plugin interfaces that go the other way from what we usually do–where the core framework logic is in apps in this new repo, and the little plugin objects are created in edx-platform (and optionally elsewhere as well). > > So to use a concrete example, say we migrate the \`learning\_sequences\` app in edx-platform today to become part of the navigation app in this new repo. The navigation app will then have the concept of OutlineProcessors–an object interface for different concerns that have to modify the set of things you can see or access in a course outline. The navigation app would have the logic for running OutlineProcessors, reading those values from a list defined in Django settings. The \`EnrollmentOutlineProcessor\` would be defined in edx-platform (and likely have imports to a bunch of things also in edx-platform), and then be specified in the settings file. > > By doing this, we can keep some of the crazy logic and dependencies in edx-platform–it would allow us to move things more incrementally, without taking huge risks. > > \# See Also > > Original post: > > \* https://discuss.openedx.org/t/attacking-the-monolith-by-extracting-a-core/5383 (There’s a personal repo of mine that I used to experiment with some ideas, but it’s in a broken state right now, so it’s definitely not worth trying to spin up locally.)