Hi all - this post is a continuation of the OEP-65 next steps/visioning post over in Announcements/Architecture. If this is unfamiliar, there’s some required reading there and in OEP-65.
It’s long. Get comfy. There’s a lot to talk about!
I’m deep in planning migration strategy for OEP-65, trying to get to the point where I can write some ADRs on how we should proceed. But it’s complex, deep, and broadly impactful work, and I think I need to bring some other folks along on the ride with me or the ideas I want to throw around aren’t going to make much sense when I share them.
This post is about more than OEP-65, and is going to challenge the way things have been done in the past. It’s both aspirational and, I think, eminently attainable. The reference implementation of OEP-65 is groundwork for this broader vision, and some of the architectural decisions for OEP-65 will make the most sense when viewed in this context.
So with that foreword, come on a little journey with me!
Assumptions
These fundamental assumptions underpin everything that follows. If we don’t agree on these, then this vision unravels.
Unified platform library
This one is likely the most debatable of the three assumptions. But we’ve had a lot of discussion and affirmation that we believe that in implementing OEP-65 we will want the new shell application to be in the same repository as frontend-platform, frontend-build, the header, footer, plugin framework, etc.
The rationale is that:
- We want to reduce dependency management overhead.
- We want to be able to develop MFEs in the shell from their own repo, which implies the MFE can use the shell and frontend-build together as a dev dependency.
- We expect the lines between these libraries to become more blurry.
- We expect them to be used together as a cohesive platform, not consumed piecemeal as we suspected 6 years ago.
I think they should be in the same repository and released as one unified platform library. The independent repositories have not proved to be worth it, it will be harder to modify them in place, and they will make it significantly more difficult to work with them in the future as these libraries converge.
Keep it simple. Most operators don’t need micro-frontends and module federation
From the perspective of big companies, micro-frontends make a lot of sense. You divide your frontend up so that independent teams can work with parts of it with autonomy. As we’re discovering now, it’s also helpful - if you need micro-frontends - to be able to compose them together in the page and share resources/dependencies between them, which is the whole point of OEP-65.
But any individual or small group operating an Open edX instance derives virtually zero value from MFEs and module federation. I’d guess that the vast majority of operators fall into this category. MFEs and module federation both significantly increase operational overhead and complexity in favor of ‘autonomy’ these operators don’t care about.
The Open edX platform’s default operating mode should align itself with the needs of these operators, which will increase adoption and stickiness, reduce frustration, and improve time to value.
We still want a small, stable core, open for extension, closed for modification
This is a central tenet of the Open edX architecture which we’ve been aspiring to build for years. We should continue to drive our platform toward this vision, we should reject choices that cause us to drift away from it, and we should seize opportunities to make it a reality.
Background and Context
As we get into this post, I’m going to be suggesting a bunch of big (sounding) changes. I arrived at these proposals by imagining the various alternatives and how they align with the base assumptions above.
How the shell and unified platform library changes things
So imagine the new shell application in its most basic form. If you’ve been following along, it will look very much like the POC ‘host’ app referenced in OEP-65.
- It acts as a host for other MFEs.
- Calls frontend-platform’s initialize() once up front so the guest MFEs don’t have to.
- It owns the header/footer.
Like today’s MFEs, you would git clone its repository somewhere and run the build command on the source code.
We could continue to do this. But remember… we want that unified platform library, and we want it to include the shell. Going along with this, we’ve investigated monorepos enough to know we want to avoid them, and envision the unified platform library as a single package.
That means to build the shell, you would need to check out the entire unified frontend library, which is the small stable core of the Open edX frontend. That flies in the face of our desire for a small extensible core that’s closed for modification, and we’d need to go out of our way to remind operators not to mess with it lest they destabilize their instance and make upgrades more difficult. It’s wild from a DevEx/usability/best practices/encapsulation perspective.
We can and should do better.
By the way, our frontend customization/build workflows are an afterthought
Regardless of where the ‘core’ code is, to work with the Open edX frontend in a general sense, operators need to minimally modify the configuration files that it uses for its dev and prod builds. They optionally need to add customizations, plugins, and a brand.
This is the primary way operators and developers interact with the framework outside of getting it hosted on the internet.
What guidance and best practices do we provide operators on how to do this? No pattern or best practices on how to organize your code that configures and customizes the frontend are provided. We mostly leave operators to figure it out on their own. Big respect to Tutor for tackling this with the Tutor MFE plugin or I’m not sure anyone would ever figure it out.
What edX did
As an operator of the open-source software, edX navigated our current architecture of git clones, configuration and customization by building a fancy, bespoke build process around tubular.
Tubular and the deployment pipeline behind it:
- Sets up a workspace in a build job.
- Git clones all the frontend code and installs its dependencies.
- Pulls config files from a separate private configuration repository and combines them with the code.
- Runs
npm installa bunch of times to replace dependencies in package.json with any customizations. - Runs the build command to create an artifact with those customizations.
- Ships the built
distoff to the web to be served. - Deletes the ephemeral ‘dirty’ git checkout after the build finishes.
- Starts over for the next build.
I imagine others are doing similarly, and I expect Tutor is also doing a lot of work to abstract away this process.
If we focus on how configuration and customization is done (steps 2 to 5), even implementing this and figuring out how to arrange everything is confusing. Documentation is spotty at best. Since you’ve checked out the platform’s source code and then overridden its dependencies, you’ve created a ‘dirty’ checkout that you can’t reasonably check in anywhere without effectively forking those core MFE repositories.
We treat configuring, extending, building and deploying the code as a poorly supported afterthought when it’s the main thing everyone needs to do!
Again, we can and should do better.
Case studies and prior art - Next.js and Wordpress
To draw a parallel, if you want to make an app with Next.js, you don’t check out the Next.js source code and work with it directly. You certainly don’t fork it and maintain your own copy. You create a project that has next as a dependency. They make this easy with helpers like create-next-app. You put your config and customizations in that project, and you check it in as a cohesive unit that represents “your app” and what makes it unique. Almost all modern frontend meta-frameworks (Next.js, Remix, Astro, etc.) work this way.
Wordpress is a very notable exception. In Wordpress (not the Saas offering) you actually do fork the complete platform and save your own copy of it. But this comes with a big caveat. The first line in Wordpress’ documentation on Why We Make Plugins is:
If there’s one cardinal rule in WordPress development, it’s this: Don’t touch WordPress core.
If they were making Wordpress today, it’d probably make sense to just make that core a dependency so folks aren’t confused. It’s important to remember where Wordpress came from. It was created in 2003, 5 years before Github existed (2008) and 7 years before npm (2010). A lot has changed in the past 21 years, and the ways in which we share and distribute software have evolved significantly. To make up for a lack of package management and package registry options, Wordpress has invested significantly in enabling operators to upgrade via migration scripts and programmatic updates to sites. They presumably have significant staff dedicated to this. Wordpress’s architectural choices have a whole lot of historical baggage which we don’t need to emulate.
I’m a broken record - we can and should do better.
Proposal: Project mentality
All of this is to say, the status quo of how we ask operators to do things today is a bad user experience. Module federation and the architectural choices around the unified platform library are going to exacerbate the problem by exposing even more source code to operators. We need to invest in making this better, and I believe it needs to happen as a fast-follow or in concert with OEP-65.
I believe this looks like shifting our mental model and thinking of the Open edX frontend platform as being akin to Next.js, Remix, Astro, or any other modern meta-framework. We should embrace a “project” mentality instead of a “check out all the source code and modify it” mentality.
The goal of Open edX pluggability and extensibility has always been, implicitly, to enable this. We’ve been talking about “a small stable core, open for extension but closed for modification” for years in our architectural talks. If we don’t expect folks to modify the core, we should stop waving the source code in their face! We should also bring their config and customizations front and center.
What is a project mentality? What do all these frameworks, including Wordpress, have in common?
There’s certainly lot to admire in Wordpress’ plugin framework, as we’ve all noted here on the forums in the past. Let’s step up a level from hooks and filters for the moment, though. If we look at how you create a plugin in Wordpress’ docs, and also think about what you get when you start a new Next.js, Remix, or Astro site… they’re actually pretty similar if you ignore where they put their internal source code.
- In all of these cases, you create a new project directory that’s yours to do with as you please and check into source control somewhere.
- You configure the platform with config files (like
next.config.jsorwp-config.php). - You put your “customizations” (i.e., your app) in a sub-directory of that project, whether it’s
srcor Wordpress’pluginsdirectory. - The framework knows how to find your source files and build them into the underlying platform, extending it with your functionality. In the case of Wordpress, this is about extending a significant body of functionality in terms of its CMS, whereas Next.js and friends are more of a blank canvas.
In all these cases, you create a project and put your customizations and configuration in it, and that’s the thing you check in - it represents your site. I think we can follow this pattern and have Open edX frontend projects, and I don’t think we’re that far off from it.
The unified platform library isn’t the project - we established we don’t want you to have to check out all that internal source code. Instead, it can be a dependency of your project, which is a new thing for us, and brings the primary way operators interact with the platform front and center, rather than leaving it as an afterthought.
Open edX frontend projects
Enter the Open edX frontend project. Hopefully you’re still with me. It represents the frontend of your entire site, and minimally consists of:
- A directory with a package.json that depends on the unified frontend platform.
- A set of “scripts” in package.json to build your project via a CLI (command line interface) provided by the unified platform dependency, much like fedx-scripts today, or the CLIs provided by any of these modern meta-frameworks.
- A few environment-specific configuration files: openedx.prod.config.js, openedx.dev.config.js, etc.
- An .eslintrc.js, tsconfig.json, and a few other config files for the build/dev tooling. Maybe a Webpack config if you need to customize it.
- An index.html file you can customize.
As a best practice, you and your organization check this project in somewhere, just like you would any other project you created using a frontend (meta-)framework (Next.js, Wordpress, React, Remix, Angular, whatever). The code is yours, and by default it’s tiny and is pre-configured to build the core product offering. (This is where we’re more like Wordpress than the meta-frameworks. Open edX is not a general-purpose blank canvas, we provide significant functionality and features out of the box.)
You don’t git clone any part of the Open edX frontend core at all. Everything you need is a versioned dependency in package.json (unless you need something more advanced, which we’ll get into). The only things you ever check out are your own code.
As mentioned, by default it’s configured to load the frontend core product apps (i.e., authn, profile, account, learning, course-authoring, learner-dashboard, etc.). It does this as direct plugins via frontend-plugin-framework’s mechanisms. You don’t need to do anything to set this up - it just works, they load seamlessly via one build command, and as part of a single unified deployment to the internet. Again, we are not actually that far off from this!
What does the unified platform library provide?
When you run the build via the CLI, the ‘shell’ application code is used as the entry point, and the shell is responsible for loading the config file in your project which then loads your customizations. The shell lives in the unified platform library, along with what’s currently frontend-platform, frontend-build, the header, footer, and frontend-plugin-framework. The shell is akin to Next.js’s underlying web server or the Wordpress core. The unified platform library also provides the recommended ESLint, Jest, TypeScript, and Webpack configurations to your project much like frontend-build does today.
To summarize, the unified platform library is released as a single npm package with a few exports for different uses:
- A CLI to use in package.json to run a build, start a dev server, etc., like fedx-scripts today.
- A ‘runtime’ library for use in your customizations, like frontend-platform/frontend-plugin-framework today.
- A set of default config files for ESLint, Jest, TypeScript, and you can pull into your project and extend as necessary in the same way that frontend-build works today.
Most of this already exists - we’re just going to package it differently and expand on the functionality by creating some more options.
Working with customizations and extensions
To customize your project beyond the configuration options, you start adding sub-directories to the project directory for your brand, custom plugins, customized replacements for apps (if you really need to), new apps, etc.
You import these into the project via the config files. Many operators won’t need anything more flexible than this, but if your organization needs more independence, you can put them in separate repositories and load them at runtime via module federation or as package.json dependencies.
The CLI will also provide commands to bootstrap your project and these sub-directories, similar to create-react-app or create-next-app:
create-site: creates a new project directory.create-plugin: creates a new plugin stub as a sub-directory in your project.create-brand: creates a new brand based on brand-openedx.create-app: creates a new app (MFE in today’s terms), which is a bit different than a plugin because it exists as a top-level route in your site.
And maybe some others as necessary. These will take the place of frontend-template-application and automate the process.
Once you run one of these commands, you write your custom code right there in the project in a sub-directory. As mentioned, we use the config files to import it into the site. If we were to get imaginative, we could even have the build crawl these sub-directories and import them automatically using metadata co-located with the code, which is effectively what Wordpress and Next.js do.
You check it in, along with your environment-specific configuration files, and just run the build as a single command in your build system. No weird npm aliases or combining source code with config files checked in somewhere else.
Advanced customization and development workflows
By default, as mentioned above, you can load your custom code into the site by importing it directly into the config file’s plugin configuration to be loaded as direct plugins. For operators with small teams and a modest amount of custom development, this will be by far the easiest way to work with the Open edX frontend.
But if your organization needs micro-frontends, independent deployments, team autonomy, or is developing a significant amount of custom code, we still got you. I want to note: if you fall into this category you presumably have more resources and developers. You should expect that your more complicated use cases will result in a more complicated build and deployment setup.
The good news is that you’re likely doing this today because of the current micro-frontend architecture. But we’re going to make it more flexible so you can get that power where you need it, and use a simpler setup where you don’t.
Multiple Git repositories
If you’d rather manage your customizations in multiple Git repositories, that works too. They can be checked out as peer directories of the project instead of sub-directories, they can be released as a library and added as a package.json dependency of the project, or they can be fully independently deployed and configured as module plugins to be loaded at runtime via module federation.
The platform’s CLI will provide helpers to build customizations in various ways; it’s all just different webpack configs. The same customization can choose between these deployment mechanisms, it’s not locked into one for all time. The code in the repo is the same.
Big scary breaking changes
Further, if there’s a large breaking change you need to roll out to your custom code iteratively, we can imagine a world where you can temporarily deploy multiple shells and split your apps/plugins between them. Big breaking changes are hard; hopefully they don’t happen very often, but there’s recourse if they do.
How frontend-app-* MFEs evolve
The bulk of an MFE’s code is unchanged. But the trimmings/platform code around them simplifies. Imagine you’re working in an MFE that’s a separate repository from your project - either one of the core product MFEs you’re developing, or a custom one you’ve broken out into its own repo.
The unified platform library provides CLI targets that let you start a development server for your MFE using the shell (we unified the shell with frontend-build so we can do this, remember?). It also turns out that the shell, even though it’s in the unified platform library, is just an MFE that runs initialize and has the header and footer, as shown in the host POC for OEP-65. So if you want to continue to deploy an MFE as a standalone application, you can do this via the shell and giving the MFE a config file, just like you can do today. It’s identical to the “large breaking change” rollout in the previous section, just for one MFE.
This last option will, in fact, be the migration path for the MFEs. We can maintain backwards compatibility with the current “application MFE architecture” by replacing an MFE’s header and footer dependencies, and its index.js file where it calls frontend-platform’s initialize, with a dependency on the new unified platform library. It will effectively build a shell with one app in it - the MFE - which can be deployed as it always has. We’ll be able to maintain backwards compatibility while also fully updating the MFE to the new shell architecture.
Core product MFEs will packaged and released as npm packages so that they can be dependencies of your project and imported as direct plugins. The exports of the package are the same as the exports of the MFE for module federation at runtime.
Conclusions
This sounds like a lot, and it is. But we’re closer than it may seem as we start working through the OEP-65 reference implementation. The majority of the code involved is already written, and many of the more complicated pieces are already figured out (or will be).
My gut feeling is that we can’t responsibly add the composability in OEP-65 without also doing the work to simplify our architecture for operators. Helpfully, there’s strong synergy between the two activities. For OEP-65 we need to turn the MFEs into plugins/modules so they can be loaded by module federation, and it’s an incremental step beyond that to also let them be loaded as direct plugins and providing a project structure around that, drastically simplifying the architecture for most operators.
We’re going from a world where customization means forking source code to one in which we have a plugin framework and a flexible architecture that provides a variety of deployment options according to an operator’s needs. We’re adding even more complexity to a part of the system that was an afterthought - we need to invest in it or it’ll collapse under its own weight. The good news is that once we invest, there’s a world right on the other side of that work that’s way, way simpler, and the investment quickly pays off.
So, like, what’s next?
As mentioned at the beginning, I’m in the process of figuring out how we responsibly migrate the frontend to the new “module MFE” architecture described in OEP-65, enabling frontend composability. For me, that work has always existed in context of this larger vision for how we orient the Open edX frontend around the needs and activities of operators.
It was a bit of a revelation to realize that if we use the shell as an entry point for the entire frontend and express loading MFEs/apps as plugins and modules, we have the opportunity to drastically simplify the platform’s base operational overhead without sacrificing MFE independent/autonomy for those that need it.
I’m going to focus first on building the necessary code to enable module federation, and will be looking to do some intentional architecture that orients that work inside this broader vision. I’ll also be on the lookout for ways to opportunistically enable the vision it where it doesn’t blow my estimates/timeline out of the water.
There’s plenty of devils and details I expect we’ll encounter, but I’m pleased to have what feels like a cohesive mental model for how all this can and should work. I figure I may have some more explaining to do to see if it feels cohesive to others, and that’s okay. Happy to have those conversations!
I don’t think I’ve ever written such a long forum post, thanks for reading if you got to the end! I’m really excited about this work, and hopefully some of that enthusiasm is rubbing off on others. 
A further clarifying points:
I’m even more excited to have the personal bandwidth and focus to go make it happen.