OEP-65: Frontend composability next steps and a bit of visionin'

Announcements Architecture
1

Hi all,

I’m excited to share that I’m going to be working with Axim to build a reference implementation of runtime module loading and shared dependencies as described in the recently-merged OEP-65! If you haven’t been following along, I encourage you to go read it for more detailed background than I’ll give here.

Next Steps

OEP-65 was merged as “Provisional”. That means we believe it’s a good idea and agreed on a way forward, but full acceptance is pending an actual implementation of the architectural decisions it describes. We’re expecting to build the implementation using webpack module federation.

At a very high level, this will look like:

  • Build Configuration: Adding webpack configuration to frontend-build that allows MFEs to be built as loadable modules rather than complete applications.
  • MFEs: Picking a few MFEs to start converting in a backwards compatible way. Tentatively, we think we might start with frontend-app-authn and frontend-app-learner-dashboard.
  • Shell Application: Create a new “shell” application which will act as the entry point for all MFEs (eventually).
  • Frontend Plugin Framework: Extend FPF with a new “module plugin” type for loading our MFE modules at runtime without an iframe.

I expect this work to take most of the rest of 2024, and as mentioned above, it should be able to proceed in a mostly backwards compatible way.

There are a lot of details yet to figure out, as well as some concerns that are important but tangentially related. There are a few big ones that stand out in my mind. Forgive me if this is a bit speculative or has any glaring holes; these are complicated decisions, which is why I wanted to shed some light on them!

Build tooling

Simultaneous to OEP-65, we’ve been discussing whether we should attempt to move from webpack to another build tool, like Vite or Rspack. While any given MFE is significantly faster to build and iterate on than the old monolith, when the entire platform is built together, it’s feeling slow. This problem overlaps with OEP-65 to the extent that we believe module federation is the right tool to improve Open edX’s frontend composability, and webpack is the most robust implementation of that around. The only other known options are Vite and Rspack.

I feel fairly strongly that we don’t want to ‘roll our own’ version of module federation, and the alternative module federation implementations are either not robust enough (Vite), or not widely adopted enough (Rspack) to feel like a safe move. We’re picking a tool for the whole platform and community; it should be as much of an ‘industry standard’ as we can manage. If Vite had better module federation tooling I think it’d be particularly compelling, but in my testing it just doesn’t seem like it’s there, and it lacks some features.

So with that in mind, I think we stick with webpack for now. That said, there are some things we can do to squeeze more performance out of webpack. I’ve found that using swc-loader instead of babel-loader shaved a few seconds off production builds, and our SCSS compilation is surprisingly slow. Further, we should perhaps look at using pnpm or yarn, since npm itself seems to be a bottleneck for us. npm install seems to take more time than the actual build, from what I’ve seen. Also, the switch to module federation will ultimately speed things up for most MFEs by virtue of needing to bundle less actual code, as some large dependencies move to the shell.

This is all to say, as part of this effort, I’m not going to be trying to replace webpack. I think this is an area, though, that we should keep our collective eye on. And if anyone wants to do a deep dive on Tutor’s MFE building with me, I’m all ears and ready to learn.

Merging shared frontend repositories

We’ve also been discussing whether some of our foundational frontend repositories should just be merged. Specifically, this is about frontend-build, frontend-platform, frontend-plugin-framework, frontend-component-header, frontend-component-footer, and the new repository for the “shell” application.

Originally we kept some of these things (frontend-platform, frontend-build) for several reasons. One, they have different jobs, and were conceptually separate from each other. Two, we weren’t sure whether they would be ubiquitously used by all our MFEs, at the time! Turns out… the ‘platform’ for all our MFEs is universal, which is actually pretty great. All MFEs should be using all of the repositories shown above, and in around 6 years we haven’t had any real push back on that recommendation, just widespread adoption. As we talk about adding this new ‘shell’ application as an entry point for all MFEs (eventually), it begs the question… do we really need 7 repositories for all this if every MFE needs all of it?

Further, it’s all becoming more inter-connected, and these libraries all depend on each other, and they’re not as conceptually isolated from each other as they used to be. frontend-build and frontend-platforms share responsibility for our environment config, which is central to fronted-plugin-framework, which we want to use to configure our headers/footers as plugin slots, all of which will be spun up and owned by the new shell application.

Should there be one repository, and dare I say it’s just frontend-platform?

A strawperson proposal:

  • frontend-platform continues to be published as a library and is a runtime dependency of MFEs.
  • frontend-plugin-framework is folded into frontend-platform and exposed through the runtime library.
  • The shell application is built in frontend-platform and owns the application initialization process that MFEs do themselves today. It consumes the ‘services’ like logging and analytics, initializes i18n, etc.
  • frontend-build is folded into frontend-platform and is joined with the shell application to provide a single development server for MFE development, exposed as a “buildtime library” to compliment the existing ‘runtime library’.
  • frontend-component-header and frontend-component-footer are folded into the shell application and wrapped in plugin slots. By default they’re bundled as ‘direct plugins’, but can easily be tree-shaken out if not needed by converting them to module plugins and loading them from somewhere else. frontend-platform exposes an API for MFEs to request a given header when being loaded. We double down on extensibility and configuration of the header/footer to reduce the need to build custom ones.

CSS and Paragon

There’s been an ongoing effort to land some PRs around design tokens, CSS variables, and external stylesheets. I think this is ultimately related. The shell application gives us the opportunity to share a single stylesheet between all MFEs by default, and SCSS/CSS processing is actually a significant chunk of our MFE build time. With the ongoing Paragon work and creation of the shell, I think we can significantly streamline our styling over time, getting to a place where we share a single Paragon stylesheet and remove SCSS processing from our MFEs all together (in a world with utility classes and css variables, it’s not adding much).

Plugin slots, configuration, and customization

Finally, we should talk about how to extend the frontend platform. This is a vision I’ve had rattling around in my brain for a number of years now, and just started to really solidify into something that feels powerful as we were working through the details of OEP-65, and as frontend-plugin-framework has started to take shape.

To say something that may sound a bit audacious - I believe that the default installation of the platform should be a single repository, frontend-platform, with all the default MFEs linked in as direct plugins through env.config.js. The MFEs are pre-built, pre-tested artifacts, released as packages on npm/Github. By default, you run one build command and have the entire platform ready to go.

Anyone who has need of more granular, decoupled deployment of MFEs can instead turn their direct plugin MFEs into module plugins loaded via module federation. The MFE is then independently deployed and is no longer built as part of the shell. This is just a configuration change (plus taking on the extra operational overhead we already have today). Additional MFEs and plugins are added via the same configuration.

Further, dependencies like the logging service, analytics service, Google analytics, tag manager, hot jar, Zendesk, and anything else loaded as a script tag in the header, can all be loaded via plugins and plugin slots; just non-visual ones.

To support all this, there are a few things we need:

  • The ability to either use env.config.js to configure MFEs, or to use a runtime module to configure MFEs (i.e., the module contains the env.config.js and is loaded first thing at runtime). The MFE plugin API will no longer be sufficient since it can only express JSON, not JavaScript.
  • A few new types of plugin slots. Plugin routes, plugin scripts, plugin stylesheets. All of these are very doable.
  • Thoughtful application of plugin slots for the above with sensible default plugins.
  • Ultimately, guard rails on this configuration to make it easy to use. This could look like Typescript types, helper functions, a CLI, linting, etc., to ensure that the configuration you’re writing is actually correct.

Conclusion

I’m really excited about all of this, and am thrilled for the opportunity to get the frontend platform to a place that works and works well for the community. Let me know what I’ve missed, what resonates, and everything in between.

7 Likes
2

Big +1 from me for consolidating all the various frontend “tooling” repos into one frontend-platform repo.

One significant annoyance with our current MFEs is overspecified dependencies. Most of our MFEs are already largely compatible with React 18 and we could be using it today AFAIK - except that it’s currently specified in a million different repositories as "peerDependencies": {"react": "^17.0.0"}. I’d love to see a world where each MFE just has one dependency on "frontend-platform": "^32" and upgrading to e.g. "frontend-platform": "^33" will pull in the newly chosen versions of react, react-intl, react-router, react-query, webpack, eslint, etc. that are all known to be compatible with each other. This is exactly how it works if you use Next.js, which you can kind of think of as an alternative to (frontend-platform+frontend-build+react-router+webpack).

To say something that may sound a bit audacious - I believe that the default installation of the platform should be a single repository, frontend-platform, with all the default MFEs linked in as direct plugins through env.config.js.

This I’m less sure about, but interested in. How do major upgrades work in this scenario? Say React Router 7 comes out and it’s completely incompatible with v6. In the scenario where MFEs consume frontend-platform as their primary dependency, it’s easy - first frontend-platform is upgraded and a new version is released, then each MFE upgrades as needed, by bumping its frontend-platform version and making the required changes at the same time. But if frontend-platform itself depends on all the MFEs… how does such an upgrade work? Does it need to support two versions of react-router simultaneously and let each MFE indicate which version it’s using?

(I wonder if it would be easier to have a separate frontend-shell that can support two different versions of frontend-platform, so e.g. all MFEs that are on v32 will be grouped together with module federation and soft navigation among them, and all other MFEs are upgraded to v33 will also be grouped together with module federation and soft navigation among them. It seems to me that this would make it much easier to deal with upgrades and breaking changes, while still achieving the “default installation of the platform should be a single repository, frontend-platform frontend-shell, with all the default MFEs linked in as direct plugins”)

2 Likes
3

Nice. Personally I have a rule of thumb: in 2024, you shouldn’t see the word babel anywhere in your project, build process, or bundle.

Need to transform JSX (or TS)? SWC or even just TSC does it much faster, as you found. (And if you’re working on something other than an Open edX MFE, try out a Node alternative like Deno or Bun which support JSX and TS/X natively, without any transform step - it’s so much nicer.)

Do you see @babel-runtime in your depdencies? That usually means some code was transpiled with babel configured to support old browsers like Internet Explorer which is long past end of life. All current “evergreen” browsers understand modern JS syntax fine, and using it without babel’s backwards compatibility polyfills is going to be faster and have a smaller bundle.

@babel/polyfill - a 7 year old version is still referenced in frontend-build, but this was deprecated in 2019 in favor of core-js, which is also no longer useful.

1 Like
4

Thanks @braden! Some thoughts.

Over-specified Dependencies

I think your point here is that if we combined some of these libraries, we’d have far fewer peer dependencies floating around, and so this problem would be minimized. They’re always going to get out of date as new major dependency versions appear, so we can either expend energy keeping them current or avoid the problem. Agreed, I think this might help, and I didn’t explicitly say it above, but I think we’d be looking at a monorepo here, just to keep concerns separated and preserve the option to use some parts independently.

Bundling Shared Libraries in frontend-platform

I like that this gives us fewer version numbers to manage, but I’m also a bit leery of it for some reason. Probably because, like many folks, I’m thinking of the Open edX frontend as an application using frameworks and libraries, not as a framework itself, and the idea of wrapping a dependency and re-exporting it feels ‘wrong’. Changing that mindset means that being more all inclusive and providing everything you need to write your Open edX application code is more of a virtue for our platform. Curious how others feel about this, I’m kind of undecided.

Major upgrades

How this all works is something I’m still developing in my brain, but I think the moment we switch to having modules loaded into a shell, we have this problem with breaking changes to ‘singleton’ dependencies regardless of whether the MFEs are direct plugins or module plugins loaded at runtime.

They all need to share the same instance of the dependency at runtime, and if one is expecting an incompatible version, it’ll blow up. For a complicated, singleton shared dependency with a breaking change, I don’t think there’s an easy or straightforward upgrade path. It ultimately takes work and coordination in the (hopefully rare) occasions when that happens. That could look like temporarily running multiple shells with hard page refreshes between them, akin to how things are today with our “siloed” MFEs.

1 Like
5

Following up on this:

I tried replacing babel-loader with swc-loader, and got it working with the Course Authoring MFE. The build time dropped only from 65s to 60s, which makes it hardly worth the effort. I then went to great pains to strip out all the SCSS from the MFE and disabled PostCSS, and the build time was reduced further from 60s to 48s, which is of course not realistic because we do need to actually compile the SCSS.

This is a far cry from the substantial improvements shown with Vite (though it was measured on a different MFE, so I can’t say it exactly compares).

So my tentative conclusion is: if our latest frontend plans depend on webpack’s module federation, we’re probably going to be stuck with slow builds. If we ultimately take a path forward that doesn’t actually use webpack module federation specifically, I’d really like us to revisit Vite.

6

Another direction I wasn’t willing to try (for lack of will, more than anything), was trying to use rspack instead of webpack. It supports module federation that’s compatible with webpack, and is built on Rust. I don’t think it’s a drop-in replacement for webpack, but it may be close, and may have significantly better performance.

Part of what stopped me was just worrying about its adoption levels; Vite is quite popular and well supported, I can’t say I know whether rspack has similar adoption. With Vercel getting into the game and Turbopack on the horizon, I just couldn’t justify it. Feels like we’re on the cusp of seeing the ecosystem shift around.

1 Like