Microfrontends: a retrospective

Development
1

A couple years ago, microfrontends (MFEs) were introduced to Open edX: the goal at the time (and now, still) was to migrate all major features from the LMS and the Studio to dedicated, standalone bundles of static assets. In a nutshell, MFEs were the “future” of Open edX development.

(There’s certainly a lot more to be said about the history of MFEs but I’m definitely not the most informed person on that topic)

Today, the Open edX releases ship with multiple MFEs enabled by default (profile, learning, account, gradebook). We (the community) have a lot of experience developing and deploying MFEs. Generally speaking, when we talk about new features we try to architect them as new MFEs. So I thought it would make sense to take a look back and consider where we stand, what the adoption of MFEs has brought us, what problems we are facing and how we can address them. Basically, I want to start a conversation to ask people how they feel about MFEs.

I’ll start with my own impressions, but I must acknowledge that they are very incomplete; for instance, I personally have zero experience creating MFEs from scratch or adding new features to existing MFEs. I’m mostly tasked with the deployment of MFEs (via Tutor, of course) and fixing minor issues.

Building on config change is costly

MFEs are bundles of frontend-only html/css/js code. Thus, they need to know about platform configuration (NODE_ENV, BASE_URL, etc.) at build-time. This means that:

  1. the build process must be run every single time that a configuration value is modified.
  2. we cannot provide plug-n-play Docker images to be run by all users.
  3. people deploying to Kubernetes must also setup a Docker image build factory.

For instance, in the context of Tutor, re-building MFE images might take 10 minutes or so. What is worrying me is that this time will increase linearly as we add new MFEs, and that all users must pay this cost for every configuration change. Configuration changes happen frequently, because users must upgrade their platforms, customize stuff, play with plugins, switch from one environment to the next, etc.

If we are to multiply the number of MFEs out there, then we must definitely find a solution to this problem.

Building is not a uniform process

The way that the final MFE image is built in Tutor is that we have single template for every app: tutor-mfe/Dockerfile at e99e844920bb6c0be597d52947b46c7b433cd8f7 · overhangio/tutor-mfe · GitHub

It basically goes like this:

  1. git clone ...
  2. npm install
  3. npm run build

Then all static assets are gathered in one place and served by a single web server.

The simplicity of this build/deploy process is one the great things about MFEs. But we have a problem when one of the MFEs needs to be deployed in a different way. This happens, for instance, when:

a. a manual patch needs to applied (a security patch for instance)
b. some dependency needs to be upgraded (such as frontend-build)
c. they depend on older requirements (webpack for instance) for which the process must be slightly different

Some of these issues are due to how Tutor implements MFEs, but honestly I cannot figure out a way to proceed that would be both robust and easy to understand for end users. Some further reflection would be needed, I guess.

Static assets are too large

From the final user perspective, accessing an MFE requires downloading a couple megabytes worth of static assets, and these assets are different for every MFE. For instance, the js and css assets from the account MFE weight ~2.7 MB. It’s possible to cache assets from one call to the next, but not across MFEs.

Again, if we are to scale the number of different MFEs, we need to figure out a way to avoid downloading heavy assets from one page to the next.

3 Likes
2

I wanted to add that one of edX’s original goals for MFEs was to greatly speed up our ability to get UI changes to Production. We went from hours for edx-platform UI changes to be deployed, down to minutes to go from development to Production in an MFE.

1 Like
3

My experience with MFEs when it comes to extending edX platform is not great too. I was not able to find any guidlines on how edX plugins are supported in MFEs. It appears we don’t have any recommend approach on applying custom themes to MFEs.

1 Like
4

In another project, I have run into a similar problem and I solved it in a simple way: there is one shared docker image for everyone, but the startup command in the Dockerfile is:

CMD npm run build && npm run start

So, everyone can configure it as needed via environment variables, and then deploy from the same image. Yes, the deployment is slower because it has to build before it runs, but overall it’s much simpler for everyone involved. And on Kubernetes, the process of rolling out a new version will of course keep running the old version until the new frontend container’s has completed its build and is passing health checks, and then start serving requests from the new frontend pod and terminate the old one. (Though this project also builds much faster than edX MFEs do.)

Can you use npm ci instead of npm install ? I think it can be significantly faster.

I suspect that this is a hard problem to solve given the current state of affairs. Certainly, there are ways to make the build much faster: replace babel with swc, remove polyfills and transforms for older browsers, use the latest webpack version, etc. But the amount of work and magnitude of changes required may be too much. And I worry the last thing Open edX needs is yet another major change to how frontends work while the last one is still in the process of being implemented.

Of that 2.7 MB, I imagine that very little is unique to the accounts MFE. It should be possible to use webpack externals to serv common bundles of JS and CSS from a CDN instead of duplicating them within each MFE. For example, React, Paragon, Font Awesome, CoreJS, and Moment could all be served from a public CDN or a “MFE core” bundle served by each Open edX instance. From a quick look at the config it seems like externals isn’t being used at all.

2 Likes
5

I think MFEs are great in one way it forces frontend out of the Open edX platform monolith and making a lot of APIs as a side product.

Like @Zia_Fazal mentioned plugins are not yet designed. I think it even harms pluggability and extensibility of the platform.

At Appsembler we’re still looking for ways to overcome this limitation – we have one solution in mind but it’s not yet mature.

6

As a programmer myself, imho worst decision was jumping on the “react” bandwagon… after angular the worst thing invented… even FB on the browser is simply crappy…

7

Looking at the comments here in this topic, I think it’s fair to say that the general attitude towards MFEs is quite polarized. Some of the reactions are kind of knee-jerk, but I guess I had it coming when I asked for the “feelings” of the community.

The question that I would like to ask is the following: do we, as a community, want to continue the push towards microfrontends? If yes, what changes do we need to implement, and with what priority? If not, what’s the alternative, drawing lessons from the past?

3 Likes
8

@regis, having recently been hired by tCRIL to help coordinate work on the Open edX frontend(s!), I find your post to be perfectly timely. :slight_smile: I find all your points valid, as well as everybody else’s (including @insad’s: I myself have mixed feelings about the whole jumping-on-the-latest-bandwagon mindset that is endemic to frontend architecture in our industry).

@braden, as usual, gives excellent suggestions on how to address the deployment problem, and @omar brings up the all-important topic of pluggability, which, as things stand, is now measurably worse than before. The latter I’ve been tasked with investigating specifically, but my first and foremost priority is actually to build a roadmap collaboratively with the community.

With this, allow me to invite you all to tomorrow’s frontend working group meeting, where I intend to kick this effort off. It’s unlikely the outcome of this first meeting will be a one-line roadmap with “Ditch MfEs!!1” at the top, but hey… The idea here is to listen to what y’all have to say. See this calendar for more info, and make sure to join #frontend-working-group on Slack.

PS: I realize joining Yet Another Meeting may not be an option for everybody, so I intend to document the sync discussion here in the forum, so it can continue in a-sync fashion.

2 Likes
9

Lot of valuable insights here about configuration, customizing, and shipping MFEs, but I’m not sure the full picture is here yet. Before drawing conclusions about the value of the framework and considering alternatives, I’d like to hear about folks’ experiences with building MFEs from scratch or working on new features for existing MFEs.

To that point, +1 to Adolfo’s suggestion to bring this to the frontend working group.

1 Like
10

Externals aren’t being used at all, it’s true. One of the original principles of MFEs was that we wanted them to be independent and flexible to adopt different versions of their dependencies as necessary. I don’t think the usefulness of that principle has proven itself, however, and that we’d benefit far more from doing what you suggest.

The devil is in the details, though. The core MFE library would enforce consistency between MFEs, and would presumably need to be versioned.

  • Do we bump its major version each time a breaking change occurs in one of our dependencies?
  • If its not versioned and MFEs use “latest”, how do we ensure that they’re compatible?
  • How do we preserve our ability to incrementally update MFEs so that we don’t have to do a “big bang” upgrade of otherwise decoupled frontends?
  • If we allow MFEs to pin themselves to a version of the core library, how do we ensure that we minimize the number of active, pinned versions so that we’re not in effectively the same situation we are today?
  • What goes in the core library and how do we vet new additions?
  • Etc.

That said, I think it’ll have a huge impact on our download size and I support it. :slight_smile:

11

Based on what you said, I would suggest this: the “core MFE bundle” is nothing more than a set of minified static .js files like this:

Open edX MFE Core - Nutmeg

  • react-17.production.min.js
  • react-dom-17.production.min.js
  • react-18.production.min.js
  • react-dom-18.production.min.js
  • paragon-4.0.min.js
  • paragon-4.1.min.js
  • bootstrap-4.min.js
  • …

Basically, include vetted major versions of common dependencies used by most MFEs, but allow each MFE to select the version that they want to use. If a given MFE needs an older version or a newer version than the recommended versions included in this “core MFE bundle”, that’s discouraged but OK - they can just include it directly as a dependency like they do now. But the idea is that wherever we have very common dependencies (e.g. React 17), they should all be downloading the exact same JS file, and only once.

I don’t think versioning of such an “MFE core bundle” needs to be complex; like a CDN, once a major version is added to the bundle it can be hosted there essentially forever. Now, say you want to add a new major dependency like React 18: you can add it to the “MFE core bundle” and start using it in various MFEs. Any Open edX deployment that’s following master will presumably deploy the master version of the “MFE core bundle” to its CDN at the same time as updating any individual MFEs, so there is no specific need to coordinate the rollout or version numbers. Anyone who is deploying named releases will also presumably be deploying a specific version of the “MFE core bundle” and any MFEs that are part of that named release version will be compatible.

2 Likes
12

I’m with ya! :+1:

-D

13

@arbrandes Unfortunately I am not going to be able to attend today’s frontend meeting :-/ I would like to convey to the working group my strongest worry concerning the first item I listed above, which is that building MFEs takes too much resources. @braden you suggest to run npm run build && npm run start at runtime, but such a change would only displace the problem. By building the MFE at runtime, we cannot leverage the Docker layer cache anymore. This means that the build time needs to be paid even when there is no configuration change. Also, on my demo server, npm run build takes 2 min 15 s of CPU time for the account MFE only. With n MFEs using up t seconds of CPU time for every build, the total build time of all MFEs becomes O(n x t). This total build time very quickly (starting with n = 1, t = 15s) becomes the dominant factor for launching any Open edX platform. The problem will become worse with linear speed as we add more MFEs.

In other words: to launch any Open edX platform, users are waiting a long time for MFEs to build, and the problem will become gradually worse with every new MFE.

If we are to ramp up the MFE effort, then we imperatively need to find a solution to this problem; this is not an opinion, it’s a fact.

1 Like
14

We’ve built 2 MFEs from scratch and the process was very straightforward, I personally have a positive experience when working with MFEs. Deploying the new MFEs using Tutor was also a breeze.

This is personally the main pain point I have when working with MFEs.

I also want to add that the UI looks way better with the MFEs. Although I understand the conversation is more focused on the experience of developing and deploying MFEs, I just wanted to mention this because I think at the end of the day, the goal is for the user to have a pleasant experience and I think MFEs are a massive step forward in that sense.

To add to the list of questions:

  • What’s in place to add a custom MFE to the User Experience? For example, we have a dashboard MFE and I am looking for a way to make it the default. I can use Django redirects or add configuration to the webserver but ideally, I’d like to easily be able to add a flag.
2 Likes
15

Very busy this week, but just I took a look at https://github.com/openedx/frontend-app-profile/blob/master/package.json - I counted 30 (!) dependencies and 11 dev dependencies, you can easily multiply the total count by 4 or 5 or so, to get an idea of the node packages that are being downloaded even to build what is a simply data entry form. Is really all this bloat the way to go forward?

Besides, for daily use (keeping up with mails, social media, reading stuff etc.) I use an older Samsung tablet, that is perfectly fine, but experience with all those react based apps (starting with the Facebook browser app - as I don’t want the Facebook spyware battery draining app) is really to cry about. It’s absolutely not a nice user experience!
Absolutely not a fan of react… lots of bloat and obscure things going on in the background, without any need imho.

16

As promised, this is me trying to distill what went on during last Thursday’s frontend working group meeting. Since almost all of it was dedicated to discussing the points raised in this thread, I figure this is the best place to post.

On MFEs in general

Because practically all solutions proposed (here and in the meeting) revolve around standardizing how Open edX micro-frontends should work, it is no surprise that there was some discussion around how MFEs came about, why they are the way they are, and what should they look like in the future.

As far as their inception, @djoy summarized it best: MFEs were supposed to make it easy for people to do what they want to do, freeing developers from the constraints of the current monolithic platform, while at the same time setting them up with best practices (testing, structure, etc). Few will dispute that this was, in fact, achieved, as @BbrSofiane was keen to point out. Having created an MFE from scratch myself, I can attest to the fact that developing one is much simpler (and thus, more pleasurable) than having to deal with all the current edx-platform baggage.

However, we’ve seen that this freedom came with a cost. For instance, the many differences between even the few existing MFEs were, for example, a heavy burden during the Maple release cycle - in particular due to the way Tutor deploys them. It’s becoming apparent that the cost of this almost total freedom is just not worth it: hence, this thread.

Creating a roadmap for how to address this - as well as the individual issues - was out of scope for the meeting. However, it is clear that we - as in, We The Community - need one. This is arguably the most important takeaway: let’s build it! I suggest we start here and now, by further discussing the individual issues until we have coherent proposals that address the worst of the pain.

The Three Main Issues

These were identified during the meeting as the three main issues.

Reducing build time

This is @regis’ first point. It was suggested during the meeting, to general agreement, that a 2-minute build time is actually not that bad. However, it was quickly acknowledged that indeed, as you add more MFEs, a full run taking dozens of minutes can be predicted easily.

The following ideas came up:

  • Changing the build backend to something else. @Ben_Warzeski suggested a Rust backend, though it wasn’t clear which one. Can you expand a bit here, Ben?
  • @Binod_Pant suggested we optimize the build process. There might be relevant gains from simple things such as making sure the process is not memory-starved, for instance. But there’s certainly more to do. Webpack themselves put out a guide.

Lack of uniformity

The original issue was raised in relation to build uniformity (or lack thereof), but it quickly became apparent that there would be gains across the board if MFEs were to be much more standardized than they are now. We briefly discussed three broad categories of standardization: the build process, engineering best practices, and dependencies. These specific points were brought up:

  • Build configuration differences should be ironed out
  • Linting rules should be the same everywhere
  • Data management should also be standard (RTK?)
  • API calls and methods should also be standardized server-side

Note that very little time was devoted to exactly how such uniformity could be achieved. For instance, as a community, how can we make sure that new MFEs follow the rules - whichever they end up being?

Reducing browser downloads

We didn’t spend much time here, but pretty much everybody was in agreement that @braden’s idea of core bundles was a very good way forward.

Pre-fetching was mentioned, somewhat tangentially. Though this does not address download size, if done right it can certainly improve UX.

The near future: Building a Roadmap

A large part of the reason I’m taking the time to commit all this to writing is that, as some of you have already heard, I recently transitioned to a full time position at tCRIL, and my first mission is to facilitate the creation of a frontend roadmap with as much community involvement as I can get. This means:

  • There’s going to be more posts like this. :stuck_out_tongue:
  • I’m going to continue nagging people to participate: we want more people involved in the decisions than just tCRIL and 2U employees!

So, with that out of the way… Let’s build a roadmap, shall we? You can help by voicing your opinion/suggestions in this thread, but if you can make it to the meeting on Thursday, please do so!

4 Likes
17

On the topic of Rust: I asked in a recent Arch Hour meeting “would it actually be practical to consider replacing some of our front end Node-based build tooling with something faster utilizing Rust, like Next.js?” I’ve since noticed that Parcel is also using Rust (and recently added a Rust-based CSS minifier). I’m still not sure how practical it would be to consider switching to one of these in the near future, but it seemed worth tossing it out there with build times becoming a concern.

2 Likes
18

@jmbowman, both Next.js and Parcel look promising in terms of cutting down build time (among many other reported advantages). The question is indeed whether it would make sense to use them. I’ll take some time over the next few days to a little digging. Thanks!

19

Adding a suggestion to reduce download bundles from @AdamStankiewicz, via #frontend-working-group:

20

I have been doing a lot of testing for this, trying to find ways to speed up the install/build process.

First I was able to make the build/install time drop by a factor of 4 by enabling the docker built kit [1] feature, by just setting DOCKER_BUILDKIT=1 docker build mfe/ this allowed docker to run or execute a command in parallel for stages which don’t depend on each other.

Secondly besides the build time, (here what I mean by build is the actual npm run build command not the docker build) almost equal time was for npm install because npm install is expensive, it writes a lot of files or modules to the systems… while trying stuff around, I tried using pnpm [2] it cut the install by half, however, it was useless, because after running pnpm install I couldn’t build/create the bundle.

However that can be promising, I didn’t want to spend a lot of time trying stuff out before we start using node 14 with npm 8 because otherwise, I might come up with something that would be unnecessary or useless once an upgrade occurs.

Another thing that might be promising is, since npm >=7, npm introduce workspace[3], which can be helpful for speeding up install/build time for projects that are related to each other. However for us to make use of it or another similar tool, it would be super useful if the MFEs would depend on the same version for shared packages, what I mean by that is, for example, there are 19 npm packages that are shared between the (account, profile, gradebook, learning) however of those 19, only 2 package are being shared with the exact version number. [4]

[1] Build images with BuildKit | Docker Documentation
[2] https://pnpm.io/
[3] workspaces | npm Docs
[4] It's just a simple script that check for the package.json for selected MFEs and it then outputs two main thing: · GitHub

3 Likes