Problems applying Customised MFE libraries (eg "frontend-platform") to a production open edx platform (using "tutor local")

[Dimitris_Angelakis]

Hello everyone.

I have been trying to make some modifications to the frontend-platform part of the MFEs so it will also support the Greek Language “el” (for the time being only a handful of languages are supported, I think just 14 of them), as, our platform runs in Greece in Greek language and without it we cannot keep up with the new Open Edx releases after tutor became the default installation method and MFEs are being used by default instead of the old django/mako templates in certain parts of the platform.

The changes needed to be made are minimal, just some minor additions to 3-4 files in the frontend-platform library.

After I bindmounted the forked MFEs’ code (eg the Learning MFE to my tutor dev installation) and edited the frontend-platform files directly inside the node_modules/@edx/frontend-platform path where they were installed, the Greek language worked without problems, but it seems impossible to transfer the changes to our “tutor local” open edx installation which we’ll use for our production platform.

First, I tried to use the following tutorial from here:
https://github.com/openedx/frontend-build#local-module-configuration-for-webpack

So I forked the frontend-platform to be able to make my tutor dev instalation use the forked one (so to edit its original code instead of the installed library inside node-modules/@edx/frontend-platform of each MFE), but it didn’t work due to some errors ending up in ignoring my “module.config.js” and using the locally installed frontend-platform inside each MFE’s node_modules.

I did the test with frontend-app-learning bindmounted to my tutor dev…

My “module.config.js” content is this:

module.exports = {
  localModules: [
    { moduleName: '@edx/frontend-platform', dir: '../src/frontend-platform', dist: 'dist' },
  ],
};

When I run “npm install && npm run build” it appears to be completed with just some warnings, although one of them seems to contain an error:

WARNING in ./node_modules/react-responsive/dist/react-responsive.js
Module Warning (from ./node_modules/source-map-loader/index.js):
(Emitted value instead of an instance of Error) Cannot find source file 'webpack:///dist/react-responsive.js': Error: Can't resolve './webpack:///dist/react-responsive.js' in '/home/user/WORK/PEK/OPENEDXREPOS/TMP/frontend-app-learning/node_modules/react-responsive/dist'
 @ ./node_modules/@edx/paragon/dist/index.js 106:0-102 106:0-102 106:0-102 106:0-102
 @ ./src/course-home/progress-tab/ProgressTab.jsx
 @ ./src/index.jsx 21:0-65 91:46-57

At the end I get this message: “webpack 5.50.0 compiled with 3 warnings in 29774 ms”

Since it seems to not produce any errors I then try to mount the frontend-app-learning with “tutor dev start learning --mount=.” but I get the error trace shown in the following file:

https://gist.github.com/dagg/46fe73a107ae4c2c3773a9cd5926e858

That wasn’t a big problem though since the changes I wanted to make were minimal so I thought of just making the changes to my forked frontend-platform, pushing it to my github repo and then creating and enabling a tutor plugin with a “mfe-dockerfile-post-npm-install” patch to make my “tutor local” open edx platform use frontend-platform from my github repo instead of the original, following these steps from here:

https://github.com/overhangio/tutor-mfe#customising-mfes

My tutor plugin’s code is very simple as it’s shown here:
(it actually shows the path to the original frontent-platorm by open edx but, before trying that, I used my own repo)

from tutor import hooks

hooks.Filters.ENV_PATCHES.add_item(
    (
        "mfe-dockerfile-post-npm-install",
"""
RUN npm install '@edx/frontend-platform@git+https://github.com/openedx/frontend-platform.git'
"""
    )
)

This produced errors (when I used my own repo) and failed to even continue with the “tutor local quickstart” installation.

The error trace is shown here:

https://gist.github.com/dagg/bfbfade3f53cee0f45f9650c5a266073

To eliminate the possibility that I had errors in my changes of the frontend-platform library (since I could not test the changes before applying them to production for the reasons I explained firstly above), I tried to use the original repo of frontend-platform from github (as shown on the plugin code I have quoted right above), but I kept getting the same errors, so it wasn’t my changes that produced the errors but the method of applying them to the platform.

I have even tried using the exact same examples as shown in the help, using the following in my plugin:

First I tried to aply the original frontend-component-header from the npm repo:

RUN npm install '@edx/frontend-component-header@npm:@edx/frontend-component-header-edx@latest'

And then the frontend-component-footer from its github repo:

RUN npm install '@edx/frontend-component-footer@git+https://github.com/edx/frontend-component-header-edx.git'

(Actually, there is an error I think in this step as it is shown in the help, the second part directs to the “frontend-component-header” instead of the “frontend-component-footer”, I believe this must be corrected. Although I tried both and none worked with similar errors)…

Again, the errors are shown in the following corresponding files:

https://gist.github.com/dagg/7a779d8ea86343870231448a5278d91d

https://gist.github.com/dagg/19d211fa51671a15f72cbfb5de8db262

Because it’s very important for us to be able to use the new releases in our native language, I would really appreciate some help on how to overcome those problems and also know if it’s something I am doing wrong or there is something wrong with the method of editing those components and applying them to the platform.

It would be very important and helpful for anyone who needs to make certain changes to the MFEs and their libraries, to be able to apply those changes to their production platform with ease and without such errors or too complicated steps.

For the record: I have no problem editing one or more of the actual MFEs (eg the Learning MFE) and applying them to the production platform (tutor local open edx installation) by changing the value which contains their source repo (eg: MFE_LEARNING_MFE_APP).
To my knowledge so far, the problem appears only with such libraries as the “frontend-platform” that are used by those MFEs and are installed inside each MFE’s node_modules.

Moreover, I can make a PR with my changes on the frontend-platform for including the Greek language (el) but even if my PR will be accepted and included to following versions, I believe this would take time and I’d prefer for the time being a solution where I can apply these changes myself instead of waiting for the next releases, as it will be much harder to migrate our data from Lilac to the latest future open edx release.

The machine I am using for my tests has the following versions of tutor, docker, docker-compose, docker compose, in case they are of importance:

tutor, version 14.1.0
Docker version 20.10.21, build baeda1f
docker-compose version 1.29.2
Docker Compose version v2.6.0

Thank you in advance for any help on this matter! I hope it’s something trivial!

@regis @arbrandes @mrtmm @djoy

[arbrandes (Adolfo R. Brandes)]

Hey @Dimitris_Angelakis,

You’re hitting some of the difficulties in using Tutor to develop MFEs locally. For the moment, most frontend developers still use the devstack, and that is what I suggest you do. The main advantage is being able to run an MFE outside a docker container (via regular npm start) and still have it connect to the rest of the Open edX instance. This allows you to configure webpack module aliases as instructed in the frontend-build README.

(@kmccormick has been hard at work at improving Tutor for development: you might want to try his quickdev plugin instead of the devstack.)

This doesn’t solve the issue of bringing the change into Tutor, of course. More on this below.

enabling a tutor plugin with a “mfe-dockerfile-post-npm-install” patch to make my “tutor local” open edx platform use frontend-platform from my github repo

You’re on the right track regarding using your own repositories. That is the easiest way to bring changes into Tutor.

However, I don’t recommend a manual install of the latest versions of any npm packages unless you’re an experienced Open edX frontend developer; you’re very likely to run into the dependency errors you’re seeing. You should instead base your modifications on the exact version of a package that the version of the MFE you’re running installs, commit them, and then install that. This applies not only to frontend-platform, but also frontend-component-* and others.

You should also consider forking the MFE itself, and making the dependency modifications in its package.json. This will make it much easier to check that builds will run as intended. You can then point Tutor to your fork via:

tutor config save --set MFE_LEARNING_MFE_APP="{'name': 'learning', 'repository': 'https://github.com/youruser/frontend-app-learning', 'port': 2000}"

there is an error I think in this step as it is shown in the help, the second part directs to the “frontend-component-header” instead of the “frontend-component-footer”,

Nice catch: there’s clearly a copy-paste error in there. PR opened to fix it.

[regis (Régis Behmo)]

I’ve been following Dimitris’ issues pretty closely as we’ve been working together on this matter. To the best of my understanding, there is an issue with installing frontend-platform from source. This is not a problem of installing the right version.

To understand what is going on, run the following commands in, for instance, the frontend-app-profile repo (open-release/nutmeg.master):

$ npm install --no-save @edx/frontend-platform@1.15.6 # this the required version in Nutmeg
$ ls node_modules/@edx/frontend-platform/
analytics  config.js      constants.js      i18n      index.js.map   initialize.js.map  logging       package.json  pubSub.js.map  README.md  utils.js
auth       config.js.map  constants.js.map  index.js  initialize.js  LICENSE            node_modules  pubSub.js     react          testing    utils.js.map

The installed package contains all the necessary files.

Now, install from the GitHub source repo:

$ npm install --no-save @edx/frontend-platform@git+https://github.com/openedx/frontend-platform#v1.15.6
$ ls node_modules/@edx/frontend-platform/
docs  jsdoc.json  LICENSE  node_modules  openedx.yaml  package.json  README.md  service-interface.png  src

Notice how in the latter example the package source files are all contained in the src folder. This causes npm run build to fail further down the road.

This leads me to conclude that there is a problem in how frontend-platform is packaged.

The fact that we cannot install frontend-platform from GitHub means that it’s extremely difficult to install a forked version of frontend-platform – which we need in some cases, such as installing an extra language/locale, which @Dimitris_Angelakis is trying to do.

How are frontend-platform developers installing their local forks in the MFE environments?

[arbrandes (Adolfo R. Brandes)]

@regis

How are frontend-platform developers installing their local forks in the MFE environments?

This is in the README:

1. Delete the node_modules directories in the host project: rm -rf node_modules/
2. Move frontend-build inside the host project and delete its node modules folder mv ../frontend-build ./ && rm -rf frontend-build/node_modules
3. Install the development version of frontend-build npm i --save-dev @edx/frontend-build@file:./frontend-build.

If you look at the build process it becomes a little clearer why these acrobatics are necessary, and why npm-installing directly from github might fail.

So… Yes, this makes customizing frontend-platform harder. It’s why I offered forking both the MFE and frontend-platform as a potential way out. At least it would help tracking the problem down.

I have a lot on my plate this week, but as soon as I get some time I can help look into a fool-proof, Tutor-flavored way of doing this so that we can put it in the README. (If anybody else wants to pitch in earlier, please do.

[regis (Régis Behmo)]

Thanks for this detailed answer Adolfo, I appreciate it. With your help I’m sure we’ll be able to move forward.

@arbrandes

If you look at the build process it becomes a little clearer why these acrobatics are necessary, and why npm-installing directly from github might fail.

Indeed the build process explains why we can’t install from GitHub, but I still don’t understand why the build process needs to be so complex. Having to manually cp and rm stuff around does not feel “right”. Of course this is a question that is less of an emergency.

[arbrandes (Adolfo R. Brandes)]

@regis

Having to manually cp and rm stuff around does not feel “right”.

Agreed. I’m going to try and see if there’s a better way to do it.

[Dimitris_Angelakis]

Adolfo and Regis thank you both so much for the help on the matter!

I should probably mention that I’m pretty new to node (and react) and I’m struggling a bit to understand how the whole procedure works (especially in combination with the tutor way of implementing things) and most of the time I’m not sure I know what I’m doing, I just try to follow the existing README help if available.

@arbrandes

You should instead base your modifications on the exact version of a package that the version of the MFE you’re running installs, commit them, and then install that.

I tried lots of things the last few days after your suggestion but they don’t seem to work or I do something wrong.
I actually used “nutmeg.2” tag code of "frontend-app-learning" and tag v1.15.6 of "frontend-platform" (I saw in frontend-app-learning's ‘package.json’ file that’s the version of "frontend-platform" used, like Regis mentioned already) without changing their code (just to test if it will work), and I tried to incorporate them to my production (tutor local) platform but I am still receiving some errors. I even disabled the other three MFEs so they won’t “complain” about the changes, using something like:

tutor config save --set MFE_ACCOUNT_MFE_APP=null

but still no luck (I’m not sure though if disabling them it also blocks them from being build in there)…

I also noticed that the other MFEs, instead of v1.15.6 of "frontend-platform" they use v1.15.1 which complicates things a bit more since it seems I cannot use a different version for each MFE using this method…

@arbrandes

You should also consider forking the MFE itself, and making the dependency modifications in its package.json. This will make it much easier to check that builds will run as intended.:

This seems to be the solution to the different version of "frontend-platform" in each of the MFEs, but when I tried it with frontend-app-learning again it failed…

@arbrandes

• Delete the node_modules directories in the host project: rm -rf node_modules/
• Move frontend-build inside the host project and delete its node modules folder mv ../frontend-build ./ && rm -rf frontend-build/node_modules
• Install the development version of frontend-build npm i --save-dev @edx/frontend-build@file:./frontend-build.

If i understand well this is for the development procedure (as it is titled in the README), right? Still I also hit the wall when I tried it but I didn’t insist on this, as my main problem is with pushing my changes in "frontend-platform" in production, and not so much the development part.

Still I really appreciate all those suggestions and help and I will keep trying in case I missed something due to my limited experience on the matter!

@arbrandes

as soon as I get some time I can help look into a fool-proof, Tutor-flavored way of doing this so that we can put it in the README

This would be a huge help for us, especially if you manage to describe, not only the development part but also the “Tutor-flavored way” of applying any changes on frontend-platform (or other components) to a "tutor local" production implementation of the platform.

@arbrandes & @regis Thank you for all the help!

[Wasabi (Daniel Herrmann)]

Dear all,

I hope it’s fine for you for me to jump in here. I’m trying to achieve exactly the same goal as Dimitris, just for the German language. I’m also running tutor locally and trying to add the German translation (which is already available on Transifex) but not one of the already supported languages.

Same as Dimitris I noticed that only very few changes are needed to add support for the German language in general, but its very hard and not documented yet how to incorporate the changes into the Tutor environment.

If you could find any time to somehow document that fool-proof / “tutor flavoured” way, that would be very much appreciated also from our side. I’m also happy to help, although I’m not sure I can be of much help except for testing. If you only write down the steps needed in short bullet points, I’m happy to support by making a readable part of the documentation out of that.

Thanks for your help!

[Anh_Vu_Nguy_n (Vu Nguyen)]

I’m having the same problem while trying to add an unsupported language in tutor MFE, I hope that there will be an easy way to do it.

[onurcan (onur)]

I’m experiencing the same issue for the Turkish language. I tried to run also the forks of MFEs and Frontend Platform but the issue "Module not found: Error: Can't resolve '@edx/frontend-platform/..." is still there.

However, I hope this helps a bit, while trying different ways to resolve the adding extra translations I figured out that Frontend Account App somehow accepts extra translations.

I only did the following changes to the fork of Frontend Account App:

src/account-settings/site-language/constants.js
// add a new language
  {
    code: 'tr-tr',
    name: 'Türkçe',
    released: true,
  },

src/i18n/index.js
// add these changes
import trMessages from './messages/tr-tr.json';

const messages = {
  'tr-tr': trMessages,
}

And finally add translation file downloaded from Transifex to the src/i18n/messages/tr-tr.json path and change MFE Dockerfile to fetch your forked Git repository.

Rebuild the MFE images tutor images build mfe and tutor local start -d. You can change your language to this extra language which was not included in Frontend Platform. Honestly, no idea why this works with Account app.

[Wasabi (Daniel Herrmann)]

Dear all,

I think I finally managed to find a solution that is also acceptable from tutor perspective. I’ll give a complete rundown here, so if someone else is facing the same issue, you should be able to follow the guide.

** Note that all changes described here must be repeated for each new release!** For the sake of an example, we’ll be modifying the most recent olive.2 release.

Background information

Some very high level information. MFEs are essentially standalone and independent ReactJS applications which communicate using the OpenEDX API with the main application stack. They all leverage common modules (like frontend-platform for example), but those are just regular npm dependencies.

When starting a Tutor environment, the tutor-mfe plugin is responsible for managing the MFEs. On a very high level, without going too much into detail, the plugin builds a Dockerfile based on a template and the plugin configuration, and this Dockerfile is then used to build the mfe container.

There is one single container (mfe) which hosts all MFEs. When the container is built, the Dockerfile contains various steps for each MFE, like clone the source code, apply some i18n changes, do npm install and build. That means the container essentially just serves static assets (the compiled react application).

Now the general idea to add additional languages is to create a fork of each MFE repository we’re interested in and make sure those repos are used to build the new MFE container.

This next section just describes the changes you need to do, the second section focusses on how to do them.

Required Changes

In general, there’s two changes that need to be done:

• Custom translation needs to be added to each MFE you’re interested in
• Language need to be added to the dropdown in the account MFE so users can actually choose

To add your language to the dropdown on the account page, you’ll need to modify src/account-settings/site-language/constants.js in frontend-app-account:

index 6854958..efea6e0 100644
--- a/src/account-settings/site-language/constants.js
+++ b/src/account-settings/site-language/constants.js
@@ -14,6 +14,11 @@ const siteLanguageList = [
     name: 'Català',
     released: false,
   },
+  {
+    code: 'de-de',
+    name: 'Deutsch',
+    released: true,
+  },
   {
     code: 'es-419',
     name: 'Español (Latinoamérica)',

Second, to add your language to the MFE, the following steps need to be done (its the same for each MFE, so you need to apply these changes to each MFE):

Update src/i18n/index.js to include your language, in this example, German has been added:

index 8a01428..b762993 100644
--- a/src/i18n/index.js
+++ b/src/i18n/index.js
@@ -1,6 +1,7 @@
 import arMessages from './messages/ar.json';
 import caMessages from './messages/ca.json';
 // no need to import en messages-- they are in the defaultMessage field
+import dedeMessages from './messages/de_DE.json';
 import es419Messages from './messages/es_419.json';
 import frMessages from './messages/fr.json';
 import zhcnMessages from './messages/zh_CN.json';
@@ -15,6 +16,7 @@ import ukMessages from './messages/uk.json';
  
 const messages = {
   ar: arMessages,
+  'de-de': deMessages,
   'es-419': es419Messages,
   fr: frMessages,
   'zh-cn': zhcnMessages,

Next, fetch your language file from Transifex (if it exists), you need a JSON file. Some common transifex repositories:

• https://www.transifex.com/open-edx/edx-platform/frontend-app-account/
• https://www.transifex.com/open-edx/edx-platform/frontend-app-authn/
• https://www.transifex.com/open-edx/edx-platform/frontend-app-learning/
• https://www.transifex.com/open-edx/edx-platform/frontend-app-profile/

Once you’ve located the transifex project you’re interested in, click on your language and click on Download for use. This should give you a JSON file. Rename this file according to the import in index.js and place it in i18n/messages/ (in our example, this would be i18n/messages/de_DE.json).

Apply changes

To apply these changes, perform the following steps for each MFE you want to modify

• Fork the MFE (make sure not only to copy the master branch, you need everything)
• Clone it to a local directory
• Go to the official OpenEDX Repository for the MFE you’re modifying and find the commit hash associated to your release (8dbf20b in this example).

• In your local clone, create a new branch off this commit: git branch your_org_olive.2 8dbf20b && git checkout your_org_olive.2
• Make the changes described above
• Next, we’ll commit all changes and move the release tag to the new version you’ve just created. Alternatively, you can use a custom tag and specify the version attribute for the MFE config later.

# Add all modified files
$ git add -A

# Commit all changes
$ git commit -m "your org specific olive.2 release"

# Delete existing release tag
$ git tag --delete open-release/olive.2

# Create a new release tag pointing to your new commit
$ git tag open-release/olive.2

# Alternatively, omit the previous two steps and simply create a new tag:
$ git tag your-org/olive.2

# Assumes origin is your remote
$ git push origin
$ git push origin --tags -f # Force (-f) is only needed if you changed the original release tag, as the tag still exists in the remote repository

Make Tutor use your new forks

The final step is to tell Tutor to use your new forks. To do this, modify your config.yml ( $(tutor config printroot)/config.yml) and add the following for each MFE:

MFE_<MFE_NAME>_MFE_APP:
  name: <mfe_name>
  port: <mfe_port>
  repository: https://github.com/your-org/frontend-app-<mfe_name>

# Example
MFE_AUTHN_MFE_APP:
  name: authn
  port: 1999
  repository: https://github.com/your-org/frontend-app-authn

# Example with version
# Optionally, if you don't use the official release tag for your changes, you can do:
MFE_AUTHN_MFE_APP:
  name: authn
  port: 1999
  repository: https://github.com/your-org/frontend-app-authn
  version: your-org/olive.2

Note that you can find the defaults (MFEs, ports and their Repositories) here. Note that you need to prefix the configuration variable names by the plugin name (MFE in this case).

Warning: While it might be tempting to simply change them in the Dockerfile ($(tutor config printroot)/env/plugins/mfe/build/mfe/Dockerfile), this is a bad idea as those URLs would be overwritten by the next tutor config save.

Last but not least, rebuild the MFE image and restart the container, picking up the new image:

$ tutor images build mfe
$ tutor local dc up -d mfe

I hope this helps someone.

[Wasabi (Daniel Herrmann)]

@onurcan you may want to consider changing your modified Dockerfile to the method I propose to avoid the URLs being overwritten.

[Dimitris_Angelakis]

Thank you @onurcan and @Wasabi for your suggestions!

Due to lots of other issues, too much work and lack of time I haven’t managed to check your suggestions yet to see if they will work on my case, but I am starting with it now and I hope to have some results or possible input on the matter.
Moreover, I will be attending the Open Edx conference next week and I hope to talk in person with @arbrandes about this, so, hopefully, I might have an update soon.

I just want to add that the whole MFE transition creates even more problems for our Open Edx implementation apart from the language issue, as we have made some changes to the previous mako templates that are of course no longer used in the MFEs, so, it’s imperative for us to be able to edit some of the MFEs and then embed them back to the platform, so, a solution to this would be much more helpful for us (and possibly others) apart from the mere implementation of a new language.

[Dimitris_Angelakis]

After some time and after several trial-and-error attempts I had finally managed to make the MFEs to work with my custom changes for supporting the Greek language (and the corresponding translation files) thanks to @Wasabi (although I did everything similarly, the only step I had missed was to checkout the appropriate tag before applying the changes to my Open Edx local implementaion), and I managed to do it just a couple of days before the Open Edx conference in MIT where I have also met with many really amazing and helpful people!

I have created my own step-by-step guide, quite similar to that of @Wasabi and I can post it here if anyone would like to see it! I might post it in one of the following comments here.

---

I still have some problems though with certain parts of the platform, and more specifically, my latest encounter of such a problem is with the ‘frontend-component-header’ (and footer) component, which needs a different approach than the MFEs, and that’s because it needs to be installed as a library inside node-modules/@edx of each of the MFEs that use it, and although according to the documentation here:
https://github.com/overhangio/tutor-mfe#customising-mfes
the implementation seems quite straightforward with the use of a patch ‘mfe-dockerfile-post-npm-install’ inside a tutor plugin, it never works and produces dependency errors…

My guess is that (similarly to the ‘frontend-platform’ component that it’s actually being used inside the ‘frontend-component-header’ component) because this particular component is used by several MFEs (like the Learning and the Account MFEs), and each of them uses a different version of the ‘frontend-component-header’, there is no way using this tutor patch to tell each MFE to use a different version of the component…

For example, the olive.3 tag of the ‘frontend-app-learning’ MFE has as a dependency the version ‘3.1.0’ of ‘frontend-component-header’, while the olive.3 tag of the ‘frontend-app-account’ MFE has as a dependency the version ‘3.2.1’ of ‘frontend-component-header’, and the other MFEs also use different versions of the component…

Does anyone know if there is an easy way to overcome this problem?

Also, I have another question: if I want to use my specific github tag or branch of my ‘frontend-component-header’ fork, is it correct if I call it like following inside my tutor plugin?:

RUN npm install '@edx/frontend-component-header@git+https://github.com/dagg/frontend-component-header.git#mybranchortagname'

where ‘mybranchortagname’ is the name of the branch or tag with my custom changes…

I tried it like this and it doesn’t work because of the dependency errors, but I’m not sure it if tries to get the code from my branch or tag, or it gets the master branch… Either way, I still wouldn’t know which version to use as each MFE uses a different version of the component like I explained before…

@arbrandes @regis