Tweaking the webpack.dev.config.js to make frontend plugin development easier

I recently created a simple frontend plugin today and I spent a decent amount of time debugging the development setup. The main issue was around having the code in an indepent repo instead of just the env.config.jsx file.

The workshop on Customizing your Open edX site with Frontend Plugin Framework, provided most of the details. It provided details on using the env.config.jsx as the source of the plugin, and using NPM packages as the source of plugins. But developing the plugin in an independent repo was not covered. I am not complaining, just providing context.

I got it working after a few rm -rf node_modules and npm installs. Here are the things that gets this working:

  1. Mark all the library dependencies like react, react-dom, @openedx/paragon…etc., as peerDependencies in the plugin’s package.json and NOT dependencies.
  2. Declare them as external in the build config (if one is used).
  3. Do an npm install --legacy-peer-deps so that those libraries are not bundled. (TIP: Creating .npmrc with legacy-peer-deps=true makes it easier for later)
  4. Edit the the webapck.dev.config.js to have resolve: { symlinks: false }. This makes the host application (the MFE), to use it’s copy of the libraries like react to be provided as the external for the plugin instead of looking into the plugin’s node_modules.
  5. Install the plugin using npm install ../relative/path/to/plugin-repo.

With this, the plugin can live in a separate repo outside of the MFE. Otheriwse, the plugin needs be inside the MFE’s folder, like in a mono-repo setup.

Almost all of these are plugin dependent except the resolve: {symlinks: false} part, which needs to go into the MFE’s webpack config. I am wondering if this is something that could be added to all the MFE repos? Are there any reason not to do it?

3 Likes

We had some similar difficulties when we tried to write a complete plugin (backend, frontend, tutor in the same repo) and you might like to look into the solutions and workarounds we found.

We wanted to make the UISlots available to the MFE build process without having to publish them in NPM because we wanted the loop of change locally and build to work for someone that can’t publish to npm.

Two critical steps where to add the code in the local context: openedx-ai-extensions/tutor/openedx_ai_extensions/plugin.py at main · openedx/openedx-ai-extensions · GitHub and the post npm install patch also required the legacy-peer-deps that you mentioned (openedx-ai-extensions/tutor/openedx_ai_extensions/patches/mfe-dockerfile-post-npm-install-authoring at main · openedx/openedx-ai-extensions · GitHub)

I completely support making this more streamlined

1 Like

In addition to making things more streamlined, which sometimes takes more time than documenting current state, I’d recommend creating a how-to (if it doesn’t already exist), that captures steps from the Workshop and additional steps being documented in this post (and comments).

Note: The how-to could point to this post or a github ticket for streamlining, so it too could be updated if the ticket is ever closed. :slight_smile:

1 Like

@arunmozhi I’ve been working on a new sample-plugin repo that show how one could have frontend and backend plugins in the same repo that work together. As a part of that, I’ve documented how to set this up for local development. Since you ran into similar issues, I’d love for you to take a look provide feedback if you have time.

1 Like

@Felipe Thank you for the reference to the full Tutor plugin. If I understand this correctly, the approach is a bit different from the one in the Workshop. The openedx-ai-extensions plugin doesn’t require running the MFE outside of Tutor. That’s really helpful. Currently, I add a mount to get the authoring container to run indpendently, and then stop that container and run npm run dev locally. It’s a bit of dance, that’s completely avoided :ok_hand:

@feanil The documentation in the sample-plugin is top notch. I wish I had used that as the reference instead of the workshop video.

On the point of webpack config, it seems like the crucial part is creating a module.config.js to let webpack resolve the local plugin and dependencies correctly, accomplishing what {symlinks: false} did for me, but being explicit about it and be trackable. Cool. :grinning_face:

Is there a reason to prefer the module.config.js over {symlinks: false}? Does the module.config.js serve other purposes? Could we save the step of creating this file?

Hold that thought on saving time and work with/without module.config.js. I realized, there is a lot more that could be saved.

Tutor is a amazing feat of engineering and one great thing about it is - it’s opinionated. So, it makes a lot of decisions and codifies it for us. But it also trying to meet different requirements for different people using patches. Which is great. But by doing so, it becomes a bit unopiniontated. That means it leaves it open for the plugin devs to decide which patches to use. The patches come with more decisions regarding “when” and “hows” - post/pre install, build/runtime mounts..etc., Again great for flexibility.

The trade off is, there is a whole lot of strings where there could be typed Python (think openedx-common-lms-settings), validated YAML (think lms-env, lms-env-features), validated JS (think mfe-* patches). Now we are getting to a place where a set of decisions are being made to achieve specific goals - the sample-plugin, but we are storing it in documentation. What if we stored those decisions in code?

Taking the frontend plugin as an example - 3 things are must:

  • mfe-dockerfile-post-npm-install to install the package
  • mfe-env-config-buildtime-imports to patch the env.config.jsx file with the imports
  • PLUGIN_SLOTS.add_items() block to patche the JS

Now each of these 3 use multiline strings for configuration. It’s a pre-made decision for the goal of adding a component in a slot. So, why make the devs write it out as if they got to make it? Why can’t we simply define it like:

FrontendPlugin(
    package="frontend-plugin-my-plugin",
    local_path="/path/to/plugin-code",
    slots=[
        Slot(
            mfe="authoring",
            slot_name="org.openedx.frontend.authoring.course_unit_sidebar.v2",
            component="MyAwesomeComponent",
            operations=INSERT,
            priority=10
        )
    ]
)

and it doesn’t matter if build-context needs to be patched, module.config.js gets created or resolve flag is changed or not. We don’t have to worry if the plugin slot definition is missing a comma or if priority is a string or an integer, don’t have to wonder if mfe-env-config-builtdtime-imports or mfe-env-config-runtime-imports needs to be patched …etc.,

With that idea, I experimented with an LLM tool to create a proof of concept library → GitHub - tecoholic/tutor-patches: Proof of concept library for writing typed, validated Tutor plugin patches.

With that, I can now simplify this Tutor plugin

from tutormfe.hooks import PLUGIN_SLOTS
from tutor import hooks

hooks.Filters.ENV_PATCHES.add_item(
    (
        "mfe-dockerfile-post-npm-install",
        """
# Install the LTI Provider frontend plugin package
RUN npm install @tecoholic/frontend-plugin-lti-provider
""",
    )
)

hooks.Filters.ENV_PATCHES.add_item(
    (
        "mfe-env-config-buildtime-imports",
        """
import { AuthoringUnitPageSidebarWidget } from '@tecoholic/frontend-plugin-lti-provider';
""",
    )
)

PLUGIN_SLOTS.add_items(
    [
        (
            "authoring",
            "org.openedx.frontend.authoring.course_unit_sidebar.v2",
            """
            {
              op: PLUGIN_OPERATIONS.Insert,
              widget: {
                priority: 60,
                id: 'lti-provider-widget',
                type: DIRECT_PLUGIN,
                RenderWidget: AuthoringUnitPageSidebarWidget
              }
            }""",
        ),
    ]
)

to a much simplified version

from tutor_patches import FrontendPlugin, Slot, Operation

FrontendPlugin(
    package="@tecoholic/frontend-plugin-lti-provider",
    slots=[
        Slot(
            mfe="authoring",
            slot_name="org.openedx.frontend.authoring.course_unit_sidebar.v2",
            component="AuthoringUnitPageSidebarWidget"
        )
    ]
).register()

The library uses Pydantic Models for FrontendPlugin and Slot. So, it catches issues like wrong types, non-existent MFEs..etc.,

It works great in editors with “hover for documentation” support.

Why do this?

Improve DevEx. We are at a point where unavoidable decisions are being made to accomplish certain goals, but adherence to those are left to the developers. So a lot of time & work are being spent in documenting them in different places and searching for that documentation. So, putting them down into a library instead feels like a big win.

Disclaimer about the library

The tutor-patches linked in this comment is not in anyway intended to be a “project”. It’s built with an LLM to express an idea. Intended to be thrown away.

Over the years, I have found npm link (which is what npm install <../relative/path> does behind the curtains, and why you need resolve: {symlinks: false}) to be unreliable to the point of uselessness. It never behaves like one would expect, and this is before taking into consideration any cross-platform issues. A more palatable native alternative is to npm pack the code in question and then npm install ../path/to/packed.tgz… but of course the problem with that is that you have to do it every time you make a change. Not-exactly-hot reloading.

It is for these reasons that frontend-build relies instead on Webpack’s resolve.alias, exposed to developers via module.config.js. It’s not perfect: the downside, as you point out, is requiring developers to create the file in the first place. But at least it behaves as one would expect… with hot-reloading, and without the need for --legacy-peer-deps or any of the other hoops you had to jump through.

All of this said, I think there’s value in what you outlined as an alternative to module.config.js. I’m just unsure about making it an official recommendation, yet. Defaulting to resolve: { symlinks: false } in webpack.dev.config.js is probably fine, but I have mixed feelings about legacy-peer-deps. It can save you in a pinch, but depending on it as a fundamental mechanism usually means there’s some bigger underlying problem.

For example, the reason you used it for - avoiding installation of peer deps - might not be the same reason the AI extensions plugin uses it (I don’t know why it’s needed there). The most common case is actually to avoid checking for dependency conflicts - which is a bad thing to rely on in production, as it can cause hard-to-debug runtime issues.

You can avoid having to stop the desired Tutor container by manually specifying which ones you actually want to run in the first place. For example, I usually start my dev environment with:

tutor dev start -d mfe

This means that none of the “mounted” ones start, so I don’t need to stop them later for npm run dev to work.

But yeah, if you want develop a plugin and don’t want to run the host MFE outside Tutor, the local build context that Felipe refers to might be your best bet. (It’s a neat idea - first I hear of it!)

Out of curiosity: did you try to use npm pack && npm install instead of npm install --legacy-peer-deps --install-links? (See the problem I have with --legacy-peer-deps, above.)

This is not the first time folks have noted that either (or both) the env.config.jsx syntax or the tutor-mfe syntax that uses it are more complicated than they need to be. We’re doing something about the first in frontend-base land, but this is the first time somebody comes up with something at the Tutor level. As a co-maintainer of tutor-mfe, I’d be interested in seeing a PR that actually does it. The idea is sound in principle. If you were to submit something like that, I’d be happy to review it!

2 Likes

@arbrandes Thank you for explaining the situation in detail.

Yeah. Fair point. This has been a bit of lazy on my part. I tend to do tutor dev launch most of the time (especially when working with tutor-main), because I never know what I am going to miss. So, I just feel it’s worth waiting the 5mins it takes to get everything sorted than to debug if something doesn’t update.

Alright! Here we go → feat: implements helper classes for configuring frontend plugin slots by tecoholic · Pull Request #281 · overhangio/tutor-mfe · GitHub. This is a draft PR with of the helper class I proposed in the previous comment. I have the tutor local version of things implemen ted. Didn’t use Pydantic like LLM did, as I didn’t want to introduce a new dependency. It uses dataclasses instead, and gives everything for the DevEx I wanted. I am going to work on getting the tutor dev setup functional next. Kindly take a look.

P.S: No LLMs were used for generating the code in PR.

1 Like