Injecting JS into HTMLs platform wide

Site Operators
1

In order to comply with various privacy laws we need to integrate CookieBot (or a similar solution) with our platform.

This is done by injecting a js script in the head section of each html.

Theoretically an unauthenticated user on our platform in our specific configuration can only reach the AUTHN MFE successfully however we would like to inject the CookieBot script platform-wide (good practice).

Since the platform is split between MFEs and Django I was thinking about two solutions:
1)

  • Modify the frontend-component-header package and inject it there, then make sure each MFE uses the frontend-component-header (we already have custom MFEs so forking isn’t a problem)
  • For the Django part, write a custom Django App Plugin that would add new middleware injecting the required part

We are already using a custom Caddy file (like fully) (in order to mitigate this max file size limit issue patchability of UPLOAD_FILE_MAX_SIZE · Issue #1661 · openedx/frontend-app-authoring · GitHub ) so why not use a custom caddy image (IIRC tutor allows this via a simple patch to the jinja2 template).

I found this https://github.com/caddyserver/replace-response and I guess the solution would be as simple as replacing the head part of the response with the script injected there.

—

Is there a better, maybe more official way on doing this that I’m missing? Option 2) seems like a hack however it is much less of a pain to implement and maintain (at least superficially).

We are using tutor==20.0.1 and therefore the release/teak.2 tag of the platform

Thanks for any help!

2

@arbrandes any advice?

3

Just wanted to say option two works as intended however still curious about other possibilities!

4

This is actually the exact same problem I had in this issue:

I ended up editing the various `frontend-app`'s `public/index.html` files:

and similar for all other ones. This is rather unfortunate and tedious.

5

Hey thanks for sharing your experience, If you or anyone else would be interested in switching to a fully custom Caddyfile I can write the process out here as it’s also not that straightforward and I don’t get it why Caddyfile is not more flexible via patches or doesn’t just have the possibility to be mounted from a different path than the one hardcoded in tutor.

I am aware creating new custom templates is doable however overriding existing templates with new ones that have to be generated to the same exact path in the environment is tough.

Unless of course I am overseeing something.

6

I would be interested to learn more about the Caddyfile option, as we already modify the caddy file with a patch:

Maybe it is possible to inject your change with the same kind of hook ?

7

I initially tried the patches listed in

however they were not enough to remove or overwrite the hardcoded 250 MB limit for the authoring MFE.

so that initially made me use a custom Caddyfile setup.

At first I just made a plugin using python’s atexit. Just registered a function that would be launched as the last thing in tutor config save. It would just overwrite the Caddyfile by path however after tinkering a bit more I came up with a cleaner solution. Still a bit complicated though.

I keep all my .py plugins in one repo so to port them between instances I have a Makefile that has make commands to install them all like:

setup-plugins: install-plugins enable-plugins
	tutor plugins disable indigo
	@echo "All plugins installed and enabled"

where install-plugins and enable-plugins is just repeating

tutor plugins install file.py
tutor plugins enable file.py

for each file in the catalog, with install-plugins looking like this for example

install-plugins:
	tutor plugins install mfe
	@for plugin in $$(ls *.py 2>/dev/null); do \
		if [ -f "$$plugin" ]; then \
			tutor plugins install $$plugin; \
		fi \
	done

This makefile will be important later.

I generate my own Caddyfile from a custom template.
To add a custom template I hijack the original list of template roots using the ENV_TEMPLATE_ROOTS Filter and PREPEND mine to the list instead of appending.

I didn’t bother looking into the source code to find out exactly how jinja2 treats multiple roots however when appending, the template from the original template root was winning and creating the Caddyfile in the env. By prepending, this custom one is read first and treated as final.

@hooks.Filters.ENV_TEMPLATE_ROOTS.add()
def hijack_template_roots(roots):
    # every other hook failed, ignore template patterns works for both
    # custom roots and original root, every template path is relative to its root
    # so if we want to have the caddyfile in same directory in env 
    # (in $(cd tutor config printroot)/env/apps/caddy/Caddyfile where its mounted from)
    # the paths checked to be ignored (even tho in different template roots)
    # are the SAME
    # we have no other way than to prepend before the origianl root template
    # so caddyfile gets read from ours and not processed from tutors 
    # (jinja keeps track and does not overwrite same file from last template)

    # still this is cleaner than overwriting the Caddyfile using python atexit

    if SETUP_REPO_PATH:
        new_templates_path = os.path.join(SETUP_REPO_PATH, "templates")
        
        return [new_templates_path] + roots
        
    return roots

now SETUP_REPO_PATH is what I’m able to get dynamically thanks to that Makefile. It’s just the original path to which the custom tutor plugins where cloned on the host. This makes the setup path agnostic.

I just add a new make command and integrate it with the main setup command like

export SETUP_REPO_PATH := $(shell pwd)

persist-path:	
	tutor config save --set SETUP_REPO_PATH=$(SETUP_REPO_PATH)

install-plugins: persist-path
	tutor plugins install mfe
        .... (as earlier) ...
	done

So to access it in the plugin I retrieve it from tutor’s config

tutor_config = get_tutor_config()
SETUP_REPO_PATH = tutor_config.get("SETUP_REPO_PATH") # our own var

Where the helpers are

import config from tutor as tutor_config

def get_tutor_root():
    return os.environ.get("TUTOR_ROOT", os.path.expanduser("~/.local/share/tutor"))

def get_tutor_config():
    root_path = get_tutor_root()
    return tutor_config.load(root_path)

All these gymnastics because there isn’t a hook in tutor (or at least I couldn’t locate one - Hooks catalog — Tutor documentation) with which I can extract the location of the plugin when it is first installed - before it’s moved to the tutor-plugins catalog.

Also there is no hook that can be run AFTER rendering the files from templates which would allow us to overwrite the Caddyfile or any other env file after generation in a tutor-friendly way.

So with the new root successfully added I just create my template at a path

/templates/apps/caddy/Caddyfile

relatively to the plugin. This way I keep everything condensely in a setup catalog.
With this approach you can not only overwrite the official Caddy with your own but with your own template!

This makes it easier to navigate between instances or dev/prod/staging if you have different values in caddy. What I do whenever that is the case is just declare a new CONFIG_DEFAULT

PROD_VAL = "foo"
DEV_VAL = "bar"
mode = get_image_mode(sys.argv)

VAR_VAL = PROD_VAL if mode == "local" else DEV_VAL
hooks.Filters.CONFIG_DEFAULTS.add_item(
    ("NEW_VAR",VAR_VAL)
)

# where get_image_mode is another simple helper 
def get_image_mode(args):
    if "dev" in args:
        return "dev"
    elif "local" in args:
        return "local"

I do this since at one point I had to use caddy in dev mode to test sth (that means having another patch in local-docker-compose-dev-services but I suppose that is not worth mentioning.

Also don’t forget to set the Caddy to your custom image!

hooks.Filters.CONFIG_DEFAULTS.add_item(
    ("DOCKER_IMAGE_CADDY", "your_container_registry/username/caddy-replace:ver")
)

I’m very curious of your opinion! Let me know what you think

8

Hum, I was more interested into the content of the caddy file itself. I am more looking at injecting the JS, the 250Mb limit is fine for me, and I don’t want to overwrite the whole caddy file. Just injecting a rule seems fine and doable with the patches.

9

Well then this is the cookie part in the Caddyfile. I declare it first with a regex that replaces head but only after the initial <html tag since some parts of js scripts (like in the account MFE) have the head string which also gets replaced and breaks the page if regex just set to replace head.

(inject_head) {
    replace {
        match {
            header Content-Type text/html*
        }
        re `(<html.*?>\s*<head>)` `$1<script id="Cookiebot" src="https://consent.cookiebot.com/uc.js" data-cbid="{{ COOKIEBOT_ID }}" data-blockingmode="auto" type="text/javascript"></script>`
    }
}

then import it in under different URLs like so:

# MFE
http://{{ MFE_HOST }} {
    request_body {
        max_size 2MB
    }
    import proxy mfe:8002 {{ MFE_HOST }}
    import inject_head

Maybe some combination of the available patches will work for you.

Okay, I checked and I’m pretty sure you can! Declare it using {{ patch(“caddyfile”) }} at the end of file tutor/tutor/templates/apps/caddy/Caddyfile at release · overhangio/tutor · GitHub

then use this “new” 8 month old “caddyfile-mfe-proxy” patch here :smiley:

That is to inject the js into all MFEs at least.

10

Is the replace command installed by default in the caddy container ? Because that’s not standard caddy feature: https://caddyserver.com/docs/modules/http.handlers.replace_response

11

Right! Sorry, I completely forgot to mention, you need to build the caddy image with this mod

More on building here GitHub - caddyserver/xcaddy: Build Caddy with plugins · GitHub

But as I moved around the platform more, I’ve found that the replace command I used breaks some functionality silently. All xBlocks for example are not showing up from the studio perspective.

Still, maybe with a better regex or injecting this in a more robust way it’d be a good approach.

12

Since y’all are willing to spend some effort here, let me give my take on what The Right Way :trade_mark: would be to do this:

Step 1. Allow MFEs to insert arbitrary scripts

frontend-platform already has a script loader:

    loadExternalScripts(externalScripts, {
      config: getConfig(),
    });

Currently, it’s only used to load Google Analytics:

  externalScripts = [GoogleAnalyticsLoader],

The thing is that externalScripts is hard-coded. All we would need to do is allow that variable to be appended to by env.config.js, and presto, the hard part would be done. For instance:

const externalScriptsImpl = getConfig().externalScripts || externalScripts;

(Yes, it might be a one-line change to frontend-platform.)

Step 2. Add support to tutor-mfe

We’d need to add explicit externalScript support to the env.config.jsx template, which would actually be the hardest part (but not rocket science). We’d just need to follow the plugin slot pattern, but for external scripts, so that one plugin’s external scripts don’t clobber another’s.

Anybody up for creating PRs? I maintain both packages, so I’d be glad to review/merge!

13

This indeed sounds like a good solution.

I unfortunately won’t have time to get to this before June, but I’ve set myself a reminder to get back to this when I have more time. That is if @jakubkl didn’t get to it first.

14

Hey @arbrandes, thanks for the directions. I submitted two new PRs. Let me know what you think and if this is more or less what you had in mind.

frontend-platform - feat: support appending externalScripts via env.config by sagrodat · Pull Request #878 · openedx/frontend-platform · GitHub

tutor-mfe - feat: add INJECTED_SCRIPTS hook for MFE script injection by sagrodat · Pull Request #289 · overhangio/tutor-mfe · GitHub

Gotta say, it’s my first contribution so lmk if I missed some guidelines or standards, I’m open for feedback :slight_smile:

Of course if this gets accepted I am down to document it on appropriate READMEs and docs where needed.

15

Looks like we’re going to be co-authors! I was working on this at the same time you were, apparently! :laughing:

Not sure what the best way to reconcile the two is going to be, but given mine are a tad more comprehensive in the code (tests, docs), yours more comprehensive in the test instructions, how about we start with mine as a basis? Care to review and test them? Feel free to shoot holes in anything you see that doesn’t look good.

(Out of curiosity, I also just put out an implementation of this in frontend-base land.)

16

Definitely didn’t expect this :laughing:. Of course if you went the trouble to implement it, keep your way considering your experience in the project and the fact it’s much more functional. I just went with the minimal but working approach.

Plus you made it fully customizable by removing the hard coded Google analytics (even tho they were off by default), and yeah I guess it’s better to declare the class or install it using a patch instead of carrying it as a 3rd string payload.

Thanks for doing this so quickly! Not sure I will have time to test until Monday but if it’s not merged by then I’ll check it out.

17

First PR is merged! That means the mechanism is now in-place. You can see an example of how it’ll work as a Tutor plugin in the proposed README:

    from tutormfe.hooks import EXTERNAL_SCRIPTS
    from tutor import hooks

    hooks.Filters.ENV_PATCHES.add_item(
        (
            "mfe-env-config-buildtime-definitions",
            """
    class CookieYesLoader {
      constructor() {}

      loadScript() {
        const script = document.createElement('script');
        script.id = 'cookieyes';
        script.src = 'https://cdn-cookieyes.com/client_data/YOUR_SITE_ID/script.js';
        document.head.appendChild(script);
      }
    }
    """,
        )
    )

    EXTERNAL_SCRIPTS.add_items([
        (
            "all",
            "CookieYesLoader",
        ),
    ])

There’s more to do beyond just these two PRs, though. In particular, we need to upgrade the frontend-platform version in all the MFEs that are going into Verawood, otherwise this won’t work across the board.

18

One question. Will this be ported to Ulmo, or only Verawood ?

19

Only Verawood, probably. We only backport new features to current (or old) releases under rare, case-by-case circumstances.

20

Alright, this feature is now fully available in tutor@main! This means you can try it out now, and also that it will come out with Verawood in a couple of months.