Deploying MFEs in the community

nimisha · September 1, 2020, 1:00pm

@Felipe and team: thanks for putting the doc together.

Regarding having multiple sub-domains, note that this is a temporary situation for what edx.org has implemented for itself. The long-term plan is to have a single domain for all MFEs and use URL paths to route to different MFEs.

So as the community designs a solution for MFE deployment for Open edX, you’ll be ahead of the curve (and edX.org can follow) if you jump to a single domain design.

nedbat · September 1, 2020, 5:25pm

@fredsmith started an OEP about routing, I’m not sure how thinking has shifted since then: https://github.com/edx/open-edx-proposals/pull/111/files

fredsmith · September 1, 2020, 6:07pm

So, we have a couple in progress but not adopted at edx.org solutions for routing MFEs. The most polished one is conductor, which is implemented in ansible: https://github.com/edx/configuration/blob/master/playbooks/conductor.yml

This playbook deploys an nginx server that routes paths to MFEs hosted in s3 buckets. A sample configuration is:

################################################################
#
#  Conductor Settings
#
CONDUCTOR_STATIC_SITES:
  - proxied_path: http://account.stage.edx.org.s3.amazonaws.com/
    router_path: account
  - proxied_path: http://gradebook.stage.edx.org.s3.amazonaws.com/
    router_path: gradebook
  - proxied_path: http://orders.stage.edx.org.s3.amazonaws.com/
    router_path: orders
  - proxied_path: http://payment.stage.edx.org.s3.amazonaws.com/
    router_path: payment
  - proxied_path: http://portal.stage.edx.org.s3.amazonaws.com/
    router_path: portal
  - proxied_path: http://profile.stage.edx.org.s3.amazonaws.com/
    router_path: profile
  - proxied_path: http://program-manager.stage.edx.org.s3.amazonaws.com/
    router_path: program-manager
  - proxied_path: http://publisher.stage.edx.org.s3.amazonaws.com/
    router_path: publisher
CONDUCTOR_REDIRECT_ROOT: true
CONDUCTOR_ROOT_REDIRECT_PATH: "/portal/"

Then running the conductor playbook (linked above) will route all your MFEs from whatever box you run it on.

The key here is that our MFEs use path based routing within their routes, so this nginx role handles some path based magic to serve the single page app:

 {% for static_site in CONDUCTOR_STATIC_SITES %}

    # Matches: <router>/<organization>/
    location = /{{ static_site.router_path }}/ {
        proxy_pass {{ static_site.proxied_path }}/index.html;
    }

    # Matches: <router>/<organization>/[.../]<file.ext>
    location ~ ^/{{ static_site.router_path }}/((?:\w+\/+)*)([\w\-\.]+\.[\w\-\.]+) {
        proxy_pass {{ static_site.proxied_path }}/$1$2;
    }

    # Matches: <router>/<organization>/<subpage>/[.../]
    location ~ ^/{{ static_site.router_path }}/([a-z0-9-]+)[/]? {
        proxy_pass {{ static_site.proxied_path }}/$1/index.html;
    }

This isn’t a perfect solution, and our long-term goal is to use k8s and ingress to handle all of this, so we’ve kinda not made a decision on it. I would love to work with the community to find out what people need and how we can make this work across the board.

regis · September 1, 2020, 6:23pm

Doesn’t this defeat the purpose of S3, in terms of performance and CDN capabilities?

fredsmith · September 1, 2020, 9:36pm

we use S3 because of resiliency and ease of deployment. Performance isn’t a motivation. We put a CDN (cloudflare) in front of our nginx servers to bring MFEs closer to the edge.

regis · September 5, 2020, 11:34am

Thanks to the help of @morenol I have managed to create a working plugin for Tutor that deploys the Gradebook microfrontend application: https://github.com/overhangio/tutor-gradebook As all Tutor plugins, it’s really easy to install., so I encourage you to try it out (not in production, though :))

A demo is available here: https://grades.demo.openedx.overhang.io (login: admin@overhang.io, password: admin)

Creating this plugin has allowed me to get a first experience in MFE deployment. Generally speaking, it was fairly easy to create, apart from the usual head-scratching issues like CORS. Yet, I’d like to mention a few points where MFEs could be improved:

Documentation: currently, the list of environment variables that need to be set for building the gradebook is spread in multiple files: src/config/index.js and src/index.jsx. I would like to see the list of all required environment variables in a single place, where they would be properly documented.
Development: in the open-release/juniper.3 version, the webpack-dev server is run with host checking, such that it can only be accessed as 0.0.0.0, localhost or 127.0.0.1: https://github.com/edx/frontend-app-gradebook/blob/open-release/juniper.3/config/webpack.dev.config.js#L140 This is quite inconvenient, as in Tutor we access the gradebook at http://grades.local.overhang.io:1994. It would be great if the host that the dev server binds to were defined as a (documented) environment variable. (It is unclear to me if issue still happens in the master branch, as I am unfamiliar with the fedx-scripts executable: https://github.com/edx/frontend-app-gradebook/blob/aa39fcc7e045f231d93c7c8ae180e73cd0d16a69/package.json#L17)
Production: as discussed here, the web server needs to redirect the /<course-id> paths to /index.html. This seems like an anti-pattern, as it prevents having well-defined 404 error. Also, it forces operators to add specific intelligence to the web server, when one of the promises of microfrontends was that we only needed to setup a dumb fileserver. Instead of adding the course ID in the path (/<course-id>), we should be using querystrings (/?course-id=...).

These are fairly minor technical hurdles that I’m sure will be easily resolved. Again, deploying the MFE was a rather pleasant experience. I don’t expect that integrating other MFEs will present much more challenging difficulties.

jalondonot · September 24, 2020, 9:37pm

Hi everyone!

We have made some progress at eduNEXT, thanks to @morenol!

His PR creates three roles to deploy the profile, grade book and account MFEs. It is possible to test this on a staging environment he set for this purpose, feel free to check it out, test it, and comment directly on the PR or on this thread:

User: staff@example.com
Pass: edx

User: honor@example.com
Pass: edx

As Luis mentioned on the PR, we are keen to receive the community feedback on this, and hopefully, clarify and stay aligned on the objective we want to achieve by the end of October in regard to the MFE deployments. Considering this, it could be useful to have a call to discuss this further and hopefully, as a result of the meeting, we will get a clear strategy and goal on this matter. I created a Doodle to reach a consensus about the day. The options I proposed are either Monday, Tuesday or Wednesday next week, 10-11 A.M, or 11-12 Eastern Time (ET), so please let me know which time works best for you all.

jalondonot · September 25, 2020, 10:36pm

Hi!
Based on the Doodle results, I’ve scheduled the meeting Monday 10-11 A.M. Eastern Time (ET). Hope that fits everyone’s agenda. Just in case someone could miss the poll, here is the link of the meeting. Come join us!

nedbat · September 25, 2020, 11:43pm

Next time, include the time in the Doodle poll, please…

jalondonot · September 28, 2020, 11:20pm

Thank you for joining the meeting, it was great discussion and collaboration. Here is the recording from this morning:

arbrandes · September 29, 2020, 5:20pm

Same here.

jalondonot · October 9, 2020, 3:44am

Sorry! I will

morenol · October 30, 2020, 12:41pm

Hi, updates on this:

frontend-build and frontend-platform where updated to support the deployment in subdirectories, you can check these PRs for reference:

There is a PR that fixes a problem with branding override in the role that we already created for MFE deployment:
https://github.com/edx/configuration/pull/6049
I created PRs against gradebook, profile and account MFEs with the latest version of frontend-build and frontend-platform (we should do that with all the MFEs that we plan to use in Koa).

There is a PR in configuration that allows the deployment of MFEs in subdirectories (I tested that using the versions of the MFEs in the PRs above), and it can be checked in:

with the credentials:
honor@example.com
edx

regis · October 30, 2020, 4:59pm

Short and sweet! I love these PRs

nedbat · October 30, 2020, 7:55pm

Just a reminder that we really want everything merged by Nov 9th if we can. Do we know of work that won’t make it by then?

Felipe · November 5, 2020, 10:00pm

Launching Native Install with MFEs

This is a happy little set of instruction to get a native install of open edX working with the new MFE deployments.
It will download the master branches of most repos and a few selected branches that are being considered to be merged before the Koa release.

It should take about 2~3 hours to do all this, but most time is unattended ansible work.

Getting started

Shopping list:

web server with ubuntu 16
public wildcard domain
SSL cert

I started this on a brand new t2.large on AWS Using the baseline ubuntu 16.
Storage disk should be at least 25GB. Go for 30 to be safe.
This is a public facing web server with ports 80 and 443 open to the world.
Public IP: 52.12.48.91

I created a DNS A record for this IP to the domain wildcard:

.mfe-example.publictest.edunext.co

For the SSL cert I used certbot:

    sudo apt udpate
    sudo apt upgrade
    sudo apt install letsencrypt
    sudo certbot certonly --standalone

Note: certbot failed due to deprecated python2 libraries. To fix it I did:

    pip install requests urllib3 --force --upgrade

I used the cli wizard to create a certificate for domains (a wildcard would work just as well):

    lms.mfe-example.publictest.edunext.co
    studio.mfe-example.publictest.edunext.co
    react-apps.mfe-example.publictest.edunext.co

The native install part:

Following the instructions on the draft PR (https://github.com/edx/configuration/pull/6128).

    mkdir mfe_example
    cd mfe_example
    export OPENEDX_RELEASE=nedbat/tk.1106c
    wget https://raw.githubusercontent.com/edx/configuration/$OPENEDX_RELEASE/util/install/ansible-bootstrap.sh -O - | sudo -E bash
    wget https://raw.githubusercontent.com/edx/configuration/$OPENEDX_RELEASE/util/install/generate-passwords.sh -O - | bash

Note: edited on 09.11.2020 to use the nedbat/tk.1106c which fixes the elasticsearch problems and it is similar to what koa will be.

nano config.yml

# config.yml

################################
# general settings for ansible #
################################

module_lang: en_US.UTF-8
module_set_locale: True

########################
# edxapp role settings #
########################

EDXAPP_LMS_BASE_SCHEME: https

EDXAPP_LMS_BASE: "lms.mfe-example.publictest.edunext.co"
EDXAPP_CMS_BASE: "studio.mfe-example.publictest.edunext.co"

EDXAPP_SESSION_COOKIE_DOMAIN: ".mfe-example.publictest.edunext.co"

EDXAPP_CSRF_COOKIE_SECURE: true
EDXAPP_SESSION_COOKIE_SECURE: true
EDXAPP_CROSS_DOMAIN_CSRF_COOKIE_DOMAIN: ".mfe-example.publictest.edunext.co"
EDXAPP_CROSS_DOMAIN_CSRF_COOKIE_NAME: "cross-domain-cookie-mfe"

EDXAPP_CORS_ORIGIN_WHITELIST:
- studio.mfe-example.publictest.edunext.co
- react-apps.mfe-example.publictest.edunext.co

EDXAPP_CSRF_TRUSTED_ORIGINS:
- react-apps.mfe-example.publictest.edunext.co

EDXAPP_LOGIN_REDIRECT_WHITELIST:
- studio.mfe-example.publictest.edunext.co
- react-apps.mfe-example.publictest.edunext.co

# Needed to link the LMS instructor dashboard to the writable gradebook micro-frontend
EDXAPP_LMS_WRITABLE_GRADEBOOK_URL: https://react-apps.mfe-example.publictest.edunext.co/gradebook

# Needed to link to the new profile micro-frontend.
EDXAPP_PROFILE_MICROFRONTEND_URL: https://react-apps.mfe-example.publictest.edunext.co/profile/u/

# Needed to link to the new account micro-frontend.
EDXAPP_ACCOUNT_MICROFRONTEND_URL: https://react-apps.mfe-example.publictest.edunext.co/account

EDXAPP_LMS_SSL_NGINX_PORT: 443
EDXAPP_CMS_SSL_NGINX_PORT: 443

EDXAPP_ENABLE_CORS_HEADERS: true
EDXAPP_ENABLE_CROSS_DOMAIN_CSRF_COOKIE: true

#######################
# nginx role settings #
#######################

NGINX_ENABLE_SSL: yes

NGINX_SSL_CERTIFICATE: '/etc/letsencrypt/live/lms.mfe-example.publictest.edunext.co/fullchain.pem'
NGINX_SSL_KEY: '/etc/letsencrypt/live/lms.mfe-example.publictest.edunext.co/privkey.pem'

#####################
# mfe role settings #
#####################

MFE_BASE_SCHEMA: https
MFE_DEPLOY_COMMON_HOSTNAME: "react-apps.mfe-example.publictest.edunext.co"

# This will override the logo in the header with a dummy logo
MFE_NPM_OVERRIDES: ["@edx/frontend-component-header@git+https://github.com/edunext/frontend-component-header-edunext.git"]

# This 2 variables make the whole process select branches of the MFEs repos where the changes are applied:
# Will not be necessary once the following PRs have been merged:
# https://github.com/edx/frontend-app-gradebook/pull/145
# https://github.com/edx/frontend-app-account/pull/340
# https://github.com/edx/frontend-app-profile/pull/386
# A similar PR will be necessary for all mfes
MFE_DEPLOY_GIT_PATH: 'edunext'
MFE_DEPLOY_VERSION: 'lmm/build'

And now we launch the full deployment:

wget https://raw.githubusercontent.com/edunext/configuration/mfe/community_deployment/util/install/native.sh -O - | bash

Note 1:
This native.sh comes from the mfe/community_deployment branch. What this does differently is that it clones the mfe/community_deployment branch before doing the ansible install. This would not be necessary once the PRs 6088 and 6049 are merged in master.

Note 2:
Running this failed once due to some deprecated SSL library. Perhaps I broke it when I did the urllib3 upgrade.
Fixed it with:

    sudo apt remove python-openssl

Note 3:
Installing the forum role did not work because of a known error in forum: initialize elasticsearch
What we did was to comment out the forum role in the mfe/community_deployment. This should not be a problem for anyone following this instructions after we already commented it. Just note that the forum will not work in this installation. Using branches named nedbat/tk.1106c fixes the forum

Configurations for edxapp:

The site is live at: https://lms.mfe-example.publictest.edunext.co/

We now need a super user:

sudo su edxapp -s /bin/bash
cd edx-platform/
source ../venvs/edxapp/bin/activate
source ../edxapp_env
python manage.py lms createsuperuser

With this, we go to https://lms.mfe-example.publictest.edunext.co/admin, log in as a super user and turn on the necessary settings.

3.1: Site configurations:
-https://lms.mfe-example.publictest.edunext.co/admin/site_configuration/siteconfiguration/add/

I edited the example.com site to match lms.mfe-example.publictest.edunext.co

Site values:

{
"ENABLE_PROFILE_MICROFRONTEND":true,
"ENABLE_ACCOUNT_MICROFRONTEND":true
}

3.2: Waffle flags:
-https://lms.mfe-example.publictest.edunext.co/admin/waffle/flag/

Here you need to add the 2 following flags for “everyone”

account.redirect_to_microfrontend
learner_profile.redirect_to_microfrontend

This is it. Now you can visit:

-https://lms.mfe-example.publictest.edunext.co/

And log in with any of the usual demo users. Honor, Verified, Staff …

All the MFEs will be deployed in the react-apps.mfe-example.publictest.edunext.co domain.
For example:

Tagging the people in the call today for more visibility: @nedbat, @sambapete, @nimisha, @antoviaque, @pdpinch, @regis. And @djoy for good meassure.

A million thanks to @morenol for all the hardwork.

nedbat · November 5, 2020, 11:41pm

@Felipe and @morenol, thanks for putting all this together. Is it possible to make this similar in complexity to the existing installation instructions (https://openedx.atlassian.net/wiki/spaces/OpenOPS/pages/1969455764/Koa+Native+Open+edX+platform+Ubuntu+20.04+64+bit+Installation)?

We want to make it as simple as possible for people to install this software. How can we smooth these steps?

Felipe · November 6, 2020, 3:27pm

To make this more simple the answer is to merge the PRs. To merge the PRs, we need people to test it.
To test it, they need to go an extra mile to be able to deploy this before the PRs are merged. Its a catch 22.

One thing to notice is that although MFEs are the shiny new thing people want to see in the browser, they are still optional and are sitting behind 3 different feature flags. Not counting the 3 or more specific settings for CORS configuration and security. Smoothing this settings will require that some more experienced people test it first. The instructions I wrote above are meant for people who is already familiar with the project and is willing to test the feature to make it smoother in the future.

xitij2000 · November 9, 2020, 8:26pm

We’re testing this with our automated deployment system (OCIM) as well, and have two tests running:

Subdomain-based deployment: https://mfe-test.sandbox.opencraft.hosting/
Subpath-based deployment: https://second-mfe-test.opencraft.hosting/

Both have standard users configured.

morenol · November 9, 2020, 9:30pm

I just tried again without https and it worked, it is not mandatory to have https for the MFEs

What I did was similar to the usual installation guide for the community.
These where the steps that I ran:

export OPENEDX_RELEASE=nedbat/tk.1106c
wget https://raw.githubusercontent.com/edx/configuration/$OPENEDX_RELEASE/util/install/ansible-bootstrap.sh -O - | sudo -E bash
wget https://raw.githubusercontent.com/edx/configuration/$OPENEDX_RELEASE/util/install/generate-passwords.sh -O - | bash

Then I created a config.yml with this, probably we need to update the role and the native playbook to include some of these configurations by default

########################
# edxapp role settings #
########################

EDXAPP_LMS_BASE: "54.189.215.67"
EDXAPP_CMS_BASE: "54.189.215.67:18010"

EDXAPP_CROSS_DOMAIN_CSRF_COOKIE_DOMAIN: "54.189.215.67"
EDXAPP_CROSS_DOMAIN_CSRF_COOKIE_NAME: "cross-domain-cookie-mfe"

EDXAPP_CORS_ORIGIN_WHITELIST:
- 54.189.215.67:18010
- 54.189.215.67:19010

EDXAPP_CSRF_TRUSTED_ORIGINS:
- 54.189.215.67:19010


EDXAPP_LOGIN_REDIRECT_WHITELIST:
- 54.189.215.67:18010
- 54.189.215.67:19010

# Needed to link the LMS instructor dashboard to the writable gradebook micro-frontend
EDXAPP_LMS_WRITABLE_GRADEBOOK_URL: http://54.189.215.67:19010/gradebook

# Needed to link to the new profile micro-frontend.
EDXAPP_PROFILE_MICROFRONTEND_URL: http://54.189.215.67:19010/profile/u/

# Needed to link to the new account micro-frontend.
EDXAPP_ACCOUNT_MICROFRONTEND_URL: http://54.189.215.67:19010/account

EDXAPP_ENABLE_CORS_HEADERS: true
EDXAPP_ENABLE_CROSS_DOMAIN_CSRF_COOKIE: true

#####################
# mfe role settings #
#####################

MFE_DEPLOY_NGINX_PORT: 19010

# This will override the logo in the header with a dummy logo
MFE_NPM_OVERRIDES: ["@edx/frontend-component-header@git+https://github.com/edunext/frontend-component-header-edunext.git"]

# This 2 variables make the whole process select branches of the MFEs repos where the changes are applied:
# Will not be necessary once the following PRs have been merged:
# https://github.com/edx/frontend-app-gradebook/pull/145
# https://github.com/edx/frontend-app-account/pull/340
# https://github.com/edx/frontend-app-profile/pull/386
# A similar PR will be necessary for all mfes
MFE_DEPLOY_GIT_PATH: 'edunext'
MFE_DEPLOY_VERSION: 'lmm/build'

and then triggered the deployment with the custom configuration branch:

wget https://raw.githubusercontent.com/edunext/configuration/mfe/community_deployment/util/install/native.sh -O - | bash

Any of you can check that by going to

http://54.189.215.67:19010/profile/u/dummy
http://54.189.215.67:19010/account

and login with this credentials:
dummy@example.com

I put the MFE by default in the port 19010, but I have not changed the default value in the role, which is the value that we should use?

Other than that I will update the PR to enable
EDXAPP_ENABLE_CORS_HEADERS, EDXAPP_ENABLE_CROSS_DOMAIN_CSRF_COOKIE,
EDXAPP_ENABLE_CORS_HEADERS and EDXAPP_ENABLE_CROSS_DOMAIN_CSRF_COOKIE by default in the native installation.

I run that in ubuntu20.04.

Note: I had a problem with the CELERY_BROKER_PASSWORD, but I think that it was because I generated the my-passwords.yml file with OPENEDX_RELEASE=nedbat/tk.1106b instead of OPENEDX_RELEASE=nedbat/tk.1106c that has this important change:

Maybe you can use these steps to test the installation:

@BbrSofiane @regis @sambapete and @nedbat

As you can see, it is using a custom branding (the logo was overrided with this one https://github.com/eduNEXT/frontend-component-header-edunext/commit/656050af3a2437429071c81dd70987b43b5d2b20)

On the other hand, I have not added the waffle flags and the site configuration changes (they are only needed to enable the redirection to the MFE by default)