Juniper Alpha JSON and YAML Configuration?

sambapete · December 26, 2019, 4:28am

In previous releases of Open edX, we were using the files lms.env.json, lms.auth.json, cms.env.json and cms.auth.json under the directory /edx/app/edxapp in order to configure the LMS and the CMS.

After recompiling the assets with our fork, I discovered some changes were not taken into effect.

It seems some of the changes I needed had to be moved into the file /edx/etc/lms.yml which seems like a mix of lms.env.json and lms.auth.json because it contains keys that are in either one file or the other.

After accidentally making an error in the /edx/etc/lms.yml file, it can confirm the values for the commands I normally use to recompile the assets were taken from /edx/etc/lms.yml.

Therefore, I have a few questions.

Are the files /edx/app/edxapp/lms.env.json and /edx/app/edxapp/lms.auth.json still used?

The values I needed to add in order to make JWT work were only put in /edx/etc/lms.yml. Are the files read in a specific order? Is any file taking precedence over the others? Thus far, I am assuming that /edx/etc/lms.yml is the file really needed by the system in Juniper while it was still /edx/app/edxapp/lms.env.json and /edx/app/edxapp/lms.auth.json for Ironwood. Am I assuming right?

Is there a way to easily transform /edx/app/edxapp/lms.env.json and /edx/app/edxapp/lms.auth.json into /edx/etc/lms.yml?

I still haven’t explored a lot yet, but I do have a copy of our fork running under juniper.alpha1 in one of our test environments.

mahyard · December 26, 2019, 1:33pm

Hi @sambapete,
You’d like to have a look at this PR
As I found those .yml files existed since ironwood.master and they will completely supersede ~edxapp/*.json files from Juniper.1 release.
if you use edx-update and server-vars.yml to store your settings then you can use it to generate your own lms.yml and cms.yml .

sambapete · December 26, 2019, 6:03pm

Thanks. I wasn’t aware of this change. I don’t believe it was documented anywhere but in this PR. Especially since the ~edxapp/*.json files are still being created during the installation. It will cause confusion… @nedbat what do you think?

aureliencroq · January 22, 2020, 1:55pm

@sambapete do you have a ready-to-use set of yml files for docker devstack to share ?
juniper devstack install fails as no LMS_CFG variable is set and of course such set of files are not yet created.

sambapete · January 22, 2020, 2:50pm

No, I do not have a ready-to-use set of yml files for docker devstack to share. I am running on native.

Under native, the LMS_CFG variable is set under /edx/app/edxapp/edxapp_env. There is no LMS_CFG variable in any of the configuration files under the /edx/etc/ directory.

If you need to run some commands as the edxapp user, you will need to source both /edx/app/edxapp/edxapp_env and /edx/app/edxapp/venvs/edxapp/bin/activate

aureliencroq · January 22, 2020, 3:31pm

Thanks, I am trying the dev stack, endedup converting *.json files in /edx/app/edxapp into yml files in /edx/etc and added export of LMS_CFG STUDIO_CFG and REVISIONS_CFG in all commands in provision-lms.sh.
It enabled migrations to work…
Let’s see if a devstack finally runs ! I will let you know.

Cory_Lee · February 10, 2020, 5:50pm

Hey guys - sorry for the trouble I will be updating the docs soon.

there is a util here you can use to do the json conversion to yaml:

github.com

edx/configuration/blob/master/util/config/merge_json_to_yaml#L1-L20


#! /usr/bin/env python


import json
import click
import yaml
import sys


@click.command()
@click.argument('files', nargs=-1, type=click.Path())
def join_json(files):
    """ This script merges multiple JSON documents into a single namespace, then dumps as YAML """
    data = dict()
    for file in files:
        click.echo('# ingested %s' % file)
        with open(file) as filehandle:
            data.update(json.load(filehandle))
    print yaml.safe_dump(data)


if __name__ == '__main__':
    join_json()

You are correct that the application intends for you to read all your config from a single yaml file instead of two separate json files in juniper, we will get the docs updated to explain it more clearly. Sorry you had to discover that on your own. Thank you for your patience.

Cory_Lee · February 10, 2020, 5:52pm

You can use it like so:

python merge_json_to_yaml --files env.json --files auth.json > edxapp.yml

Cory_Lee · September 11, 2020, 4:36pm

Update:
We have added more specific instructions here:
https://openedx.atlassian.net/wiki/spaces/AC/pages/1822916664/How+to+convert+your+LMS+and+Studio+JSON+configuration+files+to+YAML

And have merged an update to the Juniper documentation to include a link to this as well.

Topic		Replies	Views
Configuration change from Ironwood to Juniper? Site Operations Help	11	2098	January 18, 2021
JWT and Juniper? Releases juniper	11	2826	April 23, 2020
Change Database to RDS and DocumentDB Site Operators how-to , juniper	4	504	October 8, 2020
Juniper Alpha basic installation Site Operators juniper	3	1186	January 10, 2020
Setting up a subdomain for the studio cms in the juniper.2 release Site Operations Help juniper	4	2138	November 22, 2020

Juniper Alpha JSON and YAML Configuration?

Related Topics