Docker Devstack – Multiple Releases – One Machine

Development
1

Has anyone figured out how to setup multiple Docker devstacks for Open edX platform on one machine yet?

I created a thread on Slack #devstack, however, there doesn’t seem to be a lot of solutions for this yet.
https://openedx.slack.com/archives/CDAG4KN2C/p1587399705007100

Any working solutions would help.

I’m currently working on updating all Docker containers to have release. prefix. Also probably need to update the database names for MySQL and MongoDB.

Example edx.devstack.lms to hawthorn.edx.devstack.lms for example.

Whenever I restart services for the LMS after updating MySQL to hawthorn.edx.devstack.mysql it gives me the following warning.

Unknown MySQL server host ‘edx.devstack.mysql’ (0)

Performing system checks...
Unhandled exception in thread started by <function wrapper at 0x7f8e5c760938>
Traceback (most recent call last):
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/utils/autoreload.py", line 228, in wrapper
    fn(*args, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 124, in inner_run
    self.check(display_num_errors=True)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/base.py", line 359, in check
    include_deployment_checks=include_deployment_checks,
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/base.py", line 346, in _run_checks
    return checks.run_checks(**kwargs)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/checks/registry.py", line 81, in run_checks
    new_errors = check(app_configs=app_configs)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/checks/model_checks.py", line 30, in check_all_models
    errors.extend(model.check(**kwargs))
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/db/models/base.py", line 1284, in check
    errors.extend(cls._check_fields(**kwargs))
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/db/models/base.py", line 1359, in _check_fields
    errors.extend(field.check(**kwargs))
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py", line 913, in check
    errors = super(AutoField, self).check(**kwargs)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py", line 219, in check
    errors.extend(self._check_backend_specific_checks(**kwargs))
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py", line 322, in _check_backend_specific_checks
    return connections[db].validation.check_field(self, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/db/backends/mysql/validation.py", line 49, in check_field
    field_type = field.db_type(self.connection)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py", line 644, in db_type
    return connection.data_types[self.get_internal_type()] % data
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/utils/functional.py", line 35, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 174, in data_types
    if self.features.supports_microsecond_precision:
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/utils/functional.py", line 35, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/db/backends/mysql/features.py", line 53, in supports_microsecond_precision
    return self.connection.mysql_version >= (5, 6, 4) and Database.version_info >= (1, 2, 5)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/utils/functional.py", line 35, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 385, in mysql_version
    with self.temporary_connection() as cursor:
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 591, in temporary_connection
    cursor = self.cursor()
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 254, in cursor
    return self._cursor()
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 229, in _cursor
    self.ensure_connection()
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
    self.connect()
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
    self.connect()
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 189, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 274, in get_new_connection
    conn = Database.connect(**conn_params)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/MySQLdb/__init__.py", line 81, in Connect
    return Connection(*args, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/MySQLdb/connections.py", line 193, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
django.db.utils.OperationalError: (2005, "Unknown MySQL server host 'edx.devstack.mysql' (0)")

When issuing an LMS Shell command it notice that the edx-platform code has reference to the new MySQL database container hawthorn.edx.devstack.mysql.

root@ad4991af17d4:/edx/app/edxapp/edx-platform# grep -ir "edx.devstack.mysql" ./*
./cms/envs/bok_choy_docker.auth.json:            "HOST": "hawthorn.edx.devstack.mysql",
./cms/envs/bok_choy_docker.auth.json:            "HOST": "hawthorn.edx.devstack.mysql",
./lms/envs/bok_choy_docker.auth.json:            "HOST": "hawthorn.edx.devstack.mysql",
./lms/envs/bok_choy_docker.auth.json:            "HOST": "hawthorn.edx.devstack.mysql",
./scripts/reset-test-db.sh:    MYSQL_HOST="--host=hawthorn.edx.devstack.mysql"
2

I was able to get the edx.devstack.mysql not found error message removed, within the LMS container, by doing the following.

Under the .\devstack\docker-compose.yml file I updated the MySQL and MongoDB services to include the following default network aliases.

services:
  mongo:
    container_name: ${OPENEDX_RELEASE}.edx.devstack.mongo
    networks:
      default:
        aliases:
          - edx.devstack.mongo

  mysql:
    container_name: ${OPENEDX_RELEASE}.edx.devstack.mysql
    networks:
      default:
        aliases:
          - edx.devstack.mysql

We could have used a separate network devstack_default in the .\devstack\docker-compose.yml file by issuing the following, however, each service would also need to specify that it’s using that network as I have indicated in the LMS. I chose to not do this at the moment and just use the default network.

networks:
  devstack_default:
    driver: bridge

services:
  mongo:
    container_name: ${OPENEDX_RELEASE}.edx.devstack.mongo
    networks:
      devstack_default:
        aliases:
          - edx.devstack.mongo

  mysql:
    container_name: ${OPENEDX_RELEASE}.edx.devstack.mysql
    networks:
      devstack_default:
        aliases:
          - edx.devstack.mysql

  lms:
    container_name: ${OPENEDX_RELEASE}.edx.devstack.lms
    networks:
      - devstack_default

Naming each container with prefix hawthorn. helps isolate the containers per release allowing me to work on separate releases on the same machine.

I applied the network alias to the original service container name for all other services defined within .\devstack\docker-compose.yml file just to be safe.

  • chrome
  • elasticsearch
  • firefox
  • memcached
  • credentials
  • discovery
  • ecommerce
  • edx_notes_api
  • studio
  • forum
  • devpi

For the new container to be recognized on a docker command call we need to also update the .\devstack\Makefile to have this prefix as well. This is just one of several commands that we could find /replace edx.devstack with ${OPENEDX_RELEASE}. prefix.

lms-shell: ## Run a shell on the LMS container
	docker exec -it ${OPENEDX_RELEASE}.edx.devstack.lms env TERM=$(TERM) /edx/app/edxapp/devstack.sh open

I’m using the Shell environment variable $OPENEDX_RELEASE=hawthorn.master, however, we could setup a new Makefile variable OPENEDX_RELEASE_NAME=hawthorn if we needed to keep those independent. I think switching the Shell environment variable would be better.

3

@jmbowman and @kmccormick since you replied on the Slack thread about this and on Devstack creation time reduction – would it be possible to add something like this to release branches for the devstack repo? Then, anyone who wants to spin up a release-branch-based devstack could do it without disrupting their master devstack.

4

Short answer: This is worth doing and very possible. I’m not sure that anyone within edX will be able do it in the near future. We would likely review such a PR, though.

Long answer: I definitely think it would be worth supporting multiple Devstack instances on one machine.

Docker Compose actually supports this out-of-the-box via the COMPOSE_PROJECT_NAME environment variable. Currently it’s set to "devstack" in options.mk. Ideally, changing the value of that to "devstack-hawthorn" or something would “just work”, having the effect of namespacing all your containers and volumes.

Unfortunately, this only will work if we replace all the hard-coded container references like:

docker [exec|run|...] edx.devstack.<service>

with calls to docker-compose:

docker-compose [exec|run|...] <service>

This change would need to be made in the Makefile and all of the assorted shell scripts in the repository. We’d also need to stop explictly naming the containers in the docker-compose-*.yml files, and let Docker Compose choose names such as devstack-hawthorn_mysql and devstack-master_studio.

As @jmbowman mentioned in Slack, edX is focusing its efforts on improving deployments before turning to focus on developer tooling. While we’re not likely to make this change ourselves right now, we’re open to accepting PRs towards it.

5

Thanks for that response @kmccormick. I updated Docker Devstack – Multiple Releases – One Machine to include ${OPENEDX_RELEASE}. prefix in the following files.

  • .\devstack\Makefile
  • .\devstack\docker-compose.yml

You mentioned using the COMPOSE_PROJECT_NAME variable instead. That seems like a better option if Docker supports it.

Seems like we need to make a PR for edX to review. I’m going to attempt this and see where I can get with your recommended changes.

cc: @jill

1 Like
6

That would definitely be a nice feature! Please keep us up to date about it. :slight_smile:

7

@kmccormick @jill @toxinu I made a pull request for this here. Let’s see if all tests pass then someone can test further. I got this up and running on my Macbook Catalina 10.15.4 using NFS

make dev.nfs.provision
make dev.nfs.up

4 Likes
8

For what it’s worth, @Zachary_Trabookis, I tested your PR and it works perfectly. +1 to have it merged ASAP.

1 Like
9

@arbrandes Thanks for testing this PR out. Glad it worked for you. I also wanted to note that I’m keeping my major named releases in separate directories to avoid messing something up.

/Dev/EducateWorkforce/Repos/open-edx/devstack.master
/Dev/EducateWorkforce/Repos/open-edx/devstack.ironwood
/Dev/EducateWorkforce/Repos/open-edx/devstack.hawthorn

I’m using this .bash_profile alias to switch between the releases setting the appropriate virtual environment and setting me to start in the devstack directory.

220:devstack.ironwood ztraboo$ edxenv ironwood
Moved to the ironwood workspace
created virtual environment CPython3.7.7.final.0-64 in 448ms
  creator CPython3Posix(dest=/Users/ztraboo/Dev/EducateWorkforce/Repos/open-edx/devstack.ironwood/devstack/venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/Users/ztraboo/Library/Application Support/virtualenv/seed-app-data/v1.0.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
bash: deactivate: command not found
Activated the 'venv' for ironwood in the 'devstack' repo.
Set these environment variables:
DEVSTACK_WORKSPACE = /Users/ztraboo/Dev/EducateWorkforce/Repos/open-edx/devstack.ironwood
OPENEDX_RELEASE = ironwood.master
VIRTUAL_ENV = /Users/ztraboo/Dev/EducateWorkforce/Repos/open-edx/devstack.ironwood/devstack/venv
PWD = /Users/ztraboo/Dev/EducateWorkforce/Repos/open-edx/devstack.ironwood
OLDPWD = /Users/ztraboo/Dev/EducateWorkforce/Repos/open-edx/devstack.ironwood
/Users/ztraboo/Dev/EducateWorkforce/Repos/open-edx/devstack.ironwood/devstack/venv/bin/python

Here is the .bash_profile alias edxenv

10

Neat! What I and some folks at Opencraft have been doing is similar, but sorta the other way around. We use direnv so that when we’re in a particular directory, a set of variables is loaded, venvs activated, etc. The point is that this will now be even more useful with your patch! For example, this is my .envrc:

# ~/src/devstack-juniper/.envrc
export COMPOSE_PROJECT_NAME=devstack-juniper
export OPENEDX_RELEASE=devstack-juniper
export VIRTUAL_ENV=venv
layout python-venv

(I set OPENEDX_RELEASE to this unconventional value because I like to snapshot the containers immediately after provisioning, so I can restore them quickly after testing invasive/destructive patches.)

In any case, thanks again!

1 Like
11

Thanks @arbrandes for mentioning direnv. I’ll have to look into that. No problem I needed to make my local development environment easier to use.

12

Your PR is brilliant @Zachary_Trabookis! Thank you so much for working on this, it will save us a lot of time and pain!

2 Likes
13

No problem @jill. I needed a way to get this to work myself to handle multiple releases. Credit for planning this code change was from @kmccormick I just implemented this change.

1 Like
14

For anybody watching this thread, I took the liberty of backporting the latest version of @ztraboo’s work to the Ironwood branch of the devstack.

(It came in handy while testing migrating from ironwood.master to juniper.rc2. :slight_smile:)

15

And it’s merged! See Multiple isolated devstacks on the same host.

3 Likes
16

I was trying to install ironwood and juniper at the same machine and I encountered this issue but the reason behind was not exactly what is fixed in this PR. I am sharing here what was the issue in my case and how did I fix it so that it might help anyone else facing the issue.

In my case at a certain point in provision, mysql container was exiting with error 137. You can check by running command docker ps -a. And error 137 occurs when oom-killer kills any container due to memory limits. So in my case I’ve had assigned just 2GB for docker and changed it to 4GB and it worked. Also edX recommends to assign at least 8GB to docker.

2 Likes