Is building Docker images taking too long for you?

regis · February 27, 2023, 3:35pm

The fact that it’s taking too long to build Docker images for Open edX is consistently reported as a major pain point. To start researching this problem, I created a GitHub issue: Can we improve Docker image build time? · Issue #6 · openedx/wg-devops · GitHub

If you are affected by this issue, we’d like to hear from you. Please comment below with the following information:

In what context are you building Docker images? Is it on your personal computer, continuous integration scripts or on your production server?
Do you leverage Docker build cache? If not, why? (and yes: “it’s too complicated” is a perfectly fine answer)
Can you share build time for each one of the image-building steps? For instance, the following command produces great timing results: docker buildx build -t docker.io/overhangio/openedx:15.3.0 ~/.local/share/env/build/openedx
Do you have suggestions on how to improve the build time?

sambapete · February 27, 2023, 3:43pm

Don’t forget those that build images based on their own fork.

regis · February 27, 2023, 3:53pm

As an example, here’s what it takes to build the openedx Docker image (with Indigo theme) from scratch on a production server:

$ docker buildx build -t docker.io/overhangio/openedx:15.3.0 ~/.local/share/tutor/env/build/openedx                                       
[+] Building 897.4s (41/59)                                                                                                                                                                                        
 => [internal] load build definition from Dockerfile                                                                                                                                                          0.0s 
 => => transferring dockerfile: 10.05kB                                                                                                                                                                       0.0s 
 => [internal] load .dockerignore                                                                                                                                                                             0.0s 
 => => transferring context: 2B                                                                                                                                                                               0.0s 
 => [internal] load metadata for docker.io/library/ubuntu:20.04                                                                                                                                               0.0s 
 => [internal] load build context                                                                                                                                                                             0.1s 
 => => transferring context: 3.14MB                                                                                                                                                                           0.1s 
 => [minimal 1/2] FROM docker.io/library/ubuntu:20.04                                                                                                                                                         0.0s 
 => [minimal 2/2] RUN apt update &&     apt install -y build-essential curl git language-pack-en                                                                                                             69.5s 
 => [production  1/28] RUN apt update &&     apt install -y gettext gfortran graphviz graphviz-dev libffi-dev libfreetype6-dev libgeos-dev libjpeg8-dev liblapack-dev libmysqlclient-dev libpng-dev libsqli  49.6s
 => [locales 1/1] RUN cd /tmp     && curl -L -o openedx-i18n.tar.gz https://github.com/openedx/openedx-i18n/archive/open-release/olive.2.tar.gz     && tar xzf /tmp/openedx-i18n.tar.gz     && mkdir -p /ope  2.1s
 => [code 1/4] RUN mkdir -p /openedx/edx-platform &&     git clone https://github.com/openedx/edx-platform.git --branch open-release/olive.2 --depth 1 /openedx/edx-platform                                 12.1s
 => [dockerize 1/1] RUN dockerize_url="https://github.com/powerman/dockerize/releases/download/v0.16.0/dockerize-linux-$(uname -m | sed 's@aarch@arm@')"     && echo "Downloading dockerize from $dockerize_  3.4s
 => [python 1/4] RUN apt update &&     apt install -y libssl-dev zlib1g-dev libbz2-dev     libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev     xz-utils tk-dev libffi-dev li  63.5s
 => [code 2/4] WORKDIR /openedx/edx-platform                                                                                                                                                                  0.0s 
 => [code 3/4] RUN git config --global user.email "tutor@overhang.io"   && git config --global user.name "Tutor"                                                                                              0.9s
 => [code 4/4] RUN curl -fsSL https://github.com/openedx/edx-platform/commit/20b93b8b01276edadddfbbb67f15714fddd81c31.patch | git am                                                                          0.9s
 => [production  2/28] RUN if [ "1000" = 0 ]; then echo "app user may not be root" && false; fi                                                                                                               0.7s
 => [production  3/28] RUN useradd --home-dir /openedx --create-home --shell /bin/bash --uid 1000 app                                                                                                         0.5s
 => [production  4/28] COPY --from=dockerize /usr/local/bin/dockerize /usr/local/bin/dockerize                                                                                                                0.1s
[+] Building 897.6s (41/59)                         
 => [production  6/28] COPY --chown=app:app --from=locales /openedx/locale /openedx/locale                                                                                                                    0.2s
 => [python 2/4] RUN git clone https://github.com/pyenv/pyenv /opt/pyenv --branch v2.2.2 --depth 1                                                                                                            2.1s
 => [python 3/4] RUN /opt/pyenv/bin/pyenv install 3.8.12                                                                                                                                                    141.2s 
 => [python 4/4] RUN /opt/pyenv/versions/3.8.12/bin/python -m venv /openedx/venv                                                                                                                              6.2s 
 => [python-requirements 1/9] RUN apt update && apt install -y software-properties-common libmysqlclient-dev libxmlsec1-dev libgeos-dev                                                                      27.1s 
 => [nodejs-requirements 1/6] RUN pip install nodeenv==1.7.0                                                                                                                                                  2.3s 
 => [nodejs-requirements 2/6] RUN nodeenv /openedx/nodeenv --node=16.14.0 --prebuilt                                                                                                                         10.8s 
 => [production  7/28] COPY --chown=app:app --from=python /opt/pyenv /opt/pyenv                                                                                                                               3.1s 
 => [nodejs-requirements 3/6] COPY --from=code /openedx/edx-platform/package.json /openedx/edx-platform/package.json                                                                                          0.1s 
 => [nodejs-requirements 4/6] COPY --from=code /openedx/edx-platform/package-lock.json /openedx/edx-platform/package-lock.json                                                                                0.1s 
 => [nodejs-requirements 5/6] WORKDIR /openedx/edx-platform                                                                                                                                                   0.0s 
 => [nodejs-requirements 6/6] RUN npm install --verbose --registry=https://registry.npmjs.org/                                                                                                              157.0s
 => [python-requirements 2/9] RUN pip install setuptools==65.5.1 pip==22.3.1. wheel==0.38.4                                                                                                                   8.4s 
 => [python-requirements 3/9] COPY --from=code /openedx/edx-platform/requirements/edx/base.txt /tmp/base.txt                                                                                                  0.1s 
 => [python-requirements 4/9] RUN pip install -r /tmp/base.txt                                                                                                                                              476.0s 
 => [python-requirements 5/9] RUN pip install django-redis==5.2.0                                                                                                                                             5.7s 
 => [python-requirements 6/9] RUN pip install uwsgi==2.0.21                                                                                                                                                  25.4s 
 => [python-requirements 7/9] COPY ./requirements/ /openedx/requirements                                                                                                                                      0.0s 
 => [python-requirements 8/9] RUN cd /openedx/requirements/   && touch ./private.txt   && pip install -r ./private.txt                                                                                        4.5s 
 => [python-requirements 9/9] RUN pip install 'openedx-scorm-xblock>=15.0.0,<16.0.0'                                                                                                                          7.0s
[+] Building 1722.8s (60/60) FINISHED                                                                                                                                                                              
 => [internal] load build definition from Dockerfile                                                                                                                                                          0.0s 
 => => transferring dockerfile: 10.05kB                                                                                                                                                                       0.0s 
 => [internal] load .dockerignore                                                                                                                                                                             0.0s 
 => => transferring context: 2B                                                                                                                                                                               0.0s 
 => [internal] load metadata for docker.io/library/ubuntu:20.04                                                                                                                                               0.0s 
 => [internal] load build context                                                                                                                                                                             0.1s 
 => => transferring context: 3.14MB                                                                                                                                                                           0.1s 
 => [minimal 1/2] FROM docker.io/library/ubuntu:20.04                                                                                                                                                         0.0s 
 => [minimal 2/2] RUN apt update &&     apt install -y build-essential curl git language-pack-en                                                                                                             69.5s 
 => [production  1/28] RUN apt update &&     apt install -y gettext gfortran graphviz graphviz-dev libffi-dev libfreetype6-dev libgeos-dev libjpeg8-dev liblapack-dev libmysqlclient-dev libpng-dev libsqli  49.6s 
 => [locales 1/1] RUN cd /tmp     && curl -L -o openedx-i18n.tar.gz https://github.com/openedx/openedx-i18n/archive/open-release/olive.2.tar.gz     && tar xzf /tmp/openedx-i18n.tar.gz     && mkdir -p /ope  2.1s 
 => [code 1/4] RUN mkdir -p /openedx/edx-platform &&     git clone https://github.com/openedx/edx-platform.git --branch open-release/olive.2 --depth 1 /openedx/edx-platform                                 12.1s 
 => [dockerize 1/1] RUN dockerize_url="https://github.com/powerman/dockerize/releases/download/v0.16.0/dockerize-linux-$(uname -m | sed 's@aarch@arm@')"     && echo "Downloading dockerize from $dockerize_  3.4s 
 => [python 1/4] RUN apt update &&     apt install -y libssl-dev zlib1g-dev libbz2-dev     libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev     xz-utils tk-dev libffi-dev li  63.5s 
 => [code 2/4] WORKDIR /openedx/edx-platform                                                                                                                                                                  0.0s 
 => [code 3/4] RUN git config --global user.email "tutor@overhang.io"   && git config --global user.name "Tutor"                                                                                              0.9s 
 => [code 4/4] RUN curl -fsSL https://github.com/openedx/edx-platform/commit/20b93b8b01276edadddfbbb67f15714fddd81c31.patch | git am                                                                          0.9s 
 => [production  2/28] RUN if [ "1000" = 0 ]; then echo "app user may not be root" && false; fi                                                                                                               0.7s 
 => [production  3/28] RUN useradd --home-dir /openedx --create-home --shell /bin/bash --uid 1000 app                                                                                                         0.5s 
 => [production  4/28] COPY --from=dockerize /usr/local/bin/dockerize /usr/local/bin/dockerize                                                                                                                0.1s 
 => [production  5/28] COPY --chown=app:app --from=code /openedx/edx-platform /openedx/edx-platform                                                                                                           5.1s 
 => [production  6/28] COPY --chown=app:app --from=locales /openedx/locale /openedx/locale                                                                                                                    0.2s 
 => [python 2/4] RUN git clone https://github.com/pyenv/pyenv /opt/pyenv --branch v2.2.2 --depth 1                                                                                                            2.1s 
 => [python 3/4] RUN /opt/pyenv/bin/pyenv install 3.8.12                                                                                                                                                    141.2s 
 => [python 4/4] RUN /opt/pyenv/versions/3.8.12/bin/python -m venv /openedx/venv                                                                                                                              6.2s
 => [python-requirements 1/9] RUN apt update && apt install -y software-properties-common libmysqlclient-dev libxmlsec1-dev libgeos-dev                                                                      27.1s
 => [nodejs-requirements 1/6] RUN pip install nodeenv==1.7.0                                                                                                                                                  2.3s
 => [nodejs-requirements 2/6] RUN nodeenv /openedx/nodeenv --node=16.14.0 --prebuilt                                                                                                                         10.8s
 => [production  7/28] COPY --chown=app:app --from=python /opt/pyenv /opt/pyenv                                                                                                                               3.1s
 => [nodejs-requirements 3/6] COPY --from=code /openedx/edx-platform/package.json /openedx/edx-platform/package.json                                                                                          0.1s
 => [nodejs-requirements 4/6] COPY --from=code /openedx/edx-platform/package-lock.json /openedx/edx-platform/package-lock.json                                                                                0.1s
 => [nodejs-requirements 5/6] WORKDIR /openedx/edx-platform                                                                                                                                                   0.0s
 => [nodejs-requirements 6/6] RUN npm install --verbose --registry=https://registry.npmjs.org/                                                                                                              157.0s
 => [python-requirements 2/9] RUN pip install setuptools==65.5.1 pip==22.3.1. wheel==0.38.4                                                                                                                   8.4s
 => [python-requirements 3/9] COPY --from=code /openedx/edx-platform/requirements/edx/base.txt /tmp/base.txt                                                                                                  0.1s
 => [python-requirements 4/9] RUN pip install -r /tmp/base.txt                                                                                                                                              476.0s
 => [python-requirements 5/9] RUN pip install django-redis==5.2.0                                                                                                                                             5.7s
 => [python-requirements 6/9] RUN pip install uwsgi==2.0.21                                                                                                                                                  25.4s
 => [python-requirements 7/9] COPY ./requirements/ /openedx/requirements                                                                                                                                      0.0s
 => [python-requirements 8/9] RUN cd /openedx/requirements/   && touch ./private.txt   && pip install -r ./private.txt                                                                                        4.5s
 => [python-requirements 9/9] RUN pip install 'openedx-scorm-xblock>=15.0.0,<16.0.0'                                                                                                                          7.0s
 => [production  8/28] COPY --chown=app:app --from=python-requirements /openedx/venv /openedx/venv                                                                                                           16.6s
 => [production  9/28] COPY --chown=app:app --from=python-requirements /openedx/requirements /openedx/requirements                                                                                            0.1s
 => [production 10/28] COPY --chown=app:app --from=nodejs-requirements /openedx/nodeenv /openedx/nodeenv                                                                                                      5.0s
 => [production 11/28] COPY --chown=app:app --from=nodejs-requirements /openedx/edx-platform/node_modules /openedx/edx-platform/node_modules                                                                 38.0s
 => [production 12/28] WORKDIR /openedx/edx-platform                                                                                                                                                          0.0s
 => [nodejs-requirements 6/6] RUN npm install --verbose --registry=https://registry.npmjs.org/                                                                                                              157.0s
 => [python-requirements 2/9] RUN pip install setuptools==65.5.1 pip==22.3.1. wheel==0.38.4                                                                                                                   8.4s
 => [python-requirements 3/9] COPY --from=code /openedx/edx-platform/requirements/edx/base.txt /tmp/base.txt                                                                                                  0.1s
 => [python-requirements 4/9] RUN pip install -r /tmp/base.txt                                                                                                                                              476.0s
 => [python-requirements 5/9] RUN pip install django-redis==5.2.0                                                                                                                                             5.7s
 => [python-requirements 6/9] RUN pip install uwsgi==2.0.21                                                                                                                                                  25.4s
 => [python-requirements 7/9] COPY ./requirements/ /openedx/requirements                                                                                                                                      0.0s
 => [python-requirements 8/9] RUN cd /openedx/requirements/   && touch ./private.txt   && pip install -r ./private.txt                                                                                        4.5s
 => [python-requirements 9/9] RUN pip install 'openedx-scorm-xblock>=15.0.0,<16.0.0'                                                                                                                          7.0s
 => [production  8/28] COPY --chown=app:app --from=python-requirements /openedx/venv /openedx/venv                                                                                                           16.6s
 => [production  9/28] COPY --chown=app:app --from=python-requirements /openedx/requirements /openedx/requirements                                                                                            0.1s
 => [production 10/28] COPY --chown=app:app --from=nodejs-requirements /openedx/nodeenv /openedx/nodeenv                                                                                                      5.0s
 => [production 11/28] COPY --chown=app:app --from=nodejs-requirements /openedx/edx-platform/node_modules /openedx/edx-platform/node_modules                                                                 38.0s
 => [production 12/28] WORKDIR /openedx/edx-platform                                                                                                                                                          0.0s
 => [production 13/28] RUN pip install -e .                                                                                                                                                                   9.0s
 => [production 14/28] RUN mkdir -p /openedx/config ./lms/envs/tutor ./cms/envs/tutor                                                                                                                         0.7s
 => [production 15/28] COPY --chown=app:app revisions.yml /openedx/config/                                                                                                                                    0.1s
 => [production 16/28] COPY --chown=app:app settings/lms/*.py ./lms/envs/tutor/                                                                                                                               0.0s
 => [production 17/28] COPY --chown=app:app settings/cms/*.py ./cms/envs/tutor/                                                                                                                               0.0s
 => [production 18/28] RUN mkdir /openedx/locale/user                                                                                                                                                         0.5s
 => [production 19/28] COPY --chown=app:app ./locale/ /openedx/locale/user/locale/                                                                                                                            0.0s
 => [production 20/28] RUN cd /openedx/locale/user &&     django-admin compilemessages -v1                                                                                                                    0.8s
 => [production 21/28] RUN ./manage.py lms --settings=tutor.i18n compilejsi18n                                                                                                                               21.5s
 => [production 22/28] RUN ./manage.py cms --settings=tutor.i18n compilejsi18n                                                                                                                               18.5s
 => [production 23/28] COPY --chown=app:app ./bin /openedx/bin                                                                                                                                                0.4s
 => [production 24/28] RUN chmod a+x /openedx/bin/*                                                                                                                                                           1.1s
 => [production 25/28] RUN openedx-assets xmodule     && openedx-assets npm     && openedx-assets webpack --env=prod     && openedx-assets common                                                           353.9s
 => [production 26/28] COPY --chown=app:app ./themes/ /openedx/themes/                                                                                                                                        0.1s
 => [production 27/28] RUN openedx-assets themes     && openedx-assets collect --settings=tutor.assets     && rdfind -makesymlinks true -followsymlinks true /openedx/staticfiles/                          307.8s
 => [production 28/28] RUN mkdir /openedx/data                                                                                                                                                                1.3s
 => exporting to image                                                                                                                                                                                       98.3s
 => => exporting layers                                                                                                                                                                                      98.3s
 => => writing image sha256:b1e935db4a3cfddd99e836e339ca332120dd0e0687c21b658e1df0a247ad6270                                                                                                                  0.0s
 => => naming to docker.io/overhangio/openedx:15.3.0                                                                                                                                                          0.0s

It takes 1722.8s (~28 minutes) to make a full build. Here are the top offenders:

 => [python-requirements 4/9] RUN pip install -r /tmp/base.txt                                                                                                                                              476.0s
 => [production 25/28] RUN openedx-assets xmodule     && openedx-assets npm     && openedx-assets webpack --env=prod     && openedx-assets common                                                           353.9s
 => [production 27/28] RUN openedx-assets themes     && openedx-assets collect --settings=tutor.assets     && rdfind -makesymlinks true -followsymlinks true /openedx/staticfiles/                          307.8s
 => [nodejs-requirements 6/6] RUN npm install --verbose --registry=https://registry.npmjs.org/                                                                                                              157.0s

tramck · February 27, 2023, 6:08pm

We are building docker images based on our own fork of edx-platform. We build both on our dev vms and in github workflows for ci. We can afford long build times in dev, but where it really hurts our team is the fact that it can take around 35 minutes for the build on github to complete.
We aren’t explicitly disabling docker build cache anywhere, but I don’t think we are persisting anything in our github workflows to utilize the cache in ci.
Here are the timestamps for each step from running tutor images build openedx locally on my dev vm:

Mon Feb 27 17:29:18 UTC 2023| Step 1/100 : FROM docker.io/ubuntu:20.04 as minimal
Mon Feb 27 17:29:18 UTC 2023| Step 2/100 : LABEL maintainer="Overhang.io <contact@overhang.io>"
Mon Feb 27 17:29:18 UTC 2023| Step 3/100 : ENV DEBIAN_FRONTEND=noninteractive
Mon Feb 27 17:29:18 UTC 2023| Step 4/100 : RUN apt update &&     apt install -y build-essential curl git language-pack-en
Mon Feb 27 17:30:40 UTC 2023| Step 5/100 : ENV LC_ALL en_US.UTF-8
Mon Feb 27 17:30:40 UTC 2023| Step 6/100 : FROM minimal as python
Mon Feb 27 17:30:40 UTC 2023| Step 7/100 : RUN apt update &&     apt install -y libssl-dev zlib1g-dev libbz2-dev     libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev     xz-utils tk-dev libffi-dev liblzma-dev python-openssl git
Mon Feb 27 17:31:57 UTC 2023| Step 8/100 : ARG PYTHON_VERSION=3.8.12
Mon Feb 27 17:31:57 UTC 2023| Step 9/100 : ENV PYENV_ROOT /opt/pyenv
Mon Feb 27 17:31:57 UTC 2023| Step 10/100 : RUN git clone https://github.com/pyenv/pyenv $PYENV_ROOT --branch v2.2.2 --depth 1
Mon Feb 27 17:31:59 UTC 2023| Step 11/100 : RUN $PYENV_ROOT/bin/pyenv install $PYTHON_VERSION
Mon Feb 27 17:33:47 UTC 2023| Step 12/100 : RUN $PYENV_ROOT/versions/$PYTHON_VERSION/bin/python -m venv /openedx/venv
Mon Feb 27 17:33:52 UTC 2023| Step 13/100 : FROM minimal as dockerize
Mon Feb 27 17:33:52 UTC 2023| Step 14/100 : ARG DOCKERIZE_VERSION=v0.16.0
Mon Feb 27 17:33:52 UTC 2023| Step 15/100 : RUN dockerize_url="https://github.com/powerman/dockerize/releases/download/$DOCKERIZE_VERSION/dockerize-linux-$(uname -m | sed 's@aarch@arm@')"     && echo "Downloading dockerize from $dockerize_url"     && curl --fail --location --output /usr/local/bin/dockerize $dockerize_url     && chmod a+x /usr/local/bin/dockerize
Mon Feb 27 17:33:54 UTC 2023| Step 16/100 : FROM minimal as code
Mon Feb 27 17:33:54 UTC 2023| Step 17/100 : ARG EDX_PLATFORM_REPOSITORY=https://github.com/openedx/edx-platform.git
Mon Feb 27 17:33:54 UTC 2023| Step 18/100 : ARG EDX_PLATFORM_VERSION=open-release/olive.1
Mon Feb 27 17:33:54 UTC 2023| Step 19/100 : RUN mkdir -p /openedx/edx-platform &&     git clone $EDX_PLATFORM_REPOSITORY --branch $EDX_PLATFORM_VERSION --depth 1 /openedx/edx-platform
Mon Feb 27 17:34:10 UTC 2023| Step 20/100 : WORKDIR /openedx/edx-platform
Mon Feb 27 17:34:10 UTC 2023| Step 21/100 : RUN git config --global user.email "tutor@overhang.io"   && git config --global user.name "Tutor"
Mon Feb 27 17:34:11 UTC 2023| Step 22/100 : RUN curl -fsSL https://github.com/openedx/edx-platform/commit/20b93b8b01276edadddfbbb67f15714fddd81c31.patch | git am
Mon Feb 27 17:34:12 UTC 2023| Step 23/100 : FROM minimal as locales
Mon Feb 27 17:34:12 UTC 2023| Step 24/100 : ARG OPENEDX_I18N_VERSION=open-release/olive.1
Mon Feb 27 17:34:12 UTC 2023| Step 25/100 : RUN cd /tmp     && curl -L -o openedx-i18n.tar.gz https://github.com/openedx/openedx-i18n/archive/$OPENEDX_I18N_VERSION.tar.gz     && tar xzf /tmp/openedx-i18n.tar.gz     && mkdir -p /openedx/locale/contrib     && mv openedx-i18n-*/edx-platform/locale /openedx/locale/contrib     && rm -rf openedx-i18n*
Mon Feb 27 17:34:14 UTC 2023| Step 26/100 : FROM python as python-requirements
Mon Feb 27 17:34:14 UTC 2023| Step 27/100 : ENV PATH /openedx/venv/bin:${PATH}
Mon Feb 27 17:34:14 UTC 2023| Step 28/100 : ENV VIRTUAL_ENV /openedx/venv/
Mon Feb 27 17:34:15 UTC 2023| Step 29/100 : RUN apt update && apt install -y software-properties-common libmysqlclient-dev libxmlsec1-dev libgeos-dev
Mon Feb 27 17:34:47 UTC 2023| Step 30/100 : RUN pip install setuptools==65.5.1 pip==22.3.1. wheel==0.38.4
Mon Feb 27 17:34:55 UTC 2023| Step 31/100 : COPY --from=code /openedx/edx-platform/requirements/edx/base.txt /tmp/base.txt
Mon Feb 27 17:34:55 UTC 2023| Step 32/100 : RUN pip install -r /tmp/base.txt
Mon Feb 27 17:40:19 UTC 2023| Step 33/100 : RUN pip install django-redis==5.2.0
Mon Feb 27 17:40:22 UTC 2023| Step 34/100 : RUN pip install uwsgi==2.0.21
Mon Feb 27 17:40:39 UTC 2023| Step 35/100 : COPY ./requirements/ /openedx/requirements
Mon Feb 27 17:40:39 UTC 2023| Step 36/100 : RUN cd /openedx/requirements/   && touch ./private.txt   && pip install -r ./private.txt
Mon Feb 27 17:40:55 UTC 2023| Step 37/100 : RUN pip install 'openedx-scorm-xblock>=15.0.0,<16.0.0'
Mon Feb 27 17:41:00 UTC 2023| Step 38/100 : FROM python as nodejs-requirements
Mon Feb 27 17:41:00 UTC 2023| Step 39/100 : ENV PATH /openedx/nodeenv/bin:/openedx/venv/bin:${PATH}
Mon Feb 27 17:41:01 UTC 2023| Step 40/100 : RUN pip install nodeenv==1.7.0
Mon Feb 27 17:41:02 UTC 2023| Step 41/100 : RUN nodeenv /openedx/nodeenv --node=16.14.0 --prebuilt
Mon Feb 27 17:41:10 UTC 2023| Step 42/100 : ARG NPM_REGISTRY=https://registry.npmjs.org/
Mon Feb 27 17:41:10 UTC 2023| Step 43/100 : COPY --from=code /openedx/edx-platform/package.json /openedx/edx-platform/package.json
Mon Feb 27 17:41:10 UTC 2023| Step 44/100 : COPY --from=code /openedx/edx-platform/package-lock.json /openedx/edx-platform/package-lock.json
Mon Feb 27 17:41:10 UTC 2023| Step 45/100 : WORKDIR /openedx/edx-platform
Mon Feb 27 17:41:10 UTC 2023| Step 46/100 : RUN npm install --verbose --registry=$NPM_REGISTRY
Mon Feb 27 17:43:24 UTC 2023| Step 47/100 : FROM minimal as production
Mon Feb 27 17:43:24 UTC 2023| Step 48/100 : RUN apt update &&     apt install -y gettext gfortran graphviz graphviz-dev libffi-dev libfreetype6-dev libgeos-dev libjpeg8-dev liblapack-dev libmysqlclient-dev libpng-dev libsqlite3-dev libxmlsec1-dev lynx ntp pkg-config rdfind &&     rm -rf /var/lib/apt/lists/*
Mon Feb 27 17:44:23 UTC 2023| Step 49/100 : ARG APP_USER_ID=1000
Mon Feb 27 17:44:23 UTC 2023| Step 50/100 : RUN if [ "$APP_USER_ID" = 0 ]; then echo "app user may not be root" && false; fi
Mon Feb 27 17:44:24 UTC 2023| Step 51/100 : RUN useradd --home-dir /openedx --create-home --shell /bin/bash --uid ${APP_USER_ID} app
Mon Feb 27 17:44:25 UTC 2023| Step 52/100 : USER ${APP_USER_ID}
Mon Feb 27 17:44:25 UTC 2023| Step 53/100 : COPY --from=dockerize /usr/local/bin/dockerize /usr/local/bin/dockerize
Mon Feb 27 17:44:25 UTC 2023| Step 54/100 : COPY --chown=app:app --from=code /openedx/edx-platform /openedx/edx-platform
Mon Feb 27 17:44:39 UTC 2023| Step 55/100 : COPY --chown=app:app --from=locales /openedx/locale /openedx/locale
Mon Feb 27 17:44:39 UTC 2023| Step 56/100 : COPY --chown=app:app --from=python /opt/pyenv /opt/pyenv
Mon Feb 27 17:44:51 UTC 2023| Step 57/100 : COPY --chown=app:app --from=python-requirements /openedx/venv /openedx/venv
Mon Feb 27 17:45:39 UTC 2023| Step 58/100 : COPY --chown=app:app --from=python-requirements /openedx/requirements /openedx/requirements
Mon Feb 27 17:45:39 UTC 2023| Step 59/100 : COPY --chown=app:app --from=nodejs-requirements /openedx/nodeenv /openedx/nodeenv
Mon Feb 27 17:45:49 UTC 2023| Step 60/100 : COPY --chown=app:app --from=nodejs-requirements /openedx/edx-platform/node_modules /openedx/edx-platform/node_modules
Mon Feb 27 17:46:53 UTC 2023| Step 61/100 : ENV PATH /openedx/venv/bin:./node_modules/.bin:/openedx/nodeenv/bin:${PATH}
Mon Feb 27 17:46:53 UTC 2023| Step 62/100 : ENV VIRTUAL_ENV /openedx/venv/
Mon Feb 27 17:46:53 UTC 2023| Step 63/100 : WORKDIR /openedx/edx-platform
Mon Feb 27 17:46:53 UTC 2023| Step 64/100 : RUN pip install -e .
Mon Feb 27 17:46:59 UTC 2023| Step 65/100 : RUN mkdir -p /openedx/config ./lms/envs/tutor ./cms/envs/tutor
Mon Feb 27 17:47:00 UTC 2023| Step 66/100 : COPY --chown=app:app revisions.yml /openedx/config/
Mon Feb 27 17:47:00 UTC 2023| Step 67/100 : ENV LMS_CFG /openedx/config/lms.env.yml
Mon Feb 27 17:47:00 UTC 2023| Step 68/100 : ENV CMS_CFG /openedx/config/cms.env.yml
Mon Feb 27 17:47:00 UTC 2023| Step 69/100 : ENV REVISION_CFG /openedx/config/revisions.yml
Mon Feb 27 17:47:00 UTC 2023| Step 70/100 : COPY --chown=app:app settings/lms/*.py ./lms/envs/tutor/
Mon Feb 27 17:47:01 UTC 2023| Step 71/100 : COPY --chown=app:app settings/cms/*.py ./cms/envs/tutor/
Mon Feb 27 17:47:01 UTC 2023| Step 72/100 : RUN mkdir /openedx/locale/user
Mon Feb 27 17:47:02 UTC 2023| Step 73/100 : COPY --chown=app:app ./locale/ /openedx/locale/user/locale/
Mon Feb 27 17:47:02 UTC 2023| Step 74/100 : RUN cd /openedx/locale/user &&     django-admin compilemessages -v1
Mon Feb 27 17:47:03 UTC 2023| Step 75/100 : RUN ./manage.py lms --settings=tutor.i18n compilejsi18n
Mon Feb 27 17:47:14 UTC 2023| Step 76/100 : RUN ./manage.py cms --settings=tutor.i18n compilejsi18n
Mon Feb 27 17:47:24 UTC 2023| Step 77/100 : COPY --chown=app:app ./bin /openedx/bin
Mon Feb 27 17:47:24 UTC 2023| Step 78/100 : RUN chmod a+x /openedx/bin/*
Mon Feb 27 17:47:25 UTC 2023| Step 79/100 : ENV PATH /openedx/bin:${PATH}
Mon Feb 27 17:47:25 UTC 2023| Step 80/100 : ENV NO_PYTHON_UNINSTALL 1
Mon Feb 27 17:47:25 UTC 2023| Step 81/100 : ENV NO_PREREQ_INSTALL 1
Mon Feb 27 17:47:25 UTC 2023| Step 82/100 : RUN openedx-assets xmodule     && openedx-assets npm     && openedx-assets webpack --env=prod     && openedx-assets common
Mon Feb 27 17:51:11 UTC 2023| Step 83/100 : COPY --chown=app:app ./themes/ /openedx/themes/
Mon Feb 27 17:51:12 UTC 2023| Step 84/100 : RUN openedx-assets themes     && openedx-assets collect --settings=tutor.assets     && rdfind -makesymlinks true -followsymlinks true /openedx/staticfiles/
Mon Feb 27 17:54:17 UTC 2023| Step 85/100 : RUN mkdir /openedx/data
Mon Feb 27 17:54:18 UTC 2023| Step 86/100 : ENV SERVICE_VARIANT lms
Mon Feb 27 17:54:18 UTC 2023| Step 87/100 : ENV DJANGO_SETTINGS_MODULE lms.envs.tutor.production
Mon Feb 27 17:54:18 UTC 2023| Step 88/100 : EXPOSE 8000
Mon Feb 27 17:54:18 UTC 2023| Step 89/100 : FROM production as development
Mon Feb 27 17:54:18 UTC 2023| Step 90/100 : USER root
Mon Feb 27 17:54:19 UTC 2023| Step 91/100 : RUN apt update &&     apt install -y vim iputils-ping dnsutils telnet     && rm -rf /var/lib/apt/lists/*
Mon Feb 27 17:54:36 UTC 2023| Step 92/100 : USER app
Mon Feb 27 17:54:36 UTC 2023| Step 93/100 : RUN pip install -r requirements/edx/development.txt
Mon Feb 27 17:56:06 UTC 2023| Step 94/100 : RUN pip install ipdb==0.13.9 ipython==8.7.0
Mon Feb 27 17:56:15 UTC 2023| Step 95/100 : ENV PYTHONBREAKPOINT=ipdb.set_trace
Mon Feb 27 17:56:15 UTC 2023| Step 96/100 : RUN rm -r /openedx/staticfiles &&     mkdir /openedx/staticfiles &&     openedx-assets webpack --env=dev
Mon Feb 27 17:57:19 UTC 2023| Step 97/100 : ENV DJANGO_SETTINGS_MODULE lms.envs.tutor.development
Mon Feb 27 17:57:19 UTC 2023| Step 98/100 : CMD ./manage.py $SERVICE_VARIANT runserver 0.0.0.0:8000
Mon Feb 27 17:57:19 UTC 2023| Step 99/100 : FROM production as final
Mon Feb 27 17:57:19 UTC 2023| Step 100/100 : CMD uwsgi     --static-map /static=/openedx/staticfiles/     --static-map /media=/openedx/media/     --http 0.0.0.0:8000     --thunder-lock     --single-interpreter     --enable-threads     --processes=${UWSGI_WORKERS:-2}     --buffer-size=8192     --wsgi-file $SERVICE_VARIANT/wsgi.py

braden · February 27, 2023, 6:35pm

This npm install seems to happen four times (each taking exactly the same time), and accounts for fully 36% of the total time. Is this doing the same thing four times or doing four different things? Is there still some blocker preventing us from going back to npm ci which is much faster?

regis · February 28, 2023, 9:12am

My bad, my script to extract top offenders was incorrect. This step does happen just once. Yes, you’re making a good point that we should revert back to npm ci. This should gain us ~70s. I’ll add this item to the GitHub issue.

ghassan · February 28, 2023, 11:35am

I have two things to say here:
1- Can we have a base image that start from or include openedx pip depends, i.e start from an image that comes preshipped with openedx depends instead from ubuntu. I don’t know the consequence of this, it would at least require supporting two images, 1)openedx-base (which only include the depends), and 2)openedx (which comes with theme compile…etc). So instead of installing them from pip we just pull a docker image layer

2- Giving a quick look to the legacy webopack config in edx-platform it seems by default it compile files for features that are not commmonly used to my knowledge, i.e it compile certificate files which probably is already replaced by other seperate service. Add to that files/assests that are already replaced by an MFEs for example to my knowledge frontend-app-account is required to use openedx but its assets are still being compiled

kmccormick · February 28, 2023, 9:14pm

Yes, from the perspective of Tutor-as-a-Devstack-replacement, I have found & heard that Tutor spends too long building images. I’ve been collecting of issues relating to it, and found them to be in one of three categories:

Optimize dependency installation

edx-platform has a lot of dependencies, and that has a big impact on the build time.

github.com/openedx/wg-developer-experience

Convert GitHub dependencies to PyPI dependencies wherever possible

opened 06:47PM - 04 Oct 22 UTC

kdmccormick

## Background Several packages are installed directly from GitHub instead of …PyPI. For example: [edx-platform's github.in](https://github.com/openedx/edx-platform/blob/master/requirements/edx/github.in). This is non-ideal because: * GitHub-hosted requirements take significantly longer (> 4x) to install than PyPI-hosted requirements (I care about this for Tutor build speed :) * GitHub-hosted requirement pins cannot be upgraded using our standard `make upgrade` workflow. ## Acceptance Criteria In edx-platform, remove all packages from github.in. Investigate use of GitHub-hosted deps in other IDAs. Remove them, or create follow-up tickets. Add guidance that steers developers away from using GitHub-hosted deps and/or makes it easier for them to publish their packages to PyPI. Consider using a repo health check or CODEOWNERS to help us hold ourselves to this. ## Subtasks - [ ] Overall - [x] ~https://github.com/openedx/open-edx-proposals/pull/390~ - [x] ~https://github.com/openedx/edx-platform/pull/30719~ - [x] ~https://github.com/openedx/edx-platform/pull/31104~ - [ ] https://github.com/openedx/open-edx-proposals/pull/450 - [ ] https://github.com/openedx/edx-platform/pull/31824 - [ ] (Link to follow-up for IDAs) - [x] codejail - [x] https://github.com/openedx/codejail/pull/140 - [x] https://github.com/openedx/codejail/pull/151 - [x] https://github.com/openedx/edx-platform/pull/31683 - [x] ratexblock - [x] https://github.com/openedx/RateXBlock/issues/45 - [x] xblock-drag-and-drop-v2 - [x] https://github.com/openedx/xblock-drag-and-drop-v2/issues/299 - [x] blockstore - [x] https://github.com/openedx/blockstore/pull/202 - [x] https://github.com/openedx/edx-platform/pull/30601 - [x] https://github.com/openedx/edx-platform/pull/31236 - [x] https://github.com/openedx/blockstore/issues/212 - [x] django-wiki - [x] https://github.com/openedx/edx-platform/pull/31271 - [ ] mongodbproxy - [ ] TO DO: Publish this to PyPI - [ ] TO DO: install it from PyPI in edx-platform - [ ] olxcleaner - [x] https://github.com/openedx/olxcleaner/pull/16 - [x] https://github.com/openedx/olxcleaner/pull/17 - [ ] https://github.com/openedx/edx-platform/pull/31819 - [x] django-require - [x] https://github.com/openedx/edx-platform/pull/31279 - [x] xblock-poll - [x] https://github.com/open-craft/xblock-poll/issues/98 - [x] https://github.com/openedx/edx-platform/pull/31718 - [x] xblock-google-drive - [x] https://github.com/openedx/edx-platform/pull/31201

github.com/edx/edx-arch-experiments

Always install pre-built wheels

opened 08:47PM - 01 Feb 23 UTC

jmbowman

There are a few reasons we would prefer to always install Python packages as whe…els rather than source tarballs: * Faster, especially for packages with C or Rust extensions but even for pure Python packages * Enables us to eliminate hundreds of MB of compilation tools from our Docker images * More secure, by removing opportunities for arbitrary code execution during installation However, not all of our dependencies have wheels on PyPI for the versions we use. And of those that do, not all of them have all of the process architectures we would like to support. But this can be worked around. Tentative proposal: * Document the set of binary formats (OS/architecture combinations) we want to support * Make sure that all of the Open edX packages get all of the appropriate wheels pushed to PyPI with each release. This will generally be a single universal wheel unless the package contains native code extensions. * Set up at least one package server for 2U to host wheels for its private code and dependencies which don't have all the needed wheels on PyPI. Ideally there would be a second managed by tCRIL to host just the missing dependency wheels for the benefit of the entire Open edX community. * Create a repository and Docker image(s) with all of the dependencies needed to build the missing dependency wheels. This could also be used in development to try out new packages with missing wheel variants on PyPI, building them here and then supplying them to the dev environment's devpi instance. * Update pip configuration to allow installation from the custom package server(s) * Remove native compilation tooling and development header packages from all the other Docker images. Some of the benefits could be obtained by instead/also using a builder pattern for Dockerfiles such that the compilation toolchain isn't actually in the images used for development and production, but that leaves some of the security risk, a longer image build time, and occasional hassles in development when installing a new package with a missing binary wheel. Some package server options: * [devpi](https://github.com/devpi/devpi) - Just for dev environments? Already used in devstack. * [AWS Artifact Repository](https://aws.amazon.com/codeartifact/) - Easiest for 2U to start using? * [JFrog Artifactory](https://jfrog.com/artifactory/) * [Sonatype Nexus](https://www.sonatype.com/products/nexus-repository) Some tooling that helps build and upload wheels: * [cibuildwheel](https://github.com/pypa/cibuildwheel) - Official Python Packaging Authority project * [wheelwright](https://github.com/explosion/wheelwright) * [multibuild](https://github.com/multi-build/multibuild)

github.com/openedx/edx-platform

Audit and clean up edx-platform requirements

opened 06:55PM - 05 Jan 23 UTC

kdmccormick

### Background As part of https://github.com/openedx/wg-developer-experience/…issues/146, we want to optimize `pip install` for edx-platform. The more packages that edx-platform requires, the longer it will take `pip install` to run, the more bandwidth it'll use, and the larger the resulting image will be. ### Tasks Find requirements that could either be (a) removed from requirements, or (b) moved from [base](https://github.com/openedx/edx-platform/blob/master/requirements/edx/base.txt) requirements out to one of the other requirements contexts, like [development](https://github.com/openedx/edx-platform/blob/master/requirements/edx/development.txt). Some ideas: * Devise a way to check whether each edx-platform requirement (that is, the Python requirement lists in `requirements/`, and the JS requirement lists in `package.json`) is actually being used by the application. * Manually scan the list of requirements.: * Look for things that seem like they might be unused. * Look for things in base.txt that look like they belongs in development.txt. * Look for things in package.json that are marked as dependencies but should be devDependencies. * Check out the "via" comments, which tell you where the requirements are coming from--oftentimes, a change may need to be made in a package repository in order to remove a transitive dependency. As you find requirements that could be removed or moved, either make PRs yourself or write up an issue for our backlog. Add those PRs and issues here: - [ ] Remove libsass from base requirements - [ ] #31766 - [ ] #31767 - [ ] #31768

Optimize the asset build

The legacy edx-platform asset build takes a long time and it seems to get re-run a lot when you’re using Tutor for development. I think we can make it happen less often (by removing Python from it & then moving it to an independent build stage) and I think we can make it faster.

github.com/openedx/edx-platform

Build edx-platform assets without Python

opened 04:27AM - 21 Feb 23 UTC

kdmccormick

### Background Follow-up from: * https://github.com/openedx/edx-platform/iss…ues/31604 ### Goal edx-platform's static assets can be built without installing a Python interpreter or any Python packages. Benefits: * **Better staged image builds:** When building a container image for edx-platform, the build stage for assets will not need to install any Python requirements or copy in Python code. Thus, the asset build cache will not be broken due to changes to Python or Python requirements lists. * **Saner tooling:** Dropping our deprecated and/or bespoke Python frontend tools in favor of simpler, modern JS-based or shell-based tools makes the platform's build process easier to understand and maintain. ### Tasks - [ ] openedx/edx-platform#31607 - [ ] EITHER: - [ ] openedx/edx-platform#31611 OR - [ ] openedx/edx-platform#31624

OpenCraft did a great deep dive into the XModule part of the asset build, which I haven’t fully reviewed yet, but I’ll link here for those that are interested.

Reduce number of layers that need to be rebuilt

Even if the total image build time is upwards of 30 minutes, the developer experience can be hugely improved if the Dockerfile is constructed & used in such that developers rarely need to rebuild the full image.

github.com/openedx/wg-developer-experience

When building image, use layer cache from pulled image

opened 09:43PM - 07 Oct 22 UTC

kdmccormick

### Background When I run: ``` tutor images pull openedx ``` it pulls `op…enedx:VERSION` from Dockerhub. Docker reuses any image layers that are already in the cache, and it's a no-op if I already have the latest image downloaded. Great. When I run: ``` tutor dev dc build lms ``` it builds `openedx-dev:VERSION`, but it will *not* reuse the image layers pulled from `openedx:VERSION`, even though `openedx` shares several dozen layers with `openedx-dev`! It will start right from Step 1 at the top of the Dockerfile. That makes the build take way more time and bandwidth than it needs to. ### Tasks TBD

github.com/openedx/wg-developer-experience

Allow adding local dev requirements without rebuilding base image

opened 03:17PM - 07 Feb 23 UTC

kdmccormick

### Background The current [package dev notes](https://docs.tutor.overhang.io…/dev.html#xblock-and-edx-platform-plugin-development) involve: * moving the package into `$(tutor config printroot)/env/build/openedx/requirements` * listing the package in `$(tutor config printroot)/env/build/openedx/requirements/private.txt` * rebuilding the base edx-platform image (`tutor images build openedx`). Because the local packages are installed so high up in the Dockerfile, this rebuild breaks most of the cache and thus can take 20+ minutes. The comprehensive image rebuild makes sense when the user intends to use these requirements on a production site: it is important that the local packages be built into the base image, and it's important that production static assets are built afterward, etc. Developers, on the other hand, don't need the packages built into the base image. They would be happy if the package was just built in to the development variant of the image, which takes much less time (~1 minute) to rebuild. ### Tasks - [ ] Propose a solution. May require a TEP. - [ ] Implement it. ### Notes Prototype implementation here: https://github.com/kdmccormick/tutor/pull/26

ghassan · March 1, 2023, 11:35am

That all sounds good @kmccormick I have been seeing how much energy you are putting to make tutor better from dev prespestive.
I wonder what do you think about having a seperate image that comes preshipped with the default common depends, and this way when developer are adding new pip package they wouldn’t need to reinstall all other pips even if there was no cache

regis · March 2, 2023, 10:58am

Thanks for your comments @kmccormick and @ghassan! I added some links and comments to the GitHub issue.

I’d rather avoid discussing here the details of any potential solution. Let’s focus on usage reports and high-level ideas.

Allan_Taylor · March 3, 2023, 2:58pm

We use Github Actions for our CI/CD platform and I have been working recently on improving the speed of the build jobs. Prior to the improvements we were using public Github hosted Actions runners and build times alone would range from 50-75 mins, with the average being close to 58 minutes. Deployments would take another 20-25 minutes. For us there were great performance gains in moving to using self hosted runners vs the Github hosted runners.

Without any other changes our build time went from the 50-75 min range to a consistent 16-17 minutes. Deployment times didn’t improve much but improve some. A few minutes were shaved off. Since we are using Github Actions we setup the build workflows to use the Github Action caching backend. When this was tested then build time took about 20-21 minutes but it was cached. If we ran the same build job again, the build time was reduced to 2 minutes.

If any of you are using publicly hosted Github Actions runners, and feel like builds are slow, have a look at hosting the runners yourself. Our Kubernetes nodes are running on AWS r6a.large instance sizes.

https://docs.docker.com/build/cache/backends/gha/

regis · March 20, 2023, 11:42am

I propose that we start discussing candidate solutions in this issue: Can we improve Docker image build time? · Issue #6 · openedx/wg-devops · GitHub
We can then prototype solutions and measure their impact in this project: GitHub - overhangio/test-docker-build: Test Docker image building

Let’s also talk about this issue during the conference! Please come and tell me about your Tutor woes, I’ll be all ears.

Topic		Replies	Views
LONG Image Build Time Architecture lilac	1	451	June 22, 2021
Devstack creation time reduction Development	12	1105	May 5, 2020
Docker removing open source organisations - Will it affect Open edX? Site Operators	1	227	March 16, 2023
Private Open edX repo for tutor docker image Site Operators tutor	2	261	June 25, 2022
Development time Development	1	594	October 22, 2019

Optimize dependency installation

Optimize the asset build

Reduce number of layers that need to be rebuilt

Related Topics