Did I discover the reason why static files are slow in Open edX?

omar · May 1, 2021, 4:31pm

I noticed that development files are being collected despite that they shouldn’t be.

Here’s the Open edX static files process:

Individual files are collected into bundles.
Bundles are hashed and stored into /edx/var/edxapp/staticfiles.
ProductionStorage reads those hashed files e.g. common.95ef73d45fe3.js and display hashed links in the HTML files.
nginx knows how to fetch the files by their direct names.

I noticed that non-bundles are collected, which at could doubles the effort (a guesstimate):

edxapp@edxapp-server:/edx/var/edxapp/staticfiles/bundles$ ls
AboutBlockPreview.511ca7242a54.js			      EntitlementSupportPage.38bb74aa3632f57cb4a8.js			      RandomizeDescriptor.91acabaac4ba69c3d1bd.4c0d655af319.js
AboutBlockPreview.9eb5cb2eee48326c6dad.511ca7242a54.js	      EntitlementSupportPage.46e03b991dba.js				      RandomizeDescriptor.91acabaac4ba69c3d1bd.js
AboutBlockPreview.9eb5cb2eee48326c6dad.js		      EntitlementSupportPage.js						      RandomizeDescriptor.js
AboutBlockPreview.js					      EntitlementUnenrollmentFactory.3c4a97cd6fe20e4db3b3.efe44d286e7b.js     RandomizeModule.2fa73c3f470d.js
AboutBlockStudio.ab2154a1195a64fa91ea.d8fa27ea747a.js	      EntitlementUnenrollmentFactory.3c4a97cd6fe20e4db3b3.js		      RandomizeModule.a99b69ee85aced6737cf.2fa73c3f470d.js
AboutBlockStudio.ab2154a1195a64fa91ea.js		      EntitlementUnenrollmentFactory.efe44d286e7b.js			      RandomizeModule.a99b69ee85aced6737cf.js
AboutBlockStudio.d8fa27ea747a.js			      EntitlementUnenrollmentFactory.js					      RandomizeModule.js
AboutBlockStudio.js					      ErrorDescriptor.34abf6091c3fb992c825.d58276d211d9.js		      RawDescriptor.59785923ea3c66ff8381.808113d9d310.js
AlertStatusBar.0b46bcaa7e78.js				      ErrorDescriptor.34abf6091c3fb992c825.js				      RawDescriptor.59785923ea3c66ff8381.js
AlertStatusBar.8ba8b034a23585c2e705.0b46bcaa7e78.js	      ErrorDescriptor.d58276d211d9.js					      RawDescriptor.808113d9d310.js
AlertStatusBar.8ba8b034a23585c2e705.js			      ErrorDescriptor.js						      RawDescriptor.js
AlertStatusBar.js					      ErrorModule.8f5b5dc10104.js					      ReactRenderer.38f3e5ba7663cd969581.4f3dd33a415e.js
AnnotatableDescriptor.954012f84f6b.js			      ErrorModule.99686d8883c5dddcdc92.8f5b5dc10104.js			      ReactRenderer.38f3e5ba7663cd969581.js
AnnotatableDescriptor.b9e7efb3fdee506f9c7f.954012f84f6b.js    ErrorModule.99686d8883c5dddcdc92.js				      ReactRenderer.4f3dd33a415e.js
AnnotatableDescriptor.b9e7efb3fdee506f9c7f.js		      ErrorModule.js							      ReactRenderer.js
AnnotatableDescriptor.js				      HiddenDescriptor.522890abd7ec.js					      SectionDescriptor.1170c78f4147.js
AnnotatableModule.7480141091ee93ad8e90.ee84ff260c22.js	      HiddenDescriptor.995b7f41b30a1e789634.522890abd7ec.js		      SectionDescriptor.d5d50dbd2d679288eb1d.1170c78f4147.js
AnnotatableModule.7480141091ee93ad8e90.js		      HiddenDescriptor.995b7f41b30a1e789634.js				      SectionDescriptor.d5d50dbd2d679288eb1d.js
AnnotatableModule.ee84ff260c22.js			      HiddenDescriptor.js						      SectionDescriptor.js
AnnotatableModule.js					      HiddenModule.6e28f91b5543.js					      SectionModule.00f1e0824f7ca7c41d4f.1ed9844ed7c7.js
AnnouncementsView.0714f77d268b3f7f1e31.bb8dbe12a1f9.js	      HiddenModule.830864c0465f293f99ea.6e28f91b5543.js			      SectionModule.00f1e0824f7ca7c41d4f.js
AnnouncementsView.0714f77d268b3f7f1e31.js		      HiddenModule.830864c0465f293f99ea.js				      SectionModule.1ed9844ed7c7.js
AnnouncementsView.bb8dbe12a1f9.js			      HiddenModule.js							      SectionModule.js
AnnouncementsView.js					      HtmlBlockPreview.13c650c5f7b2.js					      SemanticSectionDescriptor.03bb976ebe5e.js
commons.27f5797ce8e1.js					      HtmlBlockPreview.530c6acda8c20d3bd3cc.13c650c5f7b2.js		      SemanticSectionDescriptor.fc4dd502aff49f5cb9a1.03bb976ebe5e.js
commons.c8a02edff6e3aa4f0463.045f0350cff4.js		      HtmlBlockPreview.530c6acda8c20d3bd3cc.js				      SemanticSectionDescriptor.fc4dd502aff49f5cb9a1.js
commons.c8a02edff6e3aa4f0463.js				      HtmlBlockPreview.js						      SemanticSectionDescriptor.js
commons.js						      HtmlBlockStudio.4f554a74a3c7.js					      SequenceDescriptor.6ef5f98a9fa7052d1862.84785bbc4c88.js
CompletionOnViewService.3123df84c61fcd122877.925804e258db.js  HtmlBlockStudio.995ef73d45fe338e3611.4f554a74a3c7.js		      SequenceDescriptor.6ef5f98a9fa7052d1862.js

I don’t know what’s the best way to resolve this or if it’s worth it. But I wanted to share it anyway.

regis · May 2, 2021, 8:24am

I’m very, very interested in learning more! Did you do some profiling (machete-style) to find out what’s the impact of generating the dev bundles?

regis · May 2, 2021, 8:40am

In Tutor, the non-hashed files are symlinks to the hashed files:

$ docker run --rm -it overhangio/openedx:11.2.7 bash
$ ls -l /openedx/staticfiles/bundles/AboutBlockPreview.js
lrwxrwxrwx 1 root root 62 Apr 23 09:49 /openedx/staticfiles/bundles/AboutBlockPreview.js -> /openedx/staticfiles/bundles/AboutBlockPreview.511ca7242a54.js

So generating these non-hashed files should not take too long.

regis · May 2, 2021, 8:43am

Wait no: this is almost certainly because we run rdfind as part of the Docker image building to reduce the image size: tutor/Dockerfile at 452cd229f3e0577bab1aaf33d642a1302d7ada19 · overhangio/tutor · GitHub

regis · May 2, 2021, 9:50am

On my machine, static asset collection takes 57s to run. Here are the profiling stats, sorted by decreasing cumulative time, for the collectstatic command:

$ python -m cProfile -o static.prof ./manage.py lms --settings=tutor.production collectstatic --ignore "fixtures" --ignore "karma_*.js" --ignore "spec" --ignore "spec_helpers" --ignore "spec-helpers" --ignore "xmodule_js" --ignore "geoip" --ignore "sass" --noinput

I ran the following script in the static.prof file:

import pstats
import io
s = io.StringIO()
ps = pstats.Stats("static.prof", stream=s).sort_stats(pstats.SortKey.CUMULATIVE)
s.seek(0)
open("static.prof.stats", "w).write(s.read())

Which yielded the following static.prof.stats file:

Sun May  2 08:49:09 2021    static.prof

         12907115 function calls (12671003 primitive calls) in 57.219 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   5248/1    0.040    0.000   57.236   57.236 {built-in method builtins.exec}
        1    0.000    0.000   57.236   57.236 ./manage.py:2(<module>)
        1    0.000    0.000   51.817   51.817 /openedx/venv/lib/python3.8/site-packages/django/core/management/__init__.py:378(execute_from_command_line)
        1    0.000    0.000   51.817   51.817 /openedx/venv/lib/python3.8/site-packages/django/core/management/__init__.py:301(execute)
        1    0.000    0.000   51.787   51.787 /openedx/venv/lib/python3.8/site-packages/django/core/management/base.py:306(run_from_argv)
        1    0.000    0.000   51.786   51.786 /openedx/venv/lib/python3.8/site-packages/django/core/management/base.py:342(execute)
        1    0.000    0.000   51.785   51.785 /openedx/venv/lib/python3.8/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py:148(handle)
        1    0.024    0.024   51.779   51.779 /openedx/venv/lib/python3.8/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py:86(collect)
     5958    0.586    0.000   49.943    0.008 /openedx/venv/lib/python3.8/site-packages/require/storage.py:74(post_process)
        1    0.000    0.000   38.758   38.758 /openedx/venv/lib/python3.8/site-packages/require/storage.py:35(run_optimizer)
        1    0.000    0.000   38.757   38.757 /opt/pyenv/versions/3.8.6/lib/python3.8/subprocess.py:332(call)
      120    0.001    0.000   38.750    0.323 /opt/pyenv/versions/3.8.6/lib/python3.8/subprocess.py:1074(wait)
      120    0.001    0.000   38.750    0.323 /opt/pyenv/versions/3.8.6/lib/python3.8/subprocess.py:1772(_wait)
       80    0.000    0.000   38.748    0.484 /opt/pyenv/versions/3.8.6/lib/python3.8/subprocess.py:1759(_try_wait)
       80   38.748    0.484   38.748    0.484 {built-in method posix.waitpid}
     5933    0.003    0.000    8.375    0.001 ./openedx/core/djangoapps/theming/storage.py:265(post_process)
     5914    0.004    0.000    8.284    0.001 /openedx/venv/lib/python3.8/site-packages/pipeline/storage.py:20(post_process)
5828/1157    0.050    0.000    5.429    0.005 <frozen importlib._bootstrap>:986(_find_and_load)
 5501/911    0.020    0.000    5.409    0.006 <frozen importlib._bootstrap>:956(_find_and_load_unlocked)
 4609/517    0.019    0.000    5.336    0.010 <frozen importlib._bootstrap>:650(_load_unlocked)
 4432/514    0.011    0.000    5.314    0.010 <frozen importlib._bootstrap_external>:777(exec_module)
 5757/481    0.003    0.000    5.295    0.011 <frozen importlib._bootstrap>:211(_call_with_frames_removed)
       57    0.001    0.000    4.726    0.083 /openedx/venv/lib/python3.8/site-packages/pipeline/packager.py:107(pack)
       19    0.000    0.000    4.683    0.246 /openedx/venv/lib/python3.8/site-packages/pipeline/packager.py:121(pack_javascripts)
       19    0.000    0.000    4.577    0.241 /openedx/venv/lib/python3.8/site-packages/pipeline/compressors/__init__.py:55(compress_js)
       19    0.001    0.000    4.548    0.239 /openedx/venv/lib/python3.8/site-packages/pipeline/compressors/uglifyjs.py:6(compress_js)
       19    0.001    0.000    4.544    0.239 /openedx/venv/lib/python3.8/site-packages/pipeline/compressors/__init__.py:235(execute_command)
       39    0.001    0.000    4.348    0.111 /opt/pyenv/versions/3.8.6/lib/python3.8/subprocess.py:980(communicate)
       38    0.007    0.000    4.347    0.114 /opt/pyenv/versions/3.8.6/lib/python3.8/subprocess.py:1813(_communicate)
     1198    0.003    0.000    4.325    0.004 /opt/pyenv/versions/3.8.6/lib/python3.8/selectors.py:402(select)
     1198    4.321    0.004    4.321    0.004 {method 'poll' of 'select.poll' objects}
        2    0.000    0.000    4.313    2.156 /openedx/venv/lib/python3.8/site-packages/django/__init__.py:8(setup)
        1    0.000    0.000    4.305    4.305 ./lms/startup.py:13(run)
        2    0.001    0.001    4.302    2.151 /openedx/venv/lib/python3.8/site-packages/django/apps/registry.py:61(populate)
1367/1073    0.002    0.000    4.198    0.004 /opt/pyenv/versions/3.8.6/lib/python3.8/importlib/__init__.py:109(import_module)
1512/1073    0.001    0.000    4.196    0.004 <frozen importlib._bootstrap>:1002(_gcd_import)
 1996/992    0.005    0.000    3.599    0.004 {built-in method builtins.__import__}
     5876    0.003    0.000    3.547    0.001 /openedx/venv/lib/python3.8/site-packages/django/contrib/staticfiles/storage.py:401(post_process)
     5876    0.003    0.000    3.537    0.001 /openedx/venv/lib/python3.8/site-packages/django/contrib/staticfiles/storage.py:209(post_process)
     5878    0.085    0.000    3.520    0.001 /openedx/venv/lib/python3.8/site-packages/django/contrib/staticfiles/storage.py:257(_post_process)
8148/4881    0.009    0.000    2.806    0.001 <frozen importlib._bootstrap>:1017(_handle_fromlist)
      179    0.001    0.000    1.913    0.011 /openedx/venv/lib/python3.8/site-packages/django/apps/config.py:204(import_models)
     3288    0.752    0.000    1.703    0.001 {method 'sub' of 're.Pattern' objects}
      179    0.001    0.000    1.418    0.008 /openedx/venv/lib/python3.8/site-packages/django/apps/config.py:81(create)
11145/10332    0.142    0.000    1.130    0.000 {built-in method builtins.__build_class__}
    49526    0.030    0.000    0.981    0.000 /openedx/venv/lib/python3.8/site-packages/django/core/files/storage.py:322(path)
       39    0.000    0.000    0.968    0.025 /openedx/venv/lib/python3.8/site-packages/Cryptodome/Util/_raw_api.py:86(load_lib)
    49952    0.105    0.000    0.944    0.000 /openedx/venv/lib/python3.8/site-packages/django/utils/_os.py:24(safe_join)
       19    0.001    0.000    0.936    0.049 /openedx/venv/lib/python3.8/site-packages/Cryptodome/Util/_raw_api.py:278(load_pycryptodome_raw_lib)
     5880    0.030    0.000    0.915    0.000 ./openedx/core/djangoapps/theming/storage.py:189(converter)
     5451    0.004    0.000    0.896    0.000 /openedx/venv/lib/python3.8/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py:334(copy_file)
     5451    0.248    0.000    0.892    0.000 /openedx/venv/lib/python3.8/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py:245(delete_file)
     6579    0.014    0.000    0.838    0.000 ./openedx/core/storage.py:22(hashed_name)
       39    0.001    0.000    0.836    0.021 /openedx/venv/lib/python3.8/site-packages/cffi/api.py:137(dlopen)
       21    0.002    0.000    0.836    0.040 /opt/pyenv/versions/3.8.6/lib/python3.8/ctypes/util.py:309(find_library)
       39    0.001    0.000    0.835    0.021 /openedx/venv/lib/python3.8/site-packages/cffi/api.py:830(_make_ffi_library)
       39    0.001    0.000    0.834    0.021 /openedx/venv/lib/python3.8/site-packages/cffi/api.py:804(_load_backend_lib)
     6579    0.048    0.000    0.823    0.000 /openedx/venv/lib/python3.8/site-packages/django/contrib/staticfiles/storage.py:84(hashed_name)
       80    0.004    0.000    0.822    0.010 /opt/pyenv/versions/3.8.6/lib/python3.8/subprocess.py:732(__init__)
       79    0.000    0.000    0.821    0.010 /openedx/venv/lib/python3.8/site-packages/pkg_resources/__init__.py:2445(resolve)
     4842    0.009    0.000    0.817    0.000 ./openedx/core/djangoapps/theming/storage.py:169(_url)
       80    0.010    0.000    0.814    0.010 /opt/pyenv/versions/3.8.6/lib/python3.8/subprocess.py:1550(_execute_child)
        1    0.000    0.000    0.767    0.767 /openedx/venv/lib/python3.8/site-packages/require/storage.py:57(__exit__)
        2    0.000    0.000    0.767    0.384 /opt/pyenv/versions/3.8.6/lib/python3.8/shutil.py:679(rmtree)
   2752/2    0.115    0.000    0.767    0.383 /opt/pyenv/versions/3.8.6/lib/python3.8/shutil.py:622(_rmtree_safe_fd)
    20146    0.760    0.000    0.760    0.000 {method 'update' of '_hashlib.HASH' objects}
        2    0.000    0.000    0.759    0.379 /openedx/venv/lib/python3.8/site-packages/stevedore/extension.py:92(__init__)
        2    0.000    0.000    0.759    0.379 /openedx/venv/lib/python3.8/site-packages/stevedore/extension.py:185(_load_plugins)
       28    0.000    0.000    0.738    0.026 /openedx/venv/lib/python3.8/site-packages/stevedore/extension.py:216(_load_one_plugin)
        1    0.000    0.000    0.696    0.696 ./lms/djangoapps/bulk_email/models.py:1(<module>)
   102380    0.081    0.000    0.693    0.000 /opt/pyenv/versions/3.8.6/lib/python3.8/posixpath.py:372(abspath)
        1    0.000    0.000    0.688    0.688 ./openedx/core/djangoapps/course_groups/cohorts.py:1(<module>)
        1    0.000    0.000    0.687    0.687 ./lms/djangoapps/courseware/courses.py:1(<module>)
        1    0.000    0.000    0.675    0.675 /openedx/venv/lib/python3.8/site-packages/edx_proctoring/apps.py:113(ready)
        1    0.000    0.000    0.670    0.670 /openedx/venv/lib/python3.8/site-packages/edx_proctoring/backends/software_secure.py:1(<module>)
     4432    0.039    0.000    0.642    0.000 <frozen importlib._bootstrap_external>:849(get_code)
    31752    0.033    0.000    0.626    0.000 /openedx/venv/lib/python3.8/site-packages/django/contrib/staticfiles/storage.py:42(path)
        1    0.000    0.000    0.618    0.618 /openedx/venv/lib/python3.8/site-packages/Cryptodome/Cipher/__init__.py:25(<module>)
     6471    0.021    0.000    0.607    0.000 /openedx/venv/lib/python3.8/site-packages/django/contrib/staticfiles/storage.py:73(file_hash)
        1    0.000    0.000    0.601    0.601 /openedx/venv/src/edx-jsme/edx_jsme/__init__.py:1(<module>)
     2682    0.002    0.000    0.587    0.000 /openedx/venv/lib/python3.8/site-packages/django/conf/__init__.py:76(__getattr__)
        1    0.000    0.000    0.583    0.583 ./lms/startup.py:1(<module>)
        1    0.000    0.000    0.583    0.583 /openedx/venv/lib/python3.8/site-packages/django/conf/__init__.py:51(_setup)
        1    0.000    0.000    0.583    0.583 /openedx/venv/lib/python3.8/site-packages/django/conf/__init__.py:148(__init__)
        1    0.000    0.000    0.580    0.580 ./lms/envs/tutor/production.py:2(<module>)
        1    0.000    0.000    0.580    0.580 ./lms/envs/production.py:1(<module>)
      895    0.001    0.000    0.540    0.001 /openedx/venv/lib/python3.8/site-packages/django/contrib/staticfiles/finders.py:281(get_finders)
        5    0.000    0.000    0.539    0.108 /openedx/venv/lib/python3.8/site-packages/django/contrib/staticfiles/finders.py:286(get_finder)
    26176    0.521    0.000    0.528    0.000 {built-in method io.open}
    18057    0.021    0.000    0.522    0.000 /openedx/venv/lib/python3.8/site-packages/django/core/files/storage.py:309(exists)
        1    0.000    0.000    0.521    0.521 ./openedx/core/lib/xblock_pipeline/finder.py:115(__init__)
        1    0.000    0.000    0.521    0.521 ./openedx/core/lib/xblock_pipeline/finder.py:128(<setcomp>)
       52    0.000    0.000    0.521    0.010 /openedx/venv/lib/python3.8/site-packages/xblock/plugin.py:119(load_classes)
       51    0.000    0.000    0.520    0.010 /openedx/venv/lib/python3.8/site-packages/xblock/plugin.py:64(_load_class_entry_point)
       51    0.000    0.000    0.520    0.010 /openedx/venv/lib/python3.8/site-packages/pkg_resources/__init__.py:2430(load)
   107798    0.349    0.000    0.515    0.000 /opt/pyenv/versions/3.8.6/lib/python3.8/posixpath.py:334(normpath)
        1    0.000    0.000    0.498    0.498 /openedx/edx-platform/common/lib/capa/capa/inputtypes.py:5(<module>)
    11989    0.007    0.000    0.495    0.000 /openedx/venv/lib/python3.8/site-packages/django/core/files/storage.py:34(open)
    11989    0.019    0.000    0.488    0.000 /openedx/venv/lib/python3.8/site-packages/django/core/files/storage.py:223(_open)
        1    0.049    0.049    0.484    0.484 /openedx/venv/lib/python3.8/site-packages/chem/chemcalc.py:1(<module>)
     4842    0.020    0.000    0.477    0.000 /openedx/venv/lib/python3.8/site-packages/django/contrib/staticfiles/storage.py:118(_url)
    20461    0.472    0.000    0.472    0.000 {method 'read' of '_io.BufferedReader' objects}
     9054    0.019    0.000    0.468    0.000 /opt/pyenv/versions/3.8.6/lib/python3.8/re.py:289(_compile)
     5840    0.053    0.000    0.452    0.000 <frozen importlib._bootstrap>:890(_find_spec)
    10915    0.443    0.000    0.443    0.000 {built-in method posix.unlink}
  460/409    0.020    0.000    0.441    0.001 /openedx/venv/lib/python3.8/site-packages/django/db/models/base.py:69(__new__)
       51    0.001    0.000    0.436    0.009 /openedx/venv/lib/python3.8/site-packages/pkg_resources/__init__.py:2455(require)
     1077    0.006    0.000    0.435    0.000 /opt/pyenv/versions/3.8.6/lib/python3.8/sre_compile.py:759(compile)
       52    0.006    0.000    0.431    0.008 /openedx/venv/lib/python3.8/site-packages/pkg_resources/__init__.py:715(resolve)
        1    0.000    0.000    0.427    0.427 /openedx/venv/lib/python3.8/site-packages/nltk/__init__.py:9(<module>)
      258    0.426    0.002    0.426    0.002 {built-in method posix.read}
        1    0.000    0.000    0.407    0.407 ./lms/djangoapps/courseware/access.py:1(<module>)
     1771    0.001    0.000    0.405    0.000 /opt/pyenv/versions/3.8.6/lib/python3.8/re.py:250(compile)
    77109    0.399    0.000    0.399    0.000 {built-in method posix.stat}
      905    0.001    0.000    0.396    0.000 /openedx/venv/lib/python3.8/site-packages/pkg_resources/__init__.py:2734(requires)
        1    0.000    0.000    0.394    0.394 ./openedx/features/course_duration_limits/access.py:1(<module>)
      828    0.001    0.000    0.387    0.000 /openedx/venv/lib/python3.8/site-packages/pkg_resources/__init__.py:3018(_dep_map)
       89    0.001    0.000    0.386    0.004 /openedx/venv/lib/python3.8/site-packages/pkg_resources/__init__.py:3026(_compute_dependencies)
        1    0.000    0.000    0.379    0.379 ./lms/djangoapps/courseware/utils.py:1(<module>)
        1    0.000    0.000    0.378    0.378 ./lms/djangoapps/commerce/utils.py:1(<module>)
        1    0.000    0.000    0.375    0.375 ./openedx/core/djangoapps/commerce/utils.py:1(<module>)
        1    0.000    0.000    0.372    0.372 ./openedx/core/djangoapps/oauth_dispatch/jwt.py:1(<module>)
        1    0.000    0.000    0.371    0.371 /openedx/venv/lib/python3.8/site-packages/edx_rbac/utils.py:1(<module>)
        1    0.000    0.000    0.370    0.370 /openedx/venv/lib/python3.8/site-packages/edx_rest_framework_extensions/auth/jwt/authentication.py:1(<module>)
        1    0.000    0.000    0.368    0.368 /openedx/venv/lib/python3.8/site-packages/edx_rest_framework_extensions/auth/jwt/decoder.py:1(<module>)
27909/5766    0.020    0.000    0.363    0.000 /openedx/venv/lib/python3.8/site-packages/django/contrib/staticfiles/utils.py:16(get_files)
...

We can see that most of the time is spent in a single subprocess.call. To understand what command is using most of the time, I modified the subprocess.call function to print its arguments. Here they are:

['node', '/openedx/venv/lib/python3.8/site-packages/require/resources/r.js', '-o', '/tmp/tmpo_wj6w66/lms/js/build.js', 'dir=/tmp/tmp482vi18h', 'appDir=/tmp/tmpo_wj6w66', 'baseUrl=./'],)'

38.7 out of 57 seconds are spent running that r.js script, which comes from here: GitHub - requirejs/r.js: Runs RequireJS in Node and Rhino, and used to run the RequireJS optimizer

r.js is being called by the OptimizedFilesMixin static storage backend, from django-require. We use this backend because openedx.core.storage.ProductionStorage inherits from OptimizedFilesMixin and we have STATICFILES_STORAGE = 'openedx.core.storage.ProductionStorage'.

I have no idea how to further improve this call to r.js. There are some recommendations in the require.js docs but I fail to see how they apply to edx-platform: RequireJS Optimizer

omar · May 3, 2021, 11:33am

Thanks @regis!

I didn’t do any profiling and I edited the post to mention that I was estimating. I shared this post because I was surprised to see the amount of files that will never be used in production. I was hoping for a quick win, but it turns out that there isn’t any.

57 seconds? Why I remember it being 20 minutes?

Anything under one minute seems to be very good for such a huge code base.

I’m still on Hawthorn, so perhaps I’m missing some more recent optimizations.

Topic		Replies	Views
Failed - make dev.pull.large-and-slow Development	0	224	October 15, 2023
Breaking apart edx-platform's `./common/lib` folder Architecture nutmeg , olive	0	561	June 21, 2022
Server Static HTML File through Azure Storage / s3 Bucket without public access Architecture	0	375	August 21, 2021
Open EdX changes extension of files uploaded to ORA Development	0	422	December 14, 2020
Issue with Devstack version of Open edX Development how-to	2	460	December 23, 2019

Did I discover the reason why static files are slow in Open edX?

Related Topics