# Horizontal Scaling - CMS/LMS and Workers **URL:** \ **Category:** Site Operators\ **Created:** [August 2, 2024, 6:30pm UTC](https://discuss.openedx.org/t/horizontal-scaling-cms-lms-and-workers/13576 "2024-08-02T18:30:40Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Zachary\_Trabookis](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/zachary_trabookis/32/753_2.png) [@Zachary\_Trabookis](https://discuss.openedx.org/u/Zachary_Trabookis)\ **Post date:** [August 2, 2024, 6:30pm UTC](https://discuss.openedx.org/t/horizontal-scaling-cms-lms-and-workers/13576/1 "2024-08-02T18:30:40Z")
I’ve been reading over tutor documentation on how to horizontally scale the Open edX system out for the CMS and LMS applications and their corresponding workers. [Running Open edX at scale — Tutor documentation (edly.io)](https://docs.tutor.edly.io/tutorials/scale.html) Looking at the `lms-worker` docker service, I noticed that the `lms` service was a requirement. ![image](https://us1.discourse-cdn.com/flex020/uploads/openedx/original/2X/7/7e9e23ba71d4d0b748ddbf3c7758da0c9554473b.png) If we were to use an application load balancer for the `lms-worker`, would the `lms` service need to run along side this docker container on the same machine? How does the `lms-worker` communicate with the `lms` and vice versa? Does it use Celery with Redis backend for this? We’re currently not using Kubernetes. cc @dave ---
**Author:** ![dave](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/dave/32/263_2.png) [@dave](https://discuss.openedx.org/u/dave)\ **Post date:** [August 3, 2024, 3:28pm UTC](https://discuss.openedx.org/t/horizontal-scaling-cms-lms-and-workers/13576/2 "2024-08-03T15:28:08Z")
> [@Zachary\_Trabookis](#): > > If we were to use an application load balancer for the `lms-worker`, would the `lms` service need to run along side this docker container on the same machine? > > How does the `lms-worker` communicate with the `lms` and vice versa? Does it use Celery with Redis backend for this? You are correct that the communication between `lms` and `lms-worker` happens via Redis as a Celery backend. Strictly from edx-platform’s point of view (i.e. not considering Tutor), there’s no need for `lms` and `lms-worker` to run on the same server. They do need the same code and similar configuration settings, but [edx.org](http://edx.org) has long run them as separate scaling groups. The `lms` servers render web requests to users and stick tasks onto Redis, and the `lms-worker` servers pick up those tasks and execute them. You shouldn’t have to add a load balancer for `lms-worker` specifically–the workers will all connect to your Redis instance and take advantage of Redis’s queue-like facilities to distribute work between them. > [@Zachary\_Trabookis](#): > > We’re currently not using Kubernetes. I know much less about Tutor, but my understanding is that you’ll need to start using k8s to actually spread your load across multiple physical servers. I may be misinterpreting this, but it sounds like you might be considering an attempt to run a separate, non-k8s Tutor instance to host your lms-workers, and have it talk to your existing Tutor instance via a shared Redis. If you are _not_ considering this please ignore the rest of this post. If you _are_ considering the above, I think it will be very painful. I guess it technically might work if you spin up a separate non-k8s Tutor instance with mostly just `lms-worker` instances and used config to point it to a shared Redis + shared MongoDB + shared file/object store (S3?) + shared MySQL + shared Elasticsearch (i.e. everything that persists data)… but I don’t know of anyone who runs it like that, and I imagine any such setup would be fragile and break in weird ways in practice because there’s going to be a bunch of non-obvious config that will need to be _just_ right (e.g. encryption/signing keys). So I think if you want to scale out up LMS workers, the practical path forward is either going to be scaling up your hardware to be a much bigger instance, or to start venturing in to Kubernetes. ---
**Author:** ![Zachary\_Trabookis](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/zachary_trabookis/32/753_2.png) [@Zachary\_Trabookis](https://discuss.openedx.org/u/Zachary_Trabookis)\ **Post date:** [August 5, 2024, 1:47pm UTC](https://discuss.openedx.org/t/horizontal-scaling-cms-lms-and-workers/13576/3 "2024-08-05T13:47:59Z")
Thanks @dave. That’s right. We were considering putting the lms-workers on a separate machine than the lms. Same for the cms-workers with separate EC2 instance. All of this on a non-k8s instances. That’s right, we were considering not adding a public application load balancer but rather a private EC2 instance(s) that could be used to offload background tasking for the lms-worker or cms-worker applications. It sounds like this could be done; however, we may be missing some configuration or need to make sure that the configuration is configured correctly. We’ll definitely consider either scaling up the LMS/CMS workers or start venturing into k8s configuration. This is what we were considering with our non-k8s install for our AWS infrastructure: **Persistent Data Storage** - Elasticache Redis - RDS Aurora MySQL - Atlas MongoDB - OpenSearch (Elasticsearch configuration) **MFEs** - Cloudfront with S3 Origin **EC2 (scale vertically)** - discovery **EC2 (load balancer with auto-scaling group)** - cms **EC2 (load balancer with auto-scaling group)** - lms **EC2 (scale vertically)** - cms-worker **EC2 (scale vertically)** - lms-worker ---
**Author:** ![dave](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/dave/32/263_2.png) [@dave](https://discuss.openedx.org/u/dave)\ **Post date:** [August 6, 2024, 6:30pm UTC](https://discuss.openedx.org/t/horizontal-scaling-cms-lms-and-workers/13576/4 "2024-08-06T18:30:50Z")
> [@Zachary\_Trabookis](#): > > It sounds like this could be done; however, we may be missing some configuration or need to make sure that the configuration is configured correctly. We’ll definitely consider either scaling up the LMS/CMS workers or start venturing into k8s configuration. Yeah. I would strongly urge you to either scale up your server to something bigger so you can spin up a bunch of extra lms-workers within one vanilla Tutor instance, or just go all the way to k8s. Even if you got multiple Tutor instances to work together with shared data stores, running in an unsupported mode like this will likely cause you weird problems when it comes time to upgrade. Some larger sites have completely customized the deployment process, and some don’t use Tutor at all. But down that path lies a whole lot of extra work and long term maintenance burden that I don’t think you want to tackle if you can help it. ---
**Author:** ![braden](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/braden/32/46_2.png) [@braden](https://discuss.openedx.org/u/braden)\ **Post date:** [August 7, 2024, 5:15pm UTC](https://discuss.openedx.org/t/horizontal-scaling-cms-lms-and-workers/13576/5 "2024-08-07T17:15:13Z")
If you do go k8s, definitely check out [Harmony](https://github.com/openedx/openedx-k8s-harmony/#readme) 🙂 It should help you get a scalable cluster working more easily. ---
**Author:** ![regis](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/regis/32/2640_2.png) [@regis](https://discuss.openedx.org/u/regis)\ **Post date:** [August 8, 2024, 6:46am UTC](https://discuss.openedx.org/t/horizontal-scaling-cms-lms-and-workers/13576/6 "2024-08-08T06:46:59Z")
I don’t remember the reason why lms-worker is indicated as dependent upon lms. This was implemented [a long time ago](https://github.com/overhangio/tutor/commit/e357e763d703a4c8ca0b3f7a79a7be675b296297). But I’m guessing that the lms-worker is making HTTP calls to the lms container. So both of them need to be on the same network. ---
**Author:** ![Zachary\_Trabookis](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/zachary_trabookis/32/753_2.png) [@Zachary\_Trabookis](https://discuss.openedx.org/u/Zachary_Trabookis)\ **Post date:** [August 8, 2024, 1:39pm UTC](https://discuss.openedx.org/t/horizontal-scaling-cms-lms-and-workers/13576/7 "2024-08-08T13:39:07Z")
> [@dave](#): > > Strictly from edx-platform’s point of view (i.e. not considering Tutor), there’s no need for `lms` and `lms-worker` to run on the same server. They do need the same code and similar configuration settings, but [edx.org](http://edx.org) has long run them as separate scaling groups. The `lms` servers render web requests to users and stick tasks onto Redis, and the `lms-worker` servers pick up those tasks and execute them. @dave I’m a little confused with where to install the `cms-workers` and `lms-workers` when we’re not running k8s. You mentioned above that edX has been running the `lms-worker` on separate server than the `lms`. > [@dave](#): > > Yeah. I would strongly urge you to either scale up your server to something bigger so you can spin up a bunch of extra lms-workers within one vanilla Tutor instance, or just go all the way to k8s. Even if you got multiple Tutor instances to work together with shared data stores, running in an unsupported mode like this will likely cause you weird problems when it comes time to upgrade. When you talk about scaling up your server to something bigger are you saying to vertically scale a `local` install of Tutor with all non-persistent services (cms, cms-worker, lms, lms-worker, …) and ignore setting up a load balancer and auto-scaling? We’re considering the following two configurations at the moment. Can you advise on where you would see weird problems occuring? (Note: both of these configurations would talk to external clustered configurations for persistent storage for MySQL, MongoDB, Redis, Elasticsearch). **First Configuration** Having a load balancer and auto-scaling group would account for additional user requests for the `cms` and `lms` applications. Both the `cms-worker` and `lms-worker` applications would run on separate EC2 instances and talk with the external Redis cluster. > [@](#): > > **EC2 (load balancer with auto-scaling group)** > > - cms > > **EC2 (load balancer with auto-scaling group)** > > - lms > > **EC2 (scale vertically)** > > - cms-worker > > **EC2 (scale vertically)** > > - lms-worker **Second Configuration** Based on your suggestions we’re scaling vertically but the `cms` and `lms` applications are separated. > [@](#): > > **EC2 (scale vertically)** > > - cms > - cms-worker > > **EC2 (scale vertically)** > > - lms > - lms-worker ---
**Author:** ![dave](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/dave/32/263_2.png) [@dave](https://discuss.openedx.org/u/dave)\ **Post date:** [August 8, 2024, 3:10pm UTC](https://discuss.openedx.org/t/horizontal-scaling-cms-lms-and-workers/13576/8 "2024-08-08T15:10:50Z")
> [@Zachary\_Trabookis](#): > > I’m a little confused with where to install the `cms-workers` and `lms-workers` when we’re not running k8s. You mentioned above that edX has been running the `lms-worker` on separate server than the `lms`. To be clear, edX/2U has its own complex deployment setup that is completely independent of Tutor. > [@Zachary\_Trabookis](#): > > When you talk about scaling up your server to something bigger are you saying to vertically scale a `local` install of Tutor with all non-persistent services (cms, cms-worker, lms, lms-worker, …) and ignore setting up a load balancer and auto-scaling? Yes. If you currently have a Tutor instance running and want to scale up, the easiest thing is to upgrade to some ridiculous server instance with 128+ CPU cores and gobs of memory, and crank up the worker count via configuration, ignoring the load balancer and auto-scaling. You don’t get proper failover, but you get easy scaling. But this is where I also had a misconception. I had thought that the number of celery worker processes could be scaled up or down in the same way that the uwsgi web worker processes are configured. But it looks like the PR that @arbrandes made for that wasn’t (and won’t be) merged: > > > This allows the user to configure how many Celery workers are spawned independen…tly of how many CPUs there are in the system. The default is to spawn as many workers as there are CPUs, which in some cases can consume too many resources. > > The setting should be particularly useful to people running Tutor for development on Linux machines, where reducing the concurrency to "1" can reduce RAM usage significantly. > > \### Testing > > 1. Before running this branch, launch a Tutor environment and count how many celery process there are. With something like: > > \`\`\`\` > pgrep celery | wc -l > \`\`\`\` > > You should get twice the number of CPUs on the system - one set for each of LMS and CMS - plus two parent processes. (On my machine, which has 12 real cores + 12 virtual ones, the number comes out to 26.) > > 3. Stop the environment, install this branch, and set: > > \`\`\`\` > tutor config save --set OPENEDX\_LMS\_CELERY\_WORKERS=1 --set OPENEDX\_CMS\_CELERY\_WORKERS=1 > \`\`\`\` > > 4. Relaunch the environment. The worker containers should be recreated. > > 5. Check the number of celery processes. There should now be just 4. > > It's worth checking RAM usage, too. Before, my dev environment would take up 8 or more gigabytes of RAM. After, it takes less than 3. So to have actual fine grained control today, I think you’d have to add a `--concurrency` param to the [lms-worker config in docker-compose.yml](https://github.com/overhangio/tutor/blob/8aed2251fdf15cea93abb89dbc63f13edced32d1/tutor/templates/local/docker-compose.yml#L161). > [@Zachary\_Trabookis](#): > > We’re considering the following two configurations at the moment. Can you advise on where you would see weird problems occuring? > (Note: both of these configurations would talk to external clustered configurations for persistent storage for MySQL, MongoDB, Redis, Elasticsearch). File storage would be one concern. Not the Studio Files and Uploads, which uses MongoDB, but a long tail of other kinds of file storage that uses django-storages backend for various media like forum post images, course block transformers collection data, etc. Another concern would be if there are places where the celery workers expect to have a locally accessible connection to the LMS that is not exposed externally. I don’t know if that’s ever the case. Another issue would be if there are any non-obvious config keys that need coordination. I _think_ these are all covered in the central tutor `config.yml` file, but I don’t know if there’s any other auto-generated stuff elsewhere in the stack. Upgrades/migrations might also be a problem down the line, if they assume there’s one active cluster with shared volumes that they can store things in, even temporarily. It is highly likely that there are other things out there that I’m not aware of. * * * Please don’t take this the wrong way, but while I am happy to try to explain how things work to the best of my understanding, I am not going to help support you step by step into an unsupported hybrid deployment mode. Tutor is the supported mode for single server deployment. Tutor with k8s is the supported mode for larger scale deployments with autoscaling, failover, and all that goodness–though I’m not familiar with any of the details here, since most of my operational experience comes from edX and this was not how we deployed things. ---
**Author:** ![Zachary\_Trabookis](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/zachary_trabookis/32/753_2.png) [@Zachary\_Trabookis](https://discuss.openedx.org/u/Zachary_Trabookis)\ **Post date:** [August 8, 2024, 6:14pm UTC](https://discuss.openedx.org/t/horizontal-scaling-cms-lms-and-workers/13576/9 "2024-08-08T18:14:57Z")
@dave You make a lot of good points about django-storages, celery workers expecting to have local accessibility to cms or lms applications, auto-generated config keys, upgrades. Maybe we should start over and look at k8s deployment since it is the recommendation for larger scale deployments with autoscaling, failover, etc. ---
**Author:** ![Zachary\_Trabookis](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/zachary_trabookis/32/753_2.png) [@Zachary\_Trabookis](https://discuss.openedx.org/u/Zachary_Trabookis)\ **Post date:** [August 8, 2024, 6:49pm UTC](https://discuss.openedx.org/t/horizontal-scaling-cms-lms-and-workers/13576/10 "2024-08-08T18:49:40Z")
> [@braden](#): > > If you do go k8s, definitely check out [Harmony](https://github.com/openedx/openedx-k8s-harmony/#readme) 🙂 It should help you get a scalable cluster working more easily. @braden @dave @regis @lpm0073 Aside from what Braden mentioned, what other documentation exists to explain how to deploy Open edX on k8s? I found these articles: - [Kubernetes deployment — Tutor documentation (edly.io)](https://docs.tutor.edly.io/k8s.html) - [Running Open edX At Scale With Kubernetes - Blog (lawrencemcdaniel.com)](https://blog.lawrencemcdaniel.com/running-open-edx-on-kubernetes/)