Reduce MongoDB size

Site Operators Site Operations Help
1

Hi all,

I’ve seen that in MongoDB the CMS stores not only the current version of the courses but all the history, since the course creation. This causes the Mongo DB to grow continuously, raising up to hundreds of gigas and more for long lasting instances.

Is there any safe method to reduce the size of the Mongo DB, without losing current course structures and not breaking the installation?

1 Like
2

In the tubular project:

There is a python script called structures.py that I have used to shrink the mongo db.

Since this is something that I intend to run frequently, I built a docker image for this project:

FROM python:3.8-buster

RUN git clone https://github.com/openedx/tubular.git
WORKDIR /tubular
RUN pip install -e .[dev]
WORKDIR /tubular/tubular/scripts

I ran the docker container on the same host that is running our production tutor images:

docker run -it --rm --net=tutor_local_default openedx-tubular bash

I first ran:

python3 structures.py --connection mongodb://mongodb:27017/ --database-name openedx make_plan PLAN_NAME

to make the pruning file PLAN_NAME. Then executed the plan using the following:

python3 structures.py --connection mongodb://mongodb:27017/ --database-name openedx prune PLAN_NAME

The last step is to compact the database. In the mongodb container I ran:

> use openedx
switched to db openedx
> db.getCollectionNames().forEach(function (collectionName) {
...     print('Compacting: ' + collectionName);
...     db.runCommand({ compact: collectionName });
... });

This step took quite a long time to complete. However, it did shrink my mongodb by more than 30%.

Obvisouly, make sure you have ALL your backups before you attempt this.

If there is a better method for compacting the mongodb, I would love to know it!

2 Likes
3

@Andres.Aulasneo: You might want to hold off on this a bit if you’re running Nutmeg, as there’s a recently investigated peculiarity around the MongoDB/MySQL switchover that may make this unsafe (@braden: do you remember the status of this?)

1 Like
4

Do we know by which release MySQL will replace Mongo?

5

No firm target. Definitely not something that’s going to happen in the Olive timeframe. We’re currently ramping up work on killing the Old Mongo Modulestore, which stopped being the default sometime back in 2013 or 2014, and which we had planned to kill way back in Koa.

1 Like
6

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.