Migration of videos from Mongo to GCP

Hello everyone,

I have uploaded too many videos in files & uploads, and those videos are stored in mongodb, these many videos in mongo is creating too much load on server. so i decided to move all these videos from mongo to gcp.

To achieve this, i have followed below steps:

  1. moving all the videos stored in mongo (videos are stored in file.chunks table mongo) to gcp using shell script.

  2. replacing the mongo video urls (generated via files & uploads) by gcp public urls, as there are so many course and videos i can not do it manually (manually from studio itself edit option), i must run some script which replace mongo urls to gcp urls in mongo db, how can i do this?

  3. after that i will delete all the video files (which are stored in file.chunks in mongodb) from mongo.

So all i want to know is, where the video urls are stored in mongo (in which table) so that i can replace those urls with gcp urls?

@dave Is this something that edx-val and its APIs could help with?

It’s probably easier to export the course OLX, and then have a script that parse the video tags (the XML files in the /videos directory), replace the paths–and then import it back in.

The edx-val is another layer of machinery and probably isn’t worth getting into unless you’re doing a bunch of different encodings and managing a video pipeline.

For the load part, you can use CDN. I don’t know if it automatically replaces old links with CDN ones, otherwise, you have to replace them manually.
If openedx has an option to store videos on S3, it would be much easier to manage everything (load, tag, folder structure, file version,…).

I still not have the clear idea, if the mongo urls can be replaced to gcp urls using script then how can i do it? in mongo i have 5 collections: [“fs.chunks”, “fs.files”, “modulestore.active_versions”, “modulestore.definitions”, “modulestore.structures”]

among them, where is exactly the video urls are saved?