How to upload file in "File & Uploads" to S3

Site Operators
1

Hi all,

I’m trying to change local storage to s3.
I changed all configure, which involve storage, to S3BotoStorage in {lms, cms}.auth.json and {lms, cms}.env.json files, and it worked well for ORA component, but It doesn’t work on “cms website > Content > File & Upload”

I have 2 problems, I hope you can help me that

  • Where do the file store when I upload files on “File & Upload”?
  • How can I make s3 work well on “File & Upload”?

Thank you for help.

2

The files uploaded via File and Uploads are stored in MongoDB, and AFAIK there is currently no way to change this. They are imported and exported with the course, so they can be reliably used in course content.

Files uploaded by users on the forum, course exports, csv reports etc are stored on S3 if properly configured.

1 Like
3

I can see it. Thank you for your help.

4

You can mount S3 bucket with EC2

sudo apt install s3fs
echo ACCESS_KEY_ID:SECRET_ACCESS_KEY > ${HOME}/.passwd-s3fs
chmod 600 ${HOME}/.passwd-s3fs

s3fs mybucket ~/path/to/mountpoint -o passwd_file=${HOME}/.passwd-s3fs

On errors:

sudo s3fs mybucket ~/path/to/mountpoint -o passwd_file=${HOME}/.passwd-s3fs -o dbglevel=info -f -o curldbg

Please don’t use “.” in bucket name.

Details are available: How to Mount Amazon S3 as a Filesystem in Linux, Windows, and macOS

Please also check: What’s the maximum transfer speed between Amazon EC2 and Amazon S3?

It works, but the data transfer speed from S3 to EC2 is very slow.

5

I don’t think your solution solves the problem, since the content is stored in MongoDB and not on the filesystem.

6

Yes, you are correct. The S3 mounted bucket in EC2 as a directory is ok with static files, but not workable with MongoDB. If the intent is to separate or scalable the file system, then EFS is maybe a better choice over EBS.

7

I don’t think it solves the problem for static files either. The aim with storing on S3 is usually to also deliver the files from S3 so that static files are delivered faster. By mounting an S3 bucket, you are storing the static files on S3, but they are not delivered via S3, so serving static files will get slower.

It could be useful if you have incredibly low space on your server for some reason.