AWS S3 file uploads setup

Site Operators Tutor Help
, , ,
1

My goal is to use AWS for all any upload coming from the site, I am okay if static files remains in openedx site. Am running on Tutor v20.0.1/Teak.2

I have tried to enable minio, though it sets DEFAULT_FILE_STORAGE to storages.backends.s3boto3.S3Boto3Storage but it seems to maintain local storage.

I had to disable it. I have do not know exactly how to get S3 storage activate as there is no clear documentation.

I tried tutor-contrib-s3 which was able to set DEFAULT_FILE_STORAGE to storages.backends.s3boto3.S3Boto3Storage yet my files are still locally stored (then I disabled it).

I have also set bunch of tutor config in an attempt to get this working all to no avail. Please I really need some help as I have been gaming around this for days now.

I can confirm that my bucket is private my I created IAM User and Role which I attached my EC2 instance to. So, my Openedx EC2 have access to my bucket and from the shell I have been able to upload file to the bucket. I believe I do not AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY but I generated one just to be sure that is not my issue

tutor config save
–set ENABLE_S3_FILE_STORAGE=true
–set AWS_S3_STORAGE_ENABLED=true
–set AWS_S3_BUCKET_NAME=“my-aws-s3-for-openedx”
–set AWS_S3_FILEUPLOAD_BUCKET_NAME=“my-aws-s3-for-openedx”
–set AWS_S3_FILEUPLOAD_PREFIX=“uploads”
–set ENABLE_S3=true
–set S3_STORAGE_BACKEND=aws
–set AWS_ACCESS_KEY_ID=“xxxxxxxxxxxxxxxxx”
–set AWS_SECRET_ACCESS_KEY=“xxxxxxxxxxxxxxxxxxxx”
–set AWS_STORAGE_BUCKET_NAME=“my-aws-s3-for-openedx”
–set AWS_S3_REGION_NAME=“eu-north-1”
–set AWS_DEFAULT_ACL=“private”
–set AWS_QUERYSTRING_AUTH=false
–set DEFAULT_FILE_STORAGE=“storages.backends.s3boto3.S3Boto3Storage”
–set AWS_S3_ADDRESSING_STYLE=“auto”
–set AWS_S3_FILE_OVERWRITE=false
–set MEDIA_URL=“https://my-aws-s3-for-openedx.s3.eu-north-1.amazonaws.com/”
–set FILE_UPLOAD_STORAGE_BACKEND=“storages.backends.s3boto3.S3Boto3Storage”
–set FILE_UPLOAD_STORAGE_BUCKET_NAME=“my-aws-s3-for-openedx”
–set FILE_UPLOAD_STORAGE_KWARGS=‘{“location”:“uploads/”}’

2

@joel.edwards, @arunmozhi Please if you have sometime, can you assign me on this?

3

Hi @pauldic
Regrettably I have no experience with this particular plugin, I assume you’ve read through the instructions in tutor-minio? I see that tutor-contrib-s3 is noted to be an experimental plugin, and is not part of the official tutor/openedx plugins, so support may be limited there…

Many of the configs you’ve listed don’t appear to be documented (at least as far as I can tell) so if you were using ChatGPT or similar to guide you then that may be at least part of your problem.

@Abdul_Rehman is listed as the maintainer of tutor-minio perhaps he may be able to provide some more guidance unless someone else gets to this first :slight_smile:

4

:rofl: “if you were using ChatGPT or similar to guide you then that may be at least part of your problem.” you got it just right :sweat_smile:. I also dig a bit further into tutor-minio as I read it could “server as a gateway to another object storage solution, such as S3 or Azure.”

By looking into it’s plugin and patches I believe these parameters are right because they where set but what am not so sure is what values are required and its expected behavior as there is no documentation. With this everything is looking good but the files are still store locally when I upload from the Studio → Content → Files → Add File.

tutor config save
–set OPENEDX_AWS_ACCESS_KEY=“xxxxxxxxxxxxxx”
–set OPENEDX_AWS_SECRET_ACCESS_KEY=“xxxxxxxxxxxxxxxxxxx”
–set MINIO_BUCKET_NAME=“my-aws-s3-for-openedx”
–set MINIO_CONSOLE_HOST=“minio.mydomain.com”
–set MINIO_GRADES_BUCKET_NAME=“my-aws-s3-for-openedx”
–set MINIO_DISCOVERY_BUCKET_NAME=“my-aws-s3-for-openedx”
–set MINIO_FILE_UPLOAD_BUCKET_NAME=“my-aws-s3-for-openedx”
–set MINIO_VIDEO_UPLOAD_BUCKET_NAME=“my-aws-s3-for-openedx”
–set MINIO_OPENEDX_LEARNING_BUCKET_NAME=“my-aws-s3-for-openedx”
–set MINIO_HOST=“s3.eu-north-1.amazonaws.com”
–set MINIO_QUERYSTRING_AUTH=true
–set MINIO_GATEWAY=“s3”

5

Hi @pauldic, I haven’t personally worked with Tutor’s S3 or the Minio plugins. However, looking at the config you have shared, I noticed that you have

--set MINIO_HOST=“s3.eu-north-1.amazonaws.com”

while the docs say

MINIO_HOST (default: "files.{{ LMS_HOST }}")

and

It is assumed that the MINIO_HOST DNS record points to your server.

MinIO being a generic way to map any cloud storage as a backend for storing files, it would be reasonable to assume using the S3 URL is wrong. Maybe that’s the issue?

Anyway, that’s all I could contribute here. I don’t have the knowledge to go further.

6

Yeah… see the ChatGPT warning here :sweat_smile: