We’re seeing an on-and-off 403 Forbidden error from the SCORM XBlock’s storage check, which crashes course outline pages for users. Hoping someone has hit this before.
Environment:
tutor 17(quince)
Self-hosted MinIO (S3-compatible storage) behind Caddy as reverse proxy
openedx-scorm-xblock for SCORM content
The error:
File ".../openedxscorm/scormxblock.py", line 385, in index_page_url
if self.storage.exists(
File ".../storages/backends/s3.py", line 479, in exists
self.connection.meta.client.head_object(Bucket=self.bucket_name, Key=name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
It’s triggered when a course’s BlockStructure cache is cold (log line: BlockStructure: Not found in cache), forcing a full block-tree rebuild. That calls student_view_data() on every block, including SCORM, which proactively checks storage.exists(). Since django-storages re-raises any non-404 ClientError, the 403 becomes an uncaught exception → 500 on /api/course_home/outline/....
Does anyone know about this issue and how we can resolved this ?
I checked on the Caddy side, but I’m not sure what I need to look at. Could you help me?
thing is, when we restart our LMS and CMS, the error disappears, and everything works perfectly. But then, after a few days, we start getting this issue again.
It seems a little difficult to reproduce this issue with just the setup information you have provided but the useful clue is that a restart fixes it for a few days. That usually means the keys the LMS uses to reach MinIO are going stale. The LMS reads those keys once when it starts and reuses them the whole time it’s running. If the keys change or expire on the MinIO side after a few days, the LMS keeps sending the old ones, and MinIO answers 403 (forbidden) instead of “file not found”. Restarting makes it read fresh keys, so it works again for a while.
There’s a second possible cause: if that storage user is allowed to read files but not allowed to list the bucket, then asking about a file that isn’t there comes back as 403 instead of a clean “not found”. But that wouldn’t be fixed by a restart, so your restart clue points more at the keys.
Easy way to tell which one it is. Open an LMS shell:
tutor local exec lms ./manage.py lms shell
Run these two lines:
from django.core.files.storage import default_storage as s
s.exists("scorm/<a-file-you-know-exists>")
s.exists("scorm/<a-file-you-know-does-NOT-exist>")
The first should return True, the second should return False. Do this now while everything works and note what you get, then run the exact same two lines next time it breaks and compare:
If the missing file gives a 403 even now while it’s healthy, it’s the “not allowed to list the bucket” case. Give that user list permission on the bucket and it stops.
If the existing file is fine now but starts giving 403 only when it’s broken, it’s stale keys. Check whether your MinIO keys rotate or expire, and make them stable.