I have installed Open edX via Tutor using nightly branch for both tutor and the minio plugin but am running into an issue initializing the stack. I am installing onto OpenShift 4.9, which is K8s v1.22.8+f34b40c.
I have run tutor k8s start, which brings up all of the pods successfully. When I run tutor k8s init, the minio and mysql jobs run successfully. (The minio buckets are created.)
However, when the lms job kicks in and starts running the migrations, there’s an uncaught exception at the first migration that involves saving a file.
It is a SignatureDoesNotMatch exception from boto (see partial trace at bottom of post).
I’ve searched quite a bit and although the error is fairly common, I haven’t found anything that really looks like it fits our situation. That said, I have tried a few things based on what’s come up in searches with no success:
- I’ve tried to generate a new
MINIO_AWS_SECRET_ACCESS_KEY/OPENEDX_AWS_SECRET_ACCESS_KEYas I’d seen reports of issues with keys including special chars. - I tried setting
AWS_S3_ADDRESSING_STYLE=virtual". The setting seemingly just prepends the access key to the URL so that just resulted in a different error.
I also managed to set up a trace on the minio server by curling the client and authenticating using the access key and secret key in config.yml. Here’s what I see when the job runs (verbose on):
dev-storage.apps.dev-shared... [REQUEST s3.GetObject] [2022-07-29T18:27:46.817] [Client IP: 10.x.x.x]
dev-storage.apps.dev-shared... GET /openedx/badges/badges/honor.png
dev-storage.apps.dev-shared... Proto: HTTP/1.1
dev-storage.apps.dev-shared... Host: dev-storage.apps.dev-shared...
dev-storage.apps.dev-shared... Expect: 100-continue
dev-storage.apps.dev-shared... X-Forwarded-Proto: https
dev-storage.apps.dev-shared... X-Forwarded-Port: 80
dev-storage.apps.dev-shared... Content-Length: 0
dev-storage.apps.dev-shared... Content-Type: image/png
dev-storage.apps.dev-shared... Forwarded: for=10.x.x.x;host=dev-storage.apps.dev-shared...;proto=https
dev-storage.apps.dev-shared... X-Amz-Date: 20220729T182746Z
dev-storage.apps.dev-shared... X-Forwarded-Host: dev-storage.apps.dev-shared-...
dev-storage.apps.dev-shared... X-Forwarded-For: 10.x.x.x, 10.x.x.x
dev-storage.apps.dev-shared... Accept-Encoding: identity
dev-storage.apps.dev-shared... Authorization: AWS4-HMAC-SHA256 Credential=openedx/20220729/us-east-1/s3/aws4_request, SignedHeaders=content-md5;content-type;host;x-amz-content-sha256;x-amz-date, Signature=252492109a69769e93eee76adf094eb94c37569502b92ff2ed49400ba928c2d4
dev-storage.apps.dev-shared... Content-Md5: xKZeptSRIwFI/v8jlFgYUg==
dev-storage.apps.dev-shared... User-Agent: Boto3/1.4.8 Python/3.8.12 Linux/4.18.0-305.49.1.el8_4.x86_64 Botocore/1.8.17 Resource
dev-storage.apps.dev-shared... X-Amz-Content-Sha256: da27fb3ef2c941df88055a62fda963b9df678a3bd8aec9ad63f111117d26db92
dev-storage.apps.dev-shared... <BODY>
dev-storage.apps.dev-shared... [RESPONSE] [2022-07-29T18:27:46.817] [ Duration 189µs ↑ 203 B ↓ 780 B ]
dev-storage.apps.dev-shared... 403 Forbidden
dev-storage.apps.dev-shared... Content-Type: application/xml
dev-storage.apps.dev-shared... No-Gzip-Compression: true
dev-storage.apps.dev-shared... Server: MinIO
dev-storage.apps.dev-shared... Strict-Transport-Security: max-age=31536000; includeSubDomains
dev-storage.apps.dev-shared... Vary: Origin,Accept-Encoding
dev-storage.apps.dev-shared... X-Content-Type-Options: nosniff
dev-storage.apps.dev-shared... Accept-Ranges: bytes
dev-storage.apps.dev-shared... Content-Length: 431
dev-storage.apps.dev-shared... X-Xss-Protection: 1; mode=block
dev-storage.apps.dev-shared... Content-Security-Policy: block-all-mixed-content
dev-storage.apps.dev-shared... X-Amz-Request-Id: 1706602ECAA3D26A
dev-storage.apps.dev-shared... <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><Key>badges/badges/honor.png</Key><BucketName>openedx</BucketName><Resource>/openedx/badges/badges/honor.png</Resource><RequestId>1706602ECAA3D26A</RequestId><HostId>61ce544a-4846-4f4b-9e24-07a3c078c22a</HostId></Error>
Hoping someone with some familiarity can help out.
======
Here’s the exception trace from the migration:
File "/openedx/edx-platform/lms/djangoapps/certificates/migrations/0003_data__default_modes.py", line 20, in forwards
conf.icon.save(
File "/openedx/venv/lib/python3.8/site-packages/django/db/models/fields/files.py", line 89, in save
self.name = self.storage.save(name, content, max_length=self.field.max_length)
File "/openedx/venv/lib/python3.8/site-packages/django/core/files/storage.py", line 54, in save
name = self._save(name, content)
File "/openedx/venv/lib/python3.8/site-packages/storages/backends/s3boto3.py", line 495, in _save
self._save_content(obj, content, parameters=parameters)
File "/openedx/venv/lib/python3.8/site-packages/storages/backends/s3boto3.py", line 510, in _save_content
obj.upload_fileobj(content, ExtraArgs=put_parameters)
File "/openedx/venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 511, in object_upload_fileobj
return self.meta.client.upload_fileobj(
File "/openedx/venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 431, in upload_fileobj
return future.result()
File "/openedx/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 73, in result
return self._coordinator.result()
File "/openedx/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 233, in result
raise self._exception
File "/openedx/venv/lib/python3.8/site-packages/s3transfer/tasks.py", line 126, in __call__
return self._execute_main(kwargs)
File "/openedx/venv/lib/python3.8/site-packages/s3transfer/tasks.py", line 150, in _execute_main
return_value = self._main(**kwargs)
File "/openedx/venv/lib/python3.8/site-packages/s3transfer/upload.py", line 692, in _main
client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
File "/openedx/venv/lib/python3.8/site-packages/botocore/client.py", line 317, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/openedx/venv/lib/python3.8/site-packages/botocore/client.py", line 615, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (SignatureDoesNotMatch) when calling the PutObject operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.