SignatureDoesNotMatch exception when initializing Open edX on K8s

I have installed Open edX via Tutor using nightly branch for both tutor and the minio plugin but am running into an issue initializing the stack. I am installing onto OpenShift 4.9, which is K8s v1.22.8+f34b40c.

I have run tutor k8s start, which brings up all of the pods successfully. When I run tutor k8s init, the minio and mysql jobs run successfully. (The minio buckets are created.)

However, when the lms job kicks in and starts running the migrations, there’s an uncaught exception at the first migration that involves saving a file.

It is a SignatureDoesNotMatch exception from boto (see partial trace at bottom of post).

I’ve searched quite a bit and although the error is fairly common, I haven’t found anything that really looks like it fits our situation. That said, I have tried a few things based on what’s come up in searches with no success:

  • I’ve tried to generate a new MINIO_AWS_SECRET_ACCESS_KEY/OPENEDX_AWS_SECRET_ACCESS_KEY as I’d seen reports of issues with keys including special chars.
  • I tried setting AWS_S3_ADDRESSING_STYLE=virtual". The setting seemingly just prepends the access key to the URL so that just resulted in a different error.

I also managed to set up a trace on the minio server by curling the client and authenticating using the access key and secret key in config.yml. Here’s what I see when the job runs (verbose on):

dev-storage.apps.dev-shared... [REQUEST s3.GetObject] [2022-07-29T18:27:46.817] [Client IP: 10.x.x.x]
dev-storage.apps.dev-shared... GET /openedx/badges/badges/honor.png
dev-storage.apps.dev-shared... Proto: HTTP/1.1
dev-storage.apps.dev-shared... Host: dev-storage.apps.dev-shared... 
dev-storage.apps.dev-shared... Expect: 100-continue
dev-storage.apps.dev-shared... X-Forwarded-Proto: https
dev-storage.apps.dev-shared... X-Forwarded-Port: 80
dev-storage.apps.dev-shared... Content-Length: 0
dev-storage.apps.dev-shared... Content-Type: image/png
dev-storage.apps.dev-shared... Forwarded: for=10.x.x.x;host=dev-storage.apps.dev-shared...;proto=https
dev-storage.apps.dev-shared... X-Amz-Date: 20220729T182746Z
dev-storage.apps.dev-shared... X-Forwarded-Host: dev-storage.apps.dev-shared-...
dev-storage.apps.dev-shared... X-Forwarded-For: 10.x.x.x, 10.x.x.x
dev-storage.apps.dev-shared... Accept-Encoding: identity
dev-storage.apps.dev-shared... Authorization: AWS4-HMAC-SHA256 Credential=openedx/20220729/us-east-1/s3/aws4_request, SignedHeaders=content-md5;content-type;host;x-amz-content-sha256;x-amz-date, Signature=252492109a69769e93eee76adf094eb94c37569502b92ff2ed49400ba928c2d4
dev-storage.apps.dev-shared... Content-Md5: xKZeptSRIwFI/v8jlFgYUg==
dev-storage.apps.dev-shared... User-Agent: Boto3/1.4.8 Python/3.8.12 Linux/4.18.0-305.49.1.el8_4.x86_64 Botocore/1.8.17 Resource
dev-storage.apps.dev-shared... X-Amz-Content-Sha256: da27fb3ef2c941df88055a62fda963b9df678a3bd8aec9ad63f111117d26db92
dev-storage.apps.dev-shared... <BODY>
dev-storage.apps.dev-shared... [RESPONSE] [2022-07-29T18:27:46.817] [ Duration 189µs  ↑ 203 B  ↓ 780 B ]
dev-storage.apps.dev-shared... 403 Forbidden
dev-storage.apps.dev-shared... Content-Type: application/xml
dev-storage.apps.dev-shared... No-Gzip-Compression: true
dev-storage.apps.dev-shared... Server: MinIO
dev-storage.apps.dev-shared... Strict-Transport-Security: max-age=31536000; includeSubDomains
dev-storage.apps.dev-shared... Vary: Origin,Accept-Encoding
dev-storage.apps.dev-shared... X-Content-Type-Options: nosniff
dev-storage.apps.dev-shared... Accept-Ranges: bytes
dev-storage.apps.dev-shared... Content-Length: 431
dev-storage.apps.dev-shared... X-Xss-Protection: 1; mode=block
dev-storage.apps.dev-shared... Content-Security-Policy: block-all-mixed-content
dev-storage.apps.dev-shared... X-Amz-Request-Id: 1706602ECAA3D26A
dev-storage.apps.dev-shared... <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><Key>badges/badges/honor.png</Key><BucketName>openedx</BucketName><Resource>/openedx/badges/badges/honor.png</Resource><RequestId>1706602ECAA3D26A</RequestId><HostId>61ce544a-4846-4f4b-9e24-07a3c078c22a</HostId></Error>

Hoping someone with some familiarity can help out.

======

Here’s the exception trace from the migration:

File "/openedx/edx-platform/lms/djangoapps/certificates/migrations/0003_data__default_modes.py", line 20, in forwards
    conf.icon.save(
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/fields/files.py", line 89, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/files/storage.py", line 54, in save
    name = self._save(name, content)
  File "/openedx/venv/lib/python3.8/site-packages/storages/backends/s3boto3.py", line 495, in _save
    self._save_content(obj, content, parameters=parameters)
  File "/openedx/venv/lib/python3.8/site-packages/storages/backends/s3boto3.py", line 510, in _save_content
    obj.upload_fileobj(content, ExtraArgs=put_parameters)
  File "/openedx/venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 511, in object_upload_fileobj
    return self.meta.client.upload_fileobj(
  File "/openedx/venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 431, in upload_fileobj
    return future.result()
  File "/openedx/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 73, in result
    return self._coordinator.result()
  File "/openedx/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 233, in result
    raise self._exception
  File "/openedx/venv/lib/python3.8/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/s3transfer/upload.py", line 692, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/openedx/venv/lib/python3.8/site-packages/botocore/client.py", line 317, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/botocore/client.py", line 615, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (SignatureDoesNotMatch) when calling the PutObject operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.

Hi @michaels,
First of all, it’s going to be very difficult to help you if you run on Nightly. Running on the edx-platform master branch means that it’s more difficult to reproduce your issue. So I strongly encourage you to run the latest stable version of Tutor, which is v14.1.2.

Once you have done that, also install and enable tutor-minio==14.0.0 (the latest release).

I encourage you to first try out the installation locally, with tutor local quickstart. If possible, test on a live production server with HTTPS enabled. You should not ever have to modify the default OPENEDX_AWS_* settings in order to get the minio plugin to work. If the installation does not work at this stage, then there is a bug in tutor-minio.

If running locally does work, then you should try to run the platform on Openshift again. Since you have already deployed a later version of Tutor to your Openshift cluster, downgrading is going to be difficult. I suggest you just delete the platform with tutor k8s delete. Then deploy it again with tutor k8s quickstart.

The error you are facing seems to come from the fact that the LMS cannot reach the MinIO container. You should make sure that MinIO is accessible from the Internet: run tutor config printvalue MINIO_HOST. You should be able to telnet that hostname on port 443. Same for MINIO_CONSOLE_HOST.

We should be able to troubleshoot further once we have answers to these questions.