SignatureDoesNotMatch exception when initializing Open edX on K8s

I have installed Open edX via Tutor using nightly branch for both tutor and the minio plugin but am running into an issue initializing the stack. I am installing onto OpenShift 4.9, which is K8s v1.22.8+f34b40c.

I have run tutor k8s start, which brings up all of the pods successfully. When I run tutor k8s init, the minio and mysql jobs run successfully. (The minio buckets are created.)

However, when the lms job kicks in and starts running the migrations, there’s an uncaught exception at the first migration that involves saving a file.

It is a SignatureDoesNotMatch exception from boto (see partial trace at bottom of post).

I’ve searched quite a bit and although the error is fairly common, I haven’t found anything that really looks like it fits our situation. That said, I have tried a few things based on what’s come up in searches with no success:

  • I’ve tried to generate a new MINIO_AWS_SECRET_ACCESS_KEY/OPENEDX_AWS_SECRET_ACCESS_KEY as I’d seen reports of issues with keys including special chars.
  • I tried setting AWS_S3_ADDRESSING_STYLE=virtual". The setting seemingly just prepends the access key to the URL so that just resulted in a different error.

I also managed to set up a trace on the minio server by curling the client and authenticating using the access key and secret key in config.yml. Here’s what I see when the job runs (verbose on): [REQUEST s3.GetObject] [2022-07-29T18:27:46.817] [Client IP: 10.x.x.x] GET /openedx/badges/badges/honor.png Proto: HTTP/1.1 Host: Expect: 100-continue X-Forwarded-Proto: https X-Forwarded-Port: 80 Content-Length: 0 Content-Type: image/png Forwarded: for=10.x.x.x;;proto=https X-Amz-Date: 20220729T182746Z X-Forwarded-Host: X-Forwarded-For: 10.x.x.x, 10.x.x.x Accept-Encoding: identity Authorization: AWS4-HMAC-SHA256 Credential=openedx/20220729/us-east-1/s3/aws4_request, SignedHeaders=content-md5;content-type;host;x-amz-content-sha256;x-amz-date, Signature=252492109a69769e93eee76adf094eb94c37569502b92ff2ed49400ba928c2d4 Content-Md5: xKZeptSRIwFI/v8jlFgYUg== User-Agent: Boto3/1.4.8 Python/3.8.12 Linux/4.18.0-305.49.1.el8_4.x86_64 Botocore/1.8.17 Resource X-Amz-Content-Sha256: da27fb3ef2c941df88055a62fda963b9df678a3bd8aec9ad63f111117d26db92 <BODY> [RESPONSE] [2022-07-29T18:27:46.817] [ Duration 189µs  ↑ 203 B  ↓ 780 B ] 403 Forbidden Content-Type: application/xml No-Gzip-Compression: true Server: MinIO Strict-Transport-Security: max-age=31536000; includeSubDomains Vary: Origin,Accept-Encoding X-Content-Type-Options: nosniff Accept-Ranges: bytes Content-Length: 431 X-Xss-Protection: 1; mode=block Content-Security-Policy: block-all-mixed-content X-Amz-Request-Id: 1706602ECAA3D26A <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><Key>badges/badges/honor.png</Key><BucketName>openedx</BucketName><Resource>/openedx/badges/badges/honor.png</Resource><RequestId>1706602ECAA3D26A</RequestId><HostId>61ce544a-4846-4f4b-9e24-07a3c078c22a</HostId></Error>

Hoping someone with some familiarity can help out.


Here’s the exception trace from the migration:

File "/openedx/edx-platform/lms/djangoapps/certificates/migrations/", line 20, in forwards
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/fields/", line 89, in save =, content, max_length=self.field.max_length)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/files/", line 54, in save
    name = self._save(name, content)
  File "/openedx/venv/lib/python3.8/site-packages/storages/backends/", line 495, in _save
    self._save_content(obj, content, parameters=parameters)
  File "/openedx/venv/lib/python3.8/site-packages/storages/backends/", line 510, in _save_content
    obj.upload_fileobj(content, ExtraArgs=put_parameters)
  File "/openedx/venv/lib/python3.8/site-packages/boto3/s3/", line 511, in object_upload_fileobj
    return self.meta.client.upload_fileobj(
  File "/openedx/venv/lib/python3.8/site-packages/boto3/s3/", line 431, in upload_fileobj
    return future.result()
  File "/openedx/venv/lib/python3.8/site-packages/s3transfer/", line 73, in result
    return self._coordinator.result()
  File "/openedx/venv/lib/python3.8/site-packages/s3transfer/", line 233, in result
    raise self._exception
  File "/openedx/venv/lib/python3.8/site-packages/s3transfer/", line 126, in __call__
    return self._execute_main(kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/s3transfer/", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/s3transfer/", line 692, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/openedx/venv/lib/python3.8/site-packages/botocore/", line 317, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/botocore/", line 615, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (SignatureDoesNotMatch) when calling the PutObject operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.