Open edX k8s installation

Site Operators Tutor Help
1

I am trying to install an OpenEdx production environment on a kubernetes cluster that I have installed on three servers that I had available.
The cluster consists of a master and two workers. To perform the OpenEdx installation, I have installed tutor on the master node and tried to follow the steps in the documentation (Kubernetes deployment — Tutor documentation).
However when I run

$ tutor k8s launch

process ends with an error in the execution of a MinIO cronjob.

Job minio-job-20230413121915 is running. To view the logs from this job, run:
kubectl logs --namespace=openedx --follow $(kubectl get --namespace=openedx pods --selector=job-name=minio-job-20230413121915 -o=jsonpath=“{.items[0].metadata.name}”)
Waiting for job completion…
Error: Job minio-job-20230413121915 failed. View the job logs to debug this issue.

… more info …
$ kubectl -n openedx logs minio-job-20230413121915-jv9mf
Added minio successfully. mc: Unable to make bucket minio/openedx. Put “http://minio:9000/openedx/”: dial tcp 10.101.77.225:9000: connect: connection refused mc: Unable to make bucket minio/openedxuploads. Put “http://minio:9000/openedxuploads/”: dial tcp 10.101.77.225:9000: connect: connection refused mc: Unable to make bucket minio/openedxvideos. Put “http://minio:9000/openedxvideos/”: dial tcp 10.101.77.225:9000: connect: connection refused

1 Like
2

Hi Nacho, and welcome to this community! Maybe @gabor (our resident K8s expert) can help out.

3

@nachotheix Unfortunately, we are not using minio, so I cannot really help much, though @regis may already met with this issue.

What I would check with no context if minio is running and not in a crash back loop. I can imagine that the job is trying to be executed against a failed pod.

2 Likes
4

@nachotheix Probably getting more information about the error might be helpful:

kubectl describe po <minio-pod-name> --namespace openedx
5

@nachotheix
I performed the following and it worked for me on my local machine.

Install and enable minio.

tutor plugins install minio
tutor plugins enable minio
tutor config save

The deployments, service, replica set, and pod containers for minio won’t be available until you run this step. This is why you were getting that error above.

tutor k8s reboot

Initialize minio.

tutor k8s init --limit=minio

Access minio in the browser at http://minio.local.overhang.io/

# Go to `tutor-env/env/k8s/deployments.yml` and search for `name: minio` to find the following login credentials.
# These are defined here https://github.com/overhangio/tutor-minio/blob/master/tutorminio/plugin.py#L38-L48
env:
  - name: MINIO_ROOT_USER
    value: "openedx"
  - name: MINIO_ROOT_PASSWORD
    value: "24|random_string"
1 Like
6

I am receiving this error though whenever I perform a course import.

Traceback (most recent call last):
  File "/openedx/edx-platform/./cms/djangoapps/contentstore/views/import_export.py", line 214, in _write_chunk
    storage_path = course_import_export_storage.save('olx_import/' + filename, django_file)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/files/storage.py", line 54, in save
    name = self._save(name, content)
  File "/openedx/venv/lib/python3.8/site-packages/storages/backends/s3boto3.py", line 495, in _save
    self._save_content(obj, content, parameters=parameters)
  File "/openedx/venv/lib/python3.8/site-packages/storages/backends/s3boto3.py", line 510, in _save_content
    obj.upload_fileobj(content, ExtraArgs=put_parameters)
  File "/openedx/venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 511, in object_upload_fileobj
    return self.meta.client.upload_fileobj(
  File "/openedx/venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 431, in upload_fileobj
    return future.result()
  File "/openedx/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 73, in result
    return self._coordinator.result()
  File "/openedx/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 233, in result
    raise self._exception
  File "/openedx/venv/lib/python3.8/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/s3transfer/tasks.py", line 332, in _main
    response = client.create_multipart_upload(
  File "/openedx/venv/lib/python3.8/site-packages/botocore/client.py", line 317, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/botocore/client.py", line 601, in _make_api_call
    http, parsed_response = self._endpoint.make_request(
  File "/openedx/venv/lib/python3.8/site-packages/botocore/endpoint.py", line 143, in make_request
    return self._send_request(request_dict, operation_model)
  File "/openedx/venv/lib/python3.8/site-packages/botocore/endpoint.py", line 171, in _send_request
    while self._needs_retry(attempts, operation_model, request_dict,
  File "/openedx/venv/lib/python3.8/site-packages/botocore/endpoint.py", line 262, in _needs_retry
    responses = self._event_emitter.emit(
  File "/openedx/venv/lib/python3.8/site-packages/botocore/hooks.py", line 227, in emit
    return self._emit(event_name, kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/openedx/venv/lib/python3.8/site-packages/botocore/retryhandler.py", line 250, in __call__
    should_retry = self._should_retry(attempt_number, response,
  File "/openedx/venv/lib/python3.8/site-packages/botocore/retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/openedx/venv/lib/python3.8/site-packages/botocore/retryhandler.py", line 316, in __call__
    checker_response = checker(attempt_number, response,
  File "/openedx/venv/lib/python3.8/site-packages/botocore/retryhandler.py", line 222, in __call__
    return self._check_caught_exception(
  File "/openedx/venv/lib/python3.8/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/openedx/venv/lib/python3.8/site-packages/botocore/endpoint.py", line 210, in _get_response
    http_response = self.http_session.send(
  File "/openedx/venv/lib/python3.8/site-packages/botocore/vendored/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/botocore/vendored/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
botocore.vendored.requests.exceptions.ConnectionError: ('Connection aborted.', OSError(99, 'Cannot assign requested address'))
1 Like
8

Hi @Zachary_Trabookis,

I would like to report my test case on Palm 16.1.8 and Quince 17.0.2 in Azure K8S with tutor-minio and with tutor-contrib-s3.

[1] Using Quince 17.0.2 with tutor-minio, with a nearly public access setting in Azure Storage, cms import / export course and lms instructor generate report, not working and error logs found.

[2] Using Palm 16.1.8 still with tutor-minio, Azure Storage remain unchange, lms instructor able to generate report. cms import / export course same error as [1]

[3] Using Quince 17.0.2 with tutor-contrib-s3, AWS S3, lms instructor able to generate report. cms import / export course task keep-going and never end, but the tar.gz file found in AWS S3 and no error logs found in cms