Info
OS: Latest Debian image on GCP
Kubernetes: K3s spread across 3 nodes
instance-7 Ready <none> 111m v1.27.3+k3s1
instance-5 Ready control-plane,master 126m v1.27.3+k3s1
instance-6 Ready <none> 111m v1.27.3+k3s1
Search Terms
Tried search terms in forums botocore and PutObject with no luck
Guesses
I suspect that the issues is between minio and the lms app but i have no idea what is going on here. I have checked GCP firewall and i ensured the ports are open. The minio page opens when i navigate to it. I suspect i am missing a dependency somewhere but i have no idea where. Thanks for any help yall can offer.
Software installation:
1 curl -sfL https://get.k3s.io | sh -
2 sudo cat /var/lib/rancher/k3s/server/node-token
3 kubectl get nodes
4 export KUBECONFIG=~/.kube/config
5 mkdir ~/.kube 2> /dev/null
6 sudo k3s kubectl config view --raw > "$KUBECONFIG"
7 chmod 600 "$KUBECONFIG"
8 kubectl get nodes
9 watch kubectl get nodes
10 ss -lnt
11 watch kubectl get nodes
12 pip install "tutor[full]"
13 sudo apt install python3 python3-pip libyaml-dev
14 pip install "tutor[full]"
15 tutor k8s start caddy
16 .local/bin/tutor k8s start caddy
17 .local/bin/tutor config save --interactive
18 sudo apt install docker.io
19 .local/bin/tutor config save --interactive
20 .local/bin/tutor k8s start caddy
21 kubectl --namespace openedx get services/caddy
22 watch kubectl --namespace openedx get services/caddy
23 tutor k8s logs -f caddy
24 .local/bin/tutor k8s logs -f caddy
25 .local/bin/tutor plugins enable minio
26 .local/bin/tutor k8s status
27 .local/bin/tutor k8s launch
Attempted Launch:
I have tried it with and without SSL.
==================================================
Interactive platform configuration
==================================================
Your website domain name for students (LMS) [lms.strixlabs.io]
Your website domain name for teachers (CMS) [studio.lms.strixlabs.io]
Your platform name/title [Strix Labs LMS]
Your public contact email address [XXXXXXXXX@strixlabs.io]
The default language code for the platform [en]
Activate SSL/TLS certificates for HTTPS access? Important note: this will NOT work in a development environment. [y/N] n
Configuration saved to /home/koelslaw/.local/share/tutor/config.yml
Environment generated in /home/koelslaw/.local/share/tutor/env
=====================================
Starting the platform
=====================================
kubectl get namespaces openedx
Error from server (NotFound): namespaces "openedx" not found
Namespace does not exist: now creating it...
kubectl apply --kustomize /home/koelslaw/.local/share/tutor/env --wait --selector app.kubernetes.io/component=namespace
namespace/openedx created
kubectl apply --kustomize /home/koelslaw/.local/share/tutor/env --wait --selector app.kubernetes.io/component=volume
persistentvolumeclaim/caddy created
persistentvolumeclaim/elasticsearch created
persistentvolumeclaim/minio created
persistentvolumeclaim/mongodb created
persistentvolumeclaim/mysql created
persistentvolumeclaim/redis created
kubectl apply --kustomize /home/koelslaw/.local/share/tutor/env --selector 'app.kubernetes.io/component notin (job,volume,namespace)'
configmap/caddy-config-fhd2fhkk52 created
configmap/mfe-caddy-config-2t4bk6mgc9 created
configmap/openedx-config-5h655b7764 created
configmap/openedx-settings-cms-dt4m2ccgmf created
configmap/openedx-settings-lms-k4tdb9cgd9 created
configmap/openedx-uwsgi-config-f57k74k5cc created
configmap/redis-config-fccm65mh4m created
service/caddy created
service/cms created
service/elasticsearch created
service/lms created
service/mfe created
service/minio created
service/mongodb created
service/mysql created
service/redis created
service/smtp created
deployment.apps/caddy created
deployment.apps/cms created
deployment.apps/cms-worker created
deployment.apps/elasticsearch created
deployment.apps/lms created
deployment.apps/lms-worker created
deployment.apps/mfe created
deployment.apps/minio created
deployment.apps/mongodb created
deployment.apps/mysql created
deployment.apps/redis created
deployment.apps/smtp created
================================================
Database creation and migrations
================================================
Initialising all services...
Running init task in lms
kubectl apply --kustomize /home/koelslaw/.local/share/tutor/env --selector app.kubernetes.io/name=lms-job-20230726221734
job.batch/lms-job-20230726221734 created
Job lms-job-20230726221734 is running. To view the logs from this job, run:
kubectl logs --namespace=openedx --follow $(kubectl get --namespace=openedx pods --selector=job-name=lms-job-20230726221734 -o=jsonpath="{.items[0].metadata.name}")
Waiting for job completion...
Job lms-job-20230726221734 successful.
Running init task in minio
kubectl apply --kustomize /home/koelslaw/.local/share/tutor/env --selector app.kubernetes.io/name=minio-job-20230726221750
job.batch/minio-job-20230726221750 created
Job minio-job-20230726221750 is running. To view the logs from this job, run:
kubectl logs --namespace=openedx --follow $(kubectl get --namespace=openedx pods --selector=job-name=minio-job-20230726221750 -o=jsonpath="{.items[0].metadata.name}")
Waiting for job completion...
Job minio-job-20230726221750 successful.
Running init task in mysql
kubectl apply --kustomize /home/koelslaw/.local/share/tutor/env --selector app.kubernetes.io/name=mysql-job-20230726221759
job.batch/mysql-job-20230726221759 created
Job mysql-job-20230726221759 is running. To view the logs from this job, run:
kubectl logs --namespace=openedx --follow $(kubectl get --namespace=openedx pods --selector=job-name=mysql-job-20230726221759 -o=jsonpath="{.items[0].metadata.name}")
Waiting for job completion...
Job mysql-job-20230726221759 successful.
Running init task in lms
kubectl apply --kustomize /home/koelslaw/.local/share/tutor/env --selector app.kubernetes.io/name=lms-job-20230726221808
job.batch/lms-job-20230726221808 created
Job lms-job-20230726221808 is running. To view the logs from this job, run:
kubectl logs --namespace=openedx --follow $(kubectl get --namespace=openedx pods --selector=job-name=lms-job-20230726221808 -o=jsonpath="{.items[0].metadata.name}")
Waiting for job completion...
Error: Job lms-job-20230726221808 failed. View the job logs to debug this issue.
Result:
kubectl logs --namespace=openedx --follow $(kubectl get --namespace=openedx pods --selector=job-name=lms-job-20230726221808 -o=jsonpath="{.items[0].metadata.name}")
2023/07/26 22:18:12 Ready: tcp://mysql:3306.
2023/07/26 22:18:12 Ready: tcp://mongodb:27017.
Loading settings lms.envs.tutor.production
Operations to perform:
Apply all migrations: admin, agreements, announcements, api_admin, assessment, auth, badges, blackboard, block_structure, bookmarks, branding, bulk_email, bulk_grades, bundles, calendar_sync, canvas, catalog, celery_utils, certificates, commerce, completion, consent, content_libraries, content_type_gating, contentserver, contenttypes, cornerstone, cors_csrf, course_action_state, course_apps, course_date_signals, course_duration_limits, course_goals, course_groups, course_home_api, course_live, course_modes, course_overviews, courseware, crawlers, credentials, credit, dark_lang, database_fixups, degreed, degreed2, demographics, discounts, discussions, django_celery_results, django_comment_common, django_notify, edx_name_affirmation, edx_proctoring, edx_when, edxval, email_marketing, embargo, enterprise, entitlements, experiments, external_user_ids, grades, instructor_task, integrated_channel, learner_pathway_progress, learning_sequences, lms_xblock, lti1p3_tool_config, lti_consumer, milestones, mobile_api, moodle, oauth2_provider, oauth_dispatch, organizations, outcome_surveys, program_enrollments, programs, redirects, rss_proxy, sap_success_factors, save_for_later, schedules, sessions, site_configuration, sites, social_django, splash, split_modulestore_django, staffgrader, static_replace, status, student, submissions, super_csv, support, survey, survey_report, system_wide_roles, teams, theming, third_party_auth, thumbnail, track, user_api, user_authn, user_tours, util, verify_student, video_config, video_pipeline, waffle, waffle_utils, wiki, workflow, xapi, xblock_django
Running migrations:
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying admin.0002_logentry_remove_auto_add... OK
Applying admin.0003_logentry_add_action_flag_choices... OK
Applying agreements.0001_initial... OK
Applying announcements.0001_initial... OK
Applying sites.0001_initial... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying api_admin.0001_initial... OK
Applying api_admin.0002_auto_20160325_1604... OK
Applying api_admin.0003_auto_20160404_1618... OK
Applying api_admin.0004_auto_20160412_1506... OK
Applying api_admin.0005_auto_20160414_1232... OK
Applying api_admin.0006_catalog... OK
Applying api_admin.0007_delete_historical_api_records... OK
Applying assessment.0001_initial... OK
Applying assessment.0002_staffworkflow... OK
Applying assessment.0003_expand_course_id... OK
Applying assessment.0004_historicalsharedfileupload_sharedfileupload... OK
Applying assessment.0005_add_filename_to_sharedupload... OK
Applying assessment.0006_TeamWorkflows... OK
Applying assessment.0007_staff_workflow_blank... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying auth.0010_alter_group_name_max_length... OK
Applying auth.0011_update_proxy_permissions... OK
Applying auth.0012_alter_user_first_name_max_length... OK
Applying instructor_task.0001_initial... OK
Applying certificates.0001_initial... OK
Applying certificates.0002_data__certificatehtmlviewconfiguration_data... OK
2023-07-26 22:18:45,188 INFO 17 [botocore.vendored.requests.packages.urllib3.connectionpool] [user None] [ip None] connectionpool.py:202 - Starting new HTTP connection (1): files.lms.strixlabs.io
Traceback (most recent call last):
File "./manage.py", line 106, in <module>
execute_from_command_line([sys.argv[0]] + django_args)
File "/openedx/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
utility.execute()
File "/openedx/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/openedx/venv/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
self.execute(*args, **cmd_options)
File "/openedx/venv/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
output = self.handle(*args, **options)
File "/openedx/venv/lib/python3.8/site-packages/django/core/management/base.py", line 89, in wrapped
res = handle_func(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 244, in handle
post_migrate_state = executor.migrate(
File "/openedx/venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
File "/openedx/venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File "/openedx/venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
state = migration.apply(state, schema_editor)
File "/openedx/venv/lib/python3.8/site-packages/django/db/migrations/migration.py", line 123, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "/openedx/venv/lib/python3.8/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
self.code(from_state.apps, schema_editor)
File "/openedx/edx-platform/lms/djangoapps/certificates/migrations/0003_data__default_modes.py", line 20, in forwards
conf.icon.save(
File "/openedx/venv/lib/python3.8/site-packages/django/db/models/fields/files.py", line 89, in save
self.name = self.storage.save(name, content, max_length=self.field.max_length)
File "/openedx/venv/lib/python3.8/site-packages/django/core/files/storage.py", line 54, in save
name = self._save(name, content)
File "/openedx/venv/lib/python3.8/site-packages/storages/backends/s3boto3.py", line 495, in _save
self._save_content(obj, content, parameters=parameters)
File "/openedx/venv/lib/python3.8/site-packages/storages/backends/s3boto3.py", line 510, in _save_content
obj.upload_fileobj(content, ExtraArgs=put_parameters)
File "/openedx/venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 511, in object_upload_fileobj
return self.meta.client.upload_fileobj(
File "/openedx/venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 431, in upload_fileobj
return future.result()
File "/openedx/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 73, in result
return self._coordinator.result()
File "/openedx/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 233, in result
raise self._exception
File "/openedx/venv/lib/python3.8/site-packages/s3transfer/tasks.py", line 126, in __call__
return self._execute_main(kwargs)
File "/openedx/venv/lib/python3.8/site-packages/s3transfer/tasks.py", line 150, in _execute_main
return_value = self._main(**kwargs)
File "/openedx/venv/lib/python3.8/site-packages/s3transfer/upload.py", line 692, in _main
client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
File "/openedx/venv/lib/python3.8/site-packages/botocore/client.py", line 314, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/openedx/venv/lib/python3.8/site-packages/botocore/client.py", line 612, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (404) when calling the PutObject operation: Not Found