Unable to run kustomize-based tutor commands (or: how do we generate the configmaps that tutor relies on?)

Site Operators Tutor Help
1

I’ve tried to run a few tutor commands that rely on kustomize templates. So far, I’ve tried tutor k8s createuser and tutor k8s importdemocourse. Both times, the container for the associated job fails to create.

Edit: We’re using tutor version 14.0.1

The relevant bit of information is below:

  Normal   Scheduled    10m                 default-scheduler  Successfully assigned openedx-prod/cms-job-20220726154615--1-p9thb to ip-192-168-4-24.us-east-2.compute.internal
  Warning  FailedMount  9m8s (x8 over 10m)  kubelet            MountVolume.SetUp failed for volume "settings-cms" : configmap "openedx-settings-cms-96tbh895cb" not found
  Warning  FailedMount  9m8s (x8 over 10m)  kubelet            MountVolume.SetUp failed for volume "settings-lms" : configmap "openedx-settings-lms-g95m8kkgfh" not found
  Warning  FailedMount  8m9s                kubelet            Unable to attach or mount volumes: unmounted volumes=[settings-lms settings-cms config], unattached volumes=[settings-lms settings-cms config kube-api-access-hjpg7]: timed out waiting for the condition
  Warning  FailedMount  4m (x11 over 10m)   kubelet            MountVolume.SetUp failed for volume "config" : configmap "openedx-config-4f4h797tg2" not found

Which means the config maps can’t be found. I’m not sure how to create them.

For more context, this is the complete pod description:

kubectl describe pod cms-job-20220726154615--1-p9thb --namespace=openedx-prod
Name:           cms-job-20220726154615--1-p9thb
Namespace:      openedx-prod
Priority:       0
Node:           ip-192-168-4-24.us-east-2.compute.internal/192.168.4.24
Start Time:     Tue, 26 Jul 2022 11:46:17 -0400
Labels:         app.kubernetes.io/instance=openedx-ZMdtW7IbnSS61k1qgjR7en68
                app.kubernetes.io/managed-by=tutor
                app.kubernetes.io/part-of=openedx
                controller-uid=1d5c59c2-d846-4a2f-ad4d-7e6411911be4
                job-name=cms-job-20220726154615
Annotations:    app.kubernetes.io/version: 14.0.1
                kubernetes.io/psp: eks.privileged
Status:         Pending
IP:
IPs:            <none>
Controlled By:  Job/cms-job-20220726154615
Containers:
  cms:
    Container ID:
    Image:         docker.io/overhangio/openedx:14.0.1
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Args:
      sh
      -e
      -c
      echo "Loading settings $DJANGO_SETTINGS_MODULE"

      # Import demo course
      git clone https://github.com/openedx/edx-demo-course --branch open-release/nutmeg.1 --depth 1 ../edx-demo-course
      python ./manage.py cms import ../data ../edx-demo-course

      # Re-index courses
      ./manage.py cms reindex_course --all --setup
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:
      SERVICE_VARIANT:         cms
      DJANGO_SETTINGS_MODULE:  cms.envs.tutor.production
    Mounts:
      /openedx/config from config (rw)
      /openedx/edx-platform/cms/envs/tutor/ from settings-cms (rw)
      /openedx/edx-platform/lms/envs/tutor/ from settings-lms (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hjpg7 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  settings-lms:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      openedx-settings-lms-g95m8kkgfh
    Optional:  false
  settings-cms:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      openedx-settings-cms-96tbh895cb
    Optional:  false
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      openedx-config-4f4h797tg2
    Optional:  false
  kube-api-access-hjpg7:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                 From               Message
  ----     ------       ----                ----               -------
  Normal   Scheduled    10m                 default-scheduler  Successfully assigned openedx-prod/cms-job-20220726154615--1-p9thb to ip-192-168-4-24.us-east-2.compute.internal
  Warning  FailedMount  9m8s (x8 over 10m)  kubelet            MountVolume.SetUp failed for volume "settings-cms" : configmap "openedx-settings-cms-96tbh895cb" not found
  Warning  FailedMount  9m8s (x8 over 10m)  kubelet            MountVolume.SetUp failed for volume "settings-lms" : configmap "openedx-settings-lms-g95m8kkgfh" not found
  Warning  FailedMount  8m9s                kubelet            Unable to attach or mount volumes: unmounted volumes=[settings-lms settings-cms config], unattached volumes=[settings-lms settings-cms config kube-api-access-hjpg7]: timed out waiting for the condition
  Warning  FailedMount  4m (x11 over 10m)   kubelet            MountVolume.SetUp failed for volume "config" : configmap "openedx-config-4f4h797tg2" not found

And the associated job description:

kubectl describe job cms-job-20220726154615 --namespace=openedx-prod
Name:             cms-job-20220726154615
Namespace:        openedx-prod
Selector:         controller-uid=1d5c59c2-d846-4a2f-ad4d-7e6411911be4
Labels:           app.kubernetes.io/component=job
                  app.kubernetes.io/instance=openedx-ZMdtW7IbnSS61k1qgjR7en68
                  app.kubernetes.io/managed-by=tutor
                  app.kubernetes.io/name=cms-job-20220726154615
                  app.kubernetes.io/part-of=openedx
Annotations:      app.kubernetes.io/version: 14.0.1
Parallelism:      1
Completions:      1
Completion Mode:  NonIndexed
Start Time:       Tue, 26 Jul 2022 11:46:17 -0400
Pods Statuses:    1 Running / 0 Succeeded / 0 Failed
Pod Template:
  Labels:       app.kubernetes.io/instance=openedx-ZMdtW7IbnSS61k1qgjR7en68
                app.kubernetes.io/managed-by=tutor
                app.kubernetes.io/part-of=openedx
                controller-uid=1d5c59c2-d846-4a2f-ad4d-7e6411911be4
                job-name=cms-job-20220726154615
  Annotations:  app.kubernetes.io/version: 14.0.1
  Containers:
   cms:
    Image:      docker.io/overhangio/openedx:14.0.1
    Port:       <none>
    Host Port:  <none>
    Args:
      sh
      -e
      -c
      echo "Loading settings $DJANGO_SETTINGS_MODULE"

      # Import demo course
      git clone https://github.com/openedx/edx-demo-course --branch open-release/nutmeg.1 --depth 1 ../edx-demo-course
      python ./manage.py cms import ../data ../edx-demo-course

      # Re-index courses
      ./manage.py cms reindex_course --all --setup
    Environment:
      SERVICE_VARIANT:         cms
      DJANGO_SETTINGS_MODULE:  cms.envs.tutor.production
    Mounts:
      /openedx/config from config (rw)
      /openedx/edx-platform/cms/envs/tutor/ from settings-cms (rw)
      /openedx/edx-platform/lms/envs/tutor/ from settings-lms (rw)
  Volumes:
   settings-lms:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      openedx-settings-lms-g95m8kkgfh
    Optional:  false
   settings-cms:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      openedx-settings-cms-96tbh895cb
    Optional:  false
   config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      openedx-config-4f4h797tg2
    Optional:  false
Events:
  Type    Reason            Age   From            Message
  ----    ------            ----  ----            -------
  Normal  SuccessfulCreate  50m   job-controller  Created pod: cms-job-20220726154615--1-p9thb

Finally, grepping through our configMaps using the command kubectl get configmaps --namespace=openedx-prod | grep "openedx-config-4f4h797tg2\|openedx-settings-cms-96tbh895cb\|openedx-settings-lms-g95m8kkgfh" shows that the configmaps don’t exist.

Alternatively, I’d be open to learning how to manually import the demo corse/edit users instead.

1 Like
2

Those configMaps are generated by Kustomize. You can run kubectl kustomize "$(tutor config printroot)/env" > out.yml to check the final resources that are being applied.

I think you may have changed some files and now Kustomize is generating a new hash for the configMaps but you didn’t apply those changes, therefore your cluster keeps the old maps and tutor creates the job assuming the new ones have been applied.

You can try and apply the new changes and run the job again.

You can also write a shell script with the commands used by importdemocourse and createuser. For example:

#! /bin/sh

./manage.py lms  manage_user --staff --superuser user user@example.com
./manage.py lms shell -c "from django.contrib.auth import get_user_model
u = get_user_model().objects.get(username='user')
u.set_password('password')
u.save()"

and then run kubectl exec -i lms-123abc9876-zyxwv -- bash < create_user.sh

The source for democourse can be found here and createuser here.

1 Like
3

Thanks, this was helpful!

Rerunning kubectl apply ended up fixing everything. However, the tutor k8s importdemocourse command still failed with AccessDenied errors.

But when I opened a shell on the lms container and run the commands directly, they succeeded. Odd, but I have my demo course.

4

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.