Master native installation failed to build

Looks like we have the same issue as for Koa. Latest fix was to update mysql role to use 5.7.33-1ubuntu18.04 version.

This time we have to update to .34 version.

$ sudo apt show mysql-client=5.7.33-1ubuntu18.04
E: Version '5.7.33-1ubuntu18.04' for 'mysql-client' was not found
E: Version '5.7.33-1ubuntu18.04' for 'mysql-client' was not found
E: No packages found
$ sudo apt show mysql-client=5.7.34-1ubuntu18.04
Package: mysql-client
Version: 5.7.34-1ubuntu18.04
Priority: optional
Section: database
Source: mysql-community
Maintainer: MySQL Release Engineering <mysql-build@oss.oracle.com>
Installed-Size: 80.9 kB
Depends: mysql-community-client (= 5.7.34-1ubuntu18.04)

Proposed PR here

Why was this error not reported by CI? open-release/lilac.master installs just fine on Ubuntu 20.04: build-image (#3367) · Jobs · Community / openedx-configuration-ci · GitLab

My suggestion is that it was triggered 21 hours ago.
Let’s retry it.

Retried right now on fresh instance:

TASK [mysql : install mysql-5.7-client] ****************************************
fatal: [localhost]: FAILED! => {"cache_update_time": 1618903000, "cache_updated": true, "changed": false, "msg": "'/usr/bin/apt-get -y -o \"Dpkg::Options::=--force-confdef\" -o \"Dpkg::Options::=--force-confold\"      install 'mysql-client=5.7.33-1ubuntu18.04'' failed: E: Version '5.7.33-1ubuntu18.04' for 'mysql-client' was not found\n", "rc": 100, "stderr": "E: Version '5.7.33-1ubuntu18.04' for 'mysql-client' was not found\n", "stderr_lines": ["E: Version '5.7.33-1ubuntu18.04' for 'mysql-client' was not found"], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information..."]}

PLAY RECAP *********************************************************************
localhost                  : ok=60   changed=29   unreachable=0    failed=1    skipped=186  rescued=0    ignored=1


============================================================
Ansible failed!
------------------------------------------------------------

Decoded error:
No message to decode.

============================================================
Installation failed!
------------------------------------------------------------
If you need help, see https://open.edx.org/getting-help .
When asking for help, please provide as much information as you can.
These might be helpful:
    Your log file is at /home/ubuntu/logs/install-20210420-071156.log
    Your environment:
        OPENEDX_RELEASE=open-release/lilac.master

Let’s see how today’s run go (scheduled at 2021-04-20T09:00:00Z).

1 Like

You were right @Maksim_Sokolskiy, the Lilac build failed this morning: build-image (#3372) · Jobs · Community / openedx-configuration-ci · GitLab

What’s also worrying to me is that the build failed but no error was reported by CI. But then, it’s only natural: the native.sh script does not report an error on failure:

cd /var/tmp/configuration/playbooks && sudo -E ansible-playbook -c local ./openedx_native.yml -i "localhost," $EXTRA_VARS "$@"
ansible_status=$?

if [[ $ansible_status -ne 0 ]]; then
  ...
fi
# we should add "exit $ansible_status" here
1 Like

Is Ubuntu removing packages? Is this a normal occurrence?

I’m not sure about normality but our DevOps team is used to this and use archived version from cdn for builds (I hope edx.org do the same).

Yeap, we have to add the status.

Also I have some concerns about the build flow and testing method:

  • building the AMI w/o actual deployment (though it’s really full build process and almost everything we want to test)
  • absence of E2E testing stage

Let me finish with the MVP of the Build -> Deploy -> Test (Smoke E2E tests) -> Destroy pipeline I mentioned during one our BTR meetings and I’m sure we will improve the testing stage of your pipeline.

This CI pipeline is not very close to my heart and I’ll happily accept external contributions or replace it with something else :slight_smile: This is something that I’d rather spend not too much time on, personally.

:ok_hand:

Actually I’m thinking about enhancing it by adding terraforming and testing stages.
Let see what we will have eventually.

1 Like

Just install seperately MySQL client after that run step 5th again it will solve