S3cmd v1.6.1 is Python 2 in open-release/juniper.alpha1

Site Operators
1

Hello everyone,

I submitted a PR recently to add conditionals for applying the aws Ansible
role in the configuration project (https://github.com/edx/configuration/pull/5652), but I just want to report how I noticed
this role was being run in the first place. There is a dependency on the
s3cmd package, version 1.6.1, which contains Python 2 code (a print statement
without parentheses) that generates the following exception (excerpted here from
the Ansible output and reformatted):

ERROR: Command errored out with exit status 1:
     command: /tmp/edx_config/venv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-y5_h4n1b/s3cmd/setup.py'"'"'; __file__='"'"'/tmp/pip-install-y5_h4n1b/s3cmd/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-y5_h4n1b/s3cmd/pip-egg-info
         cwd: /tmp/pip-install-y5_h4n1b/s3cmd/
    Complete output (6 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-y5_h4n1b/s3cmd/setup.py", line 18
        print "Using xml.etree.ElementTree for XML processing"
                                                             ^
    SyntaxError: Missing parentheses in call to 'print'

Here is where it’s pinned in the open-release/juniper.alpha1 branch:

Here’s the setup.py file from version 1.6.1, clearly Python 2:

I didn’t know whether to submit a PR for this because I’m not sure how this
utility fits in with your architecture and what it would mean for you to upgrade
it. We don’t use the aws Ansible role in MITx, but I just wanted to pass this
along.

Let me know if I should cross-post this, too.

-Mark

2

I probably should have done this first, but I’ve just created a Jira ticket:
https://openedx.atlassian.net/browse/CRI-181

3

s3cmd has generally been replaced with the more supported aws s3 command, which should be installed everywhere using from the awscli package. I see two usages of it in configuration, one for mongo to s3 backups, and one in send-logs-to-s3. I think these should both be ported over to use aws s3 sync

The concern I have is there’s a bunch of tried and tested arguments in https://github.com/edx/configuration/blob/cf4d221d384ed396a3008c31487f385544ef08e7/playbooks/roles/aws/templates/send-logs-to-s3.j2#L130 and I’m pretty sure none of them translate cleanly to aws s3 sync

4

That PR (https://github.com/edx/configuration/pull/5652) has now merged, but it’s a fix only on open-release/juniper.alpha1, and not on master. That means that when Juniper is eventually started officially for real from master, this will still be a problem. How are we planning to fix it for master?

5

I opened https://github.com/edx/configuration/pull/5656 which should remove all the pip installed versions of s3cmd and replaces it with the debian installed package, which is identical.

I think I could also replace this with aws s3, but I think this will remove the python2 requirement.

6

We’ve made two fixes for this, on both open-release/juniper.alpha1 and master. @mbrdlove Can you check if it solves the problem for you?

7

Thanks, @nedbat! I will be able to try that tomorrow.

8

Yes, we are no longer getting the error that I reported. Thank you!