amazon.aws.s3_bucket not working with Storagegrid

[lorenzstorm1]

Summary

Running this against our StorageGrid doesn't work:
StorageGrid does not have the "Payer Feature" implemented so I guess this is the Problem. But on the other hand the Module has StorageGrid listed as supported

Issue Type

Bug Report

Component Name

s3_bucket

Ansible Version

$ ansible --version
ansible 2.9.11
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/var/lib/awx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.6/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.6.8 (default, Apr 16 2020, 01:36:27) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]

Collection Versions

$ ansible-galaxy collection list
bash-4.4# ansible-galaxy collection list
usage: ansible-galaxy collection [-h] COLLECTION_ACTION ...
ansible-galaxy collection: error: argument COLLECTION_ACTION: invalid choice: 'list' (choose from 'init', 'build', 'publish', 'install')

AWS SDK versions

$ pip show boto boto3 botocore
bash-4.4# pip3 show boto -V
^[[28~Name: boto
Version: 2.49.0
Summary: Amazon Web Services Library
Home-page: https://github.com/boto/boto/
Author: Mitch Garnaat
Author-email: [email protected]
License: MIT
Location: /usr/local/lib/python3.6/site-packages
Requires:
bash-4.4# ~^C
bash-4.4# ^C
bash-4.4# pip3 show boto boto3 botocore
Name: boto
Version: 2.49.0
Summary: Amazon Web Services Library
Home-page: https://github.com/boto/boto/
Author: Mitch Garnaat
Author-email: [email protected]
License: MIT
Location: /usr/local/lib/python3.6/site-packages
Requires:
---
Name: boto3
Version: 1.19.0
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: None
License: Apache License 2.0
Location: /usr/local/lib/python3.6/site-packages
Requires: jmespath, botocore, s3transfer
---
Name: botocore
Version: 1.22.0
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: None
License: Apache License 2.0
Location: /usr/local/lib/python3.6/site-packages
Requires: jmespath, python-dateutil, urllib3
bash-4.4#

Configuration

$ ansible-config dump --only-changed

OS / Environment

CentOS 7

Steps to Reproduce

---
  - hosts: localhost
    tasks:
    - name: set versioning
      s3_bucket:
        name: sto0003ltest178
        state: present
        s3_url: https://s3.xxxxxxxx.de
        validate_certs: no
        versioning: no
        aws_access_key: xxxxxxx
        aws_secret_key: xxxxxxx

Expected Results

I would expect it to create a Bucket

Actual Results

{
    "msg": "Failed to get bucket request payment: An error occurred (XNotImplemented) when calling the GetBucketRequestPayment operation: The request you provided implies functionality that is not implemented.",
    "exception": "Traceback (most recent call last):\n  File \"/tmp/ansible_s3_bucket_payload_ep_yadhp/ansible_s3_bucket_payload.zip/ansible/modules/cloud/amazon/s3_bucket.py\", line 218, in create_or_update_bucket\n  File \"/tmp/ansible_s3_bucket_payload_ep_yadhp/ansible_s3_bucket_payload.zip/ansible/module_utils/cloud.py\", line 150, in retry_func\n    raise e\n  File \"/tmp/ansible_s3_bucket_payload_ep_yadhp/ansible_s3_bucket_payload.zip/ansible/module_utils/cloud.py\", line 140, in retry_func\n    return f(*args, **kwargs)\n  File \"/tmp/ansible_s3_bucket_payload_ep_yadhp/ansible_s3_bucket_payload.zip/ansible/modules/cloud/amazon/s3_bucket.py\", line 405, in get_bucket_request_payment\n  File \"/var/lib/awx/venv/ansible/lib/python3.6/site-packages/botocore/client.py\", line 357, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n  File \"/var/lib/awx/venv/ansible/lib/python3.6/site-packages/botocore/client.py\", line 661, in _make_api_call\n    raise error_class(parsed_response, operation_name)\nbotocore.exceptions.ClientError: An error occurred (XNotImplemented) when calling the GetBucketRequestPayment operation: The request you provided implies functionality that is not implemented.\n",
    "boto3_version": "1.9.223",
    "botocore_version": "1.12.253",
    "error": {
        "code": "XNotImplemented",
        "message": "The request you provided implies functionality that is not implemented.",
        "resource": "/sto0003ltest178?requestPayment"
    },
    "response_metadata": {
        "request_id": "1634824409901848",
        "host_id": "12139293",
        "http_status_code": 501,
        "http_headers": {
            "date": "Thu, 21 Oct 2021 13:53:29 GMT",
            "connection": "CLOSE",
            "server": "StorageGRID/11.5.0.1",
            "x-amz-request-id": "1634824409901848",
            "x-amz-id-2": "12139293",
            "content-length": "263",
            "x-ntap-sg-trace-id": "71951b3302703a60",
            "content-type": "application/xml"
        },
        "retry_attempts": 0
    },
    "invocation": {
        "module_args": {
            "name": "sto0003ltest178",
            "state": "present",
            "s3_url": "https://s3.xxxxxxxxxx.fhm.de",
            "validate_certs": false,
            "versioning": false,
            "aws_access_key": "xxxxxxxxxxx",
            "aws_secret_key": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "debug_botocore_endpoint_logs": false,
            "force": false,
            "requester_pays": false,
            "purge_tags": true,
            "ceph": false,
            "ec2_url": null,
            "security_token": null,
            "profile": null,
            "region": null,
            "policy": null,
            "tags": null,
            "encryption": null,
            "encryption_key_id": null
        }
    },
    "_ansible_no_log": false,
    "changed": false
}

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[ansibullbot]

Files identified in the description:

• [plugins/modules/s3_bucket.py](https://github.com/['ansible-collections/amazon.aws', 'ansible-collections/community.aws', 'ansible-collections/community.vmware']/blob/main/plugins/modules/s3_bucket.py)

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

[ansibullbot]

cc @abikouo @jillr @s-hertel @tremble @wimnat
click here for bot help

[tremble]

Thanks for taking the time to report this issue.

This issue was fixed by #154.

The version of s3_bucket shipped with Ansible 2.9 is now very out of date and no longer receiving updates.

You can use a more recent version of the collection by directly installing the collections: https://docs.ansible.com/ansible/latest/user_guide/collections_using.html

[lorenzstorm1]

I already got latest Version installed
bash-4.4# ansible-galaxy collection install amazon.aws:1.5.1
Process install dependency map
ERROR! Cannot meet requirement amazon.aws:1.5.1 as it is already installed at version '2.0.0'. Use --force to overwrite
bash-4.4#

[tremble]

The stacktrace you provided includes:
File \"/tmp/ansible_s3_bucket_payload_ep_yadhp/ansible_s3_bucket_payload.zip/ansible/modules/cloud/amazon/s3_bucket.py\", line 405, in get_bucket_request_payment

ansible/modules/cloud/amazon implies that it's still pulling the module from Ansible 2.9, and the line number (405) matches the line number from Ansible 2.9.11.

2.0.0 included a similar but different issue (#508), so if you've installed 2.0.0 and you're getting a different exception, it's possible you're hitting that.

[tremble]

What you can try is referring to the module by its fully qualified name and using "amazon.aws.s3_bucket" instead of "s3_bucket" or explicitly telling Ansible to give the collection priority by using the collections keyword collections: ['amazon.aws', 'community.aws'].

https://docs.ansible.com/ansible/latest/user_guide/collections_using.html#simplifying-module-names-with-the-collections-keyword