Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

S3 client tries to connect to http service when endpoint specifies https #4350

Open
1 task
reyescabello opened this issue Nov 15, 2024 · 2 comments
Open
1 task

S3 client tries to connect to http service when endpoint specifies https #4350

reyescabello opened this issue Nov 15, 2024 · 2 comments
Assignees
@adev-code
Labels
bug This issue is a confirmed bug. p3 This is a minor priority issue s3

Comments

@reyescabello
Copy link

Describe the bug

Creating an s3 client, when we use the 'delete_object' method, instead of making the https call, it makes the http call.

Regression Issue

  • Select this option if this issue appears to be a regression.

Expected Behavior

The client should connect to the endpoint_url when the url is https.

Current Behavior

The client tries to connect to the endpoint_url with an http connection instead of https.

The rest of the calls to the endpoint with the s3 client work correctly with the same configuration, it happens to us only in the case of the 'delete_object'. The parameter “use_ssl” is set to True by default.

DEBUG | 2024-11-14 11:15:14,367 | botocore.endpoint | Sending http request: <AWSPreparedRequest stream_output=False, method=DELETE, url=http://<external-service>, headers={'User-Agent': b'Boto3/1.34.150 md/Botocore#1.34.150 ua/2.0 os/linux#5.15.146.1-microsoft-standard-WSL2 md/arch#x86_64 lang/python#3.11.5 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.150', ...>
DEBUG | 2024-11-14 11:15:14,537 | botocore.endpoint | Exception received when sending HTTP request.
Traceback (most recent call last):
  File "/urllib3/connection.py", line 196, in _new_conn
    sock = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

Traceback (most recent call last):
  File "/botocore/httpsession.py", line 464, in send
    urllib_response = conn.urlopen(
                      ^^^^^^^^^^^^^
  File "/urllib3/connectionpool.py", line 843, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/urllib3/util/retry.py", line 449, in increment
    raise reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/urllib3/util/util.py", line 39, in reraise
    raise value
  File "/urllib3/connectionpool.py", line 789, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/urllib3/connectionpool.py", line 495, in _make_request
    conn.request(
  File "/botocore/awsrequest.py", line 96, in request
    rval = super().request(method, url, body, headers, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/urllib3/connection.py", line 398, in request
    self.endheaders()
  File "/http/client.py", line 1281, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/botocore/awsrequest.py", line 123, in _send_output
    self.send(msg)
  File "/botocore/awsrequest.py", line 223, in send
    return super().send(str)
           ^^^^^^^^^^^^^^^^^
  File "/http/client.py", line 979, in send
    self.connect()
  File "urllib3/connection.py", line 236, in connect
    self.sock = self._new_conn()
                ^^^^^^^^^^^^^^^^
  File "/urllib3/connection.py", line 211, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPConnection object at 0x7fc8c1e166d0>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/botocore/endpoint.py", line 279, in _do_get_response
    http_response = self._send(request)
                    ^^^^^^^^^^^^^^^^^^^
  File "/botocore/endpoint.py", line 375, in _send
    return self.http_session.send(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/botocore/httpsession.py", line 493, in send
    raise EndpointConnectionError(endpoint_url=request.url, error=e)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://<external-service>"

Reproduction Steps

import boto3

endpoint_url = "https://<external-service>"
s3_client = boto3.client("s3", aws_access_key_id="access-key-id", endpoint_url=endpoint_url, region_name="region-name")

s3_client.delete_object(Bucket="bucket-name", Key="key")

Possible Solution

No response

Additional Information/Context

No response

SDK version used

1.34.150

Environment details (OS name and version, etc.)

5.15.146.1 WSL2 kernel, python 3.11.5, boto3 1.34.150
@reyescabello reyescabello added bug This issue is a confirmed bug. needs-triage This issue or PR still needs to be triaged. labels Nov 15, 2024
@reyescabello reyescabello changed the title (short issue description) S3 client tries to connect to http service when endpoint specifies https Nov 15, 2024
@adev-code adev-code self-assigned this Nov 19, 2024
@adev-code adev-code added investigating This issue is being investigated and/or work is in progress to resolve the issue. s3 p3 This is a minor priority issue and removed needs-triage This issue or PR still needs to be triaged. labels Nov 19, 2024
@adev-code
Copy link

Hi @reyescabello, thanks for reaching out. I have replicated from my side and was not able to get the same issue. I tried with an S3 endpoint and declaring HTTPS and I was able to connect via HTTPS. For further look, could you please provide the full debug logs by adding the line: boto3.set_stream_logger('') under import boto3 and redacting any sensitive information. Also, please ensure that the endpoint you are trying to connect to has HTTPS. Thank you.

@adev-code adev-code added response-requested Waiting on additional information or feedback. and removed investigating This issue is being investigated and/or work is in progress to resolve the issue. labels Nov 20, 2024
@reyescabello
Copy link
Author

Hi @adev-code, thanks so much for your quick response. Are these the logs you need to take a look at? Let me know if there’s anything else I should add.

Thanks again!

DEBUG | 2024-11-21 12:42:22,960 | botocore.hooks | Event before-parameter-build.s3.DeleteObject: calling handler <function validate_bucket_name at 0x7f7524b61bc0>
2024-11-21 12:42:22,960 botocore.hooks [DEBUG] Event before-parameter-build.s3.DeleteObject: calling handler <function validate_bucket_name at 0x7f7524b61bc0>
DEBUG | 2024-11-21 12:42:22,960 | botocore.hooks | Event before-parameter-build.s3.DeleteObject: calling handler <function remove_bucket_from_url_paths_from_model at 0x7f7524b63d80>
2024-11-21 12:42:22,960 botocore.hooks [DEBUG] Event before-parameter-build.s3.DeleteObject: calling handler <function remove_bucket_from_url_paths_from_model at 0x7f7524b63d80>
DEBUG | 2024-11-21 12:42:22,961 | botocore.hooks | Event before-parameter-build.s3.DeleteObject: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x7f750b2e6e90>>
2024-11-21 12:42:22,961 botocore.hooks [DEBUG] Event before-parameter-build.s3.DeleteObject: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x7f750b2e6e90>>
DEBUG | 2024-11-21 12:42:22,961 | botocore.hooks | Event before-parameter-build.s3.DeleteObject: calling handler <bound method ClientCreator._inject_s3_input_parameters of <botocore.client.ClientCreator object at 0x7f7523f41990>>
2024-11-21 12:42:22,961 botocore.hooks [DEBUG] Event before-parameter-build.s3.DeleteObject: calling handler <bound method ClientCreator._inject_s3_input_parameters of <botocore.client.ClientCreator object at 0x7f7523f41990>>
DEBUG | 2024-11-21 12:42:22,961 | botocore.hooks | Event before-parameter-build.s3.DeleteObject: calling handler <function generate_idempotent_uuid at 0x7f7524b619e0>
2024-11-21 12:42:22,961 botocore.hooks [DEBUG] Event before-parameter-build.s3.DeleteObject: calling handler <function generate_idempotent_uuid at 0x7f7524b619e0>
DEBUG | 2024-11-21 12:42:22,961 | botocore.hooks | Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7f7524b63f60>
2024-11-21 12:42:22,961 botocore.hooks [DEBUG] Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7f7524b63f60>
DEBUG | 2024-11-21 12:42:22,961 | botocore.hooks | Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7f750b2e6e90>>
2024-11-21 12:42:22,961 botocore.hooks [DEBUG] Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7f750b2e6e90>>
DEBUG | 2024-11-21 12:42:22,961 | botocore.regions | Calling endpoint provider with parameters: {'Bucket': '<bucket>', 'Region': '<region>', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://<external-service>', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': False, 'Key': '<key>', 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
2024-11-21 12:42:22,961 botocore.regions [DEBUG] Calling endpoint provider with parameters: {'Bucket': '<bucket>', 'Region': '<region>', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://<external-service>', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': False, 'Key': '<key>', 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
DEBUG | 2024-11-21 12:42:22,962 | botocore.regions | Endpoint provider result: https://<external-service>
2024-11-21 12:42:22,962 botocore.regions [DEBUG] Endpoint provider result: https://<external-service>
DEBUG | 2024-11-21 12:42:22,962 | botocore.regions | Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None"
2024-11-21 12:42:22,962 botocore.regions [DEBUG] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None"
DEBUG | 2024-11-21 12:42:22,962 | botocore.regions | Selected auth type "v4" as "v4" with signing context params: {'region': '<region>', 'signing_name': 's3', 'disableDoubleEncoding': True}
2024-11-21 12:42:22,962 botocore.regions [DEBUG] Selected auth type "v4" as "v4" with signing context params: {'region': '<region>', 'signing_name': 's3', 'disableDoubleEncoding': True}
DEBUG | 2024-11-21 12:42:22,962 | botocore.hooks | Event before-call.s3.DeleteObject: calling handler <function add_expect_header at 0x7f7524b61f80>
2024-11-21 12:42:22,962 botocore.hooks [DEBUG] Event before-call.s3.DeleteObject: calling handler <function add_expect_header at 0x7f7524b61f80>
DEBUG | 2024-11-21 12:42:22,962 | botocore.hooks | Event before-call.s3.DeleteObject: calling handler <bound method S3ExpressIdentityResolver.apply_signing_cache_key of <botocore.utils.S3ExpressIdentityResolver object at 0x7f750b2eaf90>>
2024-11-21 12:42:22,962 botocore.hooks [DEBUG] Event before-call.s3.DeleteObject: calling handler <bound method S3ExpressIdentityResolver.apply_signing_cache_key of <botocore.utils.S3ExpressIdentityResolver object at 0x7f750b2eaf90>>
DEBUG | 2024-11-21 12:42:22,962 | botocore.hooks | Event before-call.s3.DeleteObject: calling handler <function add_recursion_detection_header at 0x7f7524b60220>
2024-11-21 12:42:22,962 botocore.hooks [DEBUG] Event before-call.s3.DeleteObject: calling handler <function add_recursion_detection_header at 0x7f7524b60220>
DEBUG | 2024-11-21 12:42:22,963 | botocore.hooks | Event before-call.s3.DeleteObject: calling handler <function add_query_compatibility_header at 0x7f7524b784a0>
2024-11-21 12:42:22,963 botocore.hooks [DEBUG] Event before-call.s3.DeleteObject: calling handler <function add_query_compatibility_header at 0x7f7524b784a0>
DEBUG | 2024-11-21 12:42:22,963 | botocore.hooks | Event before-call.s3.DeleteObject: calling handler <function inject_api_version_header_if_needed at 0x7f7524b634c0>
2024-11-21 12:42:22,963 botocore.hooks [DEBUG] Event before-call.s3.DeleteObject: calling handler <function inject_api_version_header_if_needed at 0x7f7524b634c0>
DEBUG | 2024-11-21 12:42:22,963 | botocore.endpoint | Making request for OperationModel(name=DeleteObject) with params: {'url_path': '<url_path>', 'query_string': {}, 'method': 'DELETE', 'headers': {'User-Agent': 'Boto3/1.35.58 md/Botocore#1.35.58 ua/2.0 os/linux#5.15.146.1-microsoft-standard-WSL2 md/arch#x86_64 lang/python#3.11.5 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.35.58'}, 'body': b'', 'auth_path': '<auth_path>', 'url': 'http://<external-service>', 'context': {'client_region': 'region', 'client_config': <botocore.config.Config object at 0x7f750b2d4410>, 'has_streaming_input': False, 'auth_type': 'v4', 'unsigned_payload': None, 's3_redirect': {'redirected': False, 'bucket': '<bucket>', 'params': {'Bucket': '<bucket>', 'Key': '<key>'}}, 'input_params': {'Bucket': '<bucket>', 'Key': '<key>'}, 'signing': {'region': '<region>', 'signing_name': 's3', 'disableDoubleEncoding': True}, 'endpoint_properties': {'authSchemes': [{'disableDoubleEncoding': True, 'name': 'sigv4', 'signingName': 's3', 'signingRegion': 'region'}]}}}
2024-11-21 12:42:22,963 botocore.endpoint [DEBUG] Making request for OperationModel(name=DeleteObject) with params: {'url_path': '<url_path>', 'query_string': {}, 'method': 'DELETE', 'headers': {'User-Agent': 'Boto3/1.35.58 md/Botocore#1.35.58 ua/2.0 os/linux#5.15.146.1-microsoft-standard-WSL2 md/arch#x86_64 lang/python#3.11.5 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.35.58'}, 'body': b'', 'auth_path': '<auth_path>', 'url': 'http://<external-service>', 'context': {'client_region': 'region', 'client_config': <botocore.config.Config object at 0x7f750b2d4410>, 'has_streaming_input': False, 'auth_type': 'v4', 'unsigned_payload': None, 's3_redirect': {'redirected': False, 'bucket': '<bucket>', 'params': {'Bucket': '<bucket>', 'Key': '<key>'}}, 'input_params': {'Bucket': '<bucket>', 'Key': '<key>'}, 'signing': {'region': '<region>', 'signing_name': 's3', 'disableDoubleEncoding': True}, 'endpoint_properties': {'authSchemes': [{'disableDoubleEncoding': True, 'name': 'sigv4', 'signingName': 's3', 'signingRegion': 'region'}]}}}
DEBUG | 2024-11-21 12:42:22,963 | botocore.hooks | Event request-created.s3.DeleteObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f750b2d43d0>>
2024-11-21 12:42:22,963 botocore.hooks [DEBUG] Event request-created.s3.DeleteObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f750b2d43d0>>
DEBUG | 2024-11-21 12:42:22,963 | botocore.hooks | Event choose-signer.s3.DeleteObject: calling handler <function set_operation_specific_signer at 0x7f7524b61800>
2024-11-21 12:42:22,963 botocore.hooks [DEBUG] Event choose-signer.s3.DeleteObject: calling handler <function set_operation_specific_signer at 0x7f7524b61800>
DEBUG | 2024-11-21 12:42:22,963 | botocore.hooks | Event before-sign.s3.DeleteObject: calling handler <function remove_arn_from_signing_path at 0x7f7524b63ec0>
2024-11-21 12:42:22,963 botocore.hooks [DEBUG] Event before-sign.s3.DeleteObject: calling handler <function remove_arn_from_signing_path at 0x7f7524b63ec0>
DEBUG | 2024-11-21 12:42:22,963 | botocore.hooks | Event before-sign.s3.DeleteObject: calling handler <bound method S3ExpressIdentityResolver.resolve_s3express_identity of <botocore.utils.S3ExpressIdentityResolver object at 0x7f750b2eaf90>>
2024-11-21 12:42:22,963 botocore.hooks [DEBUG] Event before-sign.s3.DeleteObject: calling handler <bound method S3ExpressIdentityResolver.resolve_s3express_identity of <botocore.utils.S3ExpressIdentityResolver object at 0x7f750b2eaf90>>
DEBUG | 2024-11-21 12:42:22,964 | botocore.auth | Calculating signature using v4 auth.
2024-11-21 12:42:22,964 botocore.auth [DEBUG] Calculating signature using v4 auth.
DEBUG | 2024-11-21 12:42:22,964 | botocore.auth | CanonicalRequest:
DELETE
<url_path>

host:<host>
x-amz-content-sha256:<>
x-amz-date:20241121T114222Z

host;x-amz-content-sha256;x-amz-date
e3b0c23145fc1c149afbf4c8996fb12345ae41e4649
2024-11-21 12:42:22,964 botocore.auth [DEBUG] CanonicalRequest:
DELETE
<url_path>

host:<host>
x-amz-content-sha256:<>
x-amz-date:20241121T114222Z

host;x-amz-content-sha256;x-amz-date
e3b0c12345fc1c149afbf4c3454b855
DEBUG | 2024-11-21 12:42:22,964 | botocore.auth | StringToSign:
AWS4-HMAC-SHA256
20241121T114222Z
20241121/region/s3/aws4_request

2024-11-21 12:42:22,964 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20241121T114222Z
20241121/region/s3/aws4_request
14256912702bd4f5744c2cd7660ac2b2f9458461181
DEBUG | 2024-11-21 12:42:22,964 | botocore.auth | Signature:
signature
2024-11-21 12:42:22,964 botocore.auth [DEBUG] Signature:
signature
DEBUG | 2024-11-21 12:42:22,965 | botocore.hooks | Event request-created.s3.DeleteObject: calling handler <function add_retry_headers at 0x7f7524b63ce0>
2024-11-21 12:42:22,965 botocore.hooks [DEBUG] Event request-created.s3.DeleteObject: calling handler <function add_retry_headers at 0x7f7524b63ce0>
DEBUG | 2024-11-21 12:42:22,965 | botocore.endpoint | Sending http request: <AWSPreparedRequest stream_output=False, method=DELETE, url=http://<external-service>, headers={'User-Agent': b'Boto3/1.35.58 md/Botocore#1.35.58 ua/2.0 os/linux#5.15.146.1-microsoft-standard-WSL2 md/arch#x86_64 lang/python#3.11.5 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.35.58', 'X-Amz-Date': b'20241121T114222Z', 'X-Amz-Content-SHA256': b'<>', 'Authorization': b'<authorization>', 'amz-sdk-invocation-id': b'4fc1d4b6-9b5c-4281-a9ec-e3cb2679bbbf', 'amz-sdk-request': b'attempt=1', 'Content-Length': '0'}>
2024-11-21 12:42:22,965 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=DELETE, url=http://<external-service>, headers={'User-Agent': b'Boto3/1.35.58 md/Botocore#1.35.58 ua/2.0 os/linux#5.15.146.1-microsoft-standard-WSL2 md/arch#x86_64 lang/python#3.11.5 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.35.58', 'X-Amz-Date': b'20241121T114222Z', 'X-Amz-Content-SHA256': b'<>', 'Authorization': b'<authorization>', 'amz-sdk-invocation-id': b'4fc1d4b6-9b5c-4281-a9ec-e3cb2679bbbf', 'amz-sdk-request': b'attempt=1', 'Content-Length': '0'}>
DEBUG | 2024-11-21 12:42:23,094 | botocore.endpoint | Exception received when sending HTTP request.
Traceback (most recent call last):
  File "/urllib3/connection.py", line 199, in _new_conn
    sock = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

@github-actions github-actions bot removed the response-requested Waiting on additional information or feedback. label Nov 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@adev-code adev-code

Labels
bug This issue is a confirmed bug. p3 This is a minor priority issue s3
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@adev-code @reyescabello