boto3 v1.36 is causing issues (S3)

[Zerotask]

After we updated our dependencies, specifically after the update from boto3 to the lastest 1.36 version, we noticed that some file related tests fail. Also manually uploading an image stopped working.
So there must be some breaking changes regarding S3.

boto3 Release Notes

[aq1]

Same here. I had to manually specify version in requirements.txt in my project.

boto3==1.35.54
botocore==1.35.54

[oliverhaas]

We observed probably the same error when trying to upload files during e.g. collectstatic:

botocore.exceptions.ClientError: An error occurred (SignatureDoesNotMatch) when calling the PutObject operation: Invalid argument.

Our setup is a little unusual (Google Cloud Storage with the S3 interface), but for us using signature_version = "s3" also solved the issue, but AWS mostly does not support this signature_version anymore, so ymmv.

Here is the related boto3 issue boto/boto3#4400.

[StefanBrand]

I get this error when calling File.save: ClientError: An error occurred (AccessDenied) when calling the PutObject operation: None

Resolved by downgrading to boto3==1.35.99

[TheDJVG]

Hitting the same problem here too. Reading the official documentation I get the impression that setting the env. vars AWS_REQUEST_CHECKSUM_CALCULATION and AWS_RESPONSE_CHECKSUM_VALIDATION to WHEN_REQUIRED should fix this, however so far it does not help for me (using Ceph radosgw here).

Is it possible that django-storages is not passing these through or is boto3 not reading them. We might need to expose these settings just like how other AWS related variables are exposed through Django settings.

[9128305]

Hitting the same problem here too. Reading the official documentation I get the impression that setting the env. vars AWS_REQUEST_CHECKSUM_CALCULATION and AWS_RESPONSE_CHECKSUM_VALIDATION to WHEN_REQUIRED should fix this, however so far it does not help for me (using Ceph radosgw here).

Is it possible that django-storages is not passing these through or is boto3 not reading them. We might need to expose these settings just like how other AWS related variables are exposed through Django settings.

Try using a custom botocore Config, but you also need this pr boto/s3transfer#329.

I tried with it and it works now.

[TheDJVG]

Hitting the same problem here too. Reading the official documentation I get the impression that setting the env. vars AWS_REQUEST_CHECKSUM_CALCULATION and AWS_RESPONSE_CHECKSUM_VALIDATION to WHEN_REQUIRED should fix this, however so far it does not help for me (using Ceph radosgw here).
Is it possible that django-storages is not passing these through or is boto3 not reading them. We might need to expose these settings just like how other AWS related variables are exposed through Django settings.

Try using a custom botocore Config, but you also need this pr boto/s3transfer#329.

I tried with it and it works now.

Thanks! I will have to try this release. I only tested when they just released 1.36 (and tests started to fail).

[terencehonles]

surprisingly it seems we're not seeing this issue, but subscribing to this thread nonetheless.

Does anyone want to share a minimal reproducible config?

[9128305]

It does not work for 3rd party cloud providers

[9128305]

Thanks! I will have to try this release. I only tested when they just released 1.36 (and tests started to fail).

Released.

[TheDJVG]

Thanks! I will have to try this release. I only tested when they just released 1.36 (and tests started to fail).

Released.

Confirmed this fixed the problem and with the env. vars AWS_REQUEST_CHECKSUM_CALCULATION and AWS_RESPONSE_CHECKSUM_VALIDATION set to WHEN_REQUIRED makes it work again at least with ceph radosgw.