Allow urllib3 2.0.0 to be installed #2926
Comments
epicserve
added
feature-request
|
Hi @epicserve, thanks for reaching out. The pin is currently intentional while we wait for the major version bump to stabilize. We intend to move the pin shortly after Requests does to ensure we're not negatively impacting users. We'll keep this issue updated and feel free to follow along in psf/requests#6432 as well if you like. |
nateprewitt
added
dependencies
|
I started getting this error message in my Lambda logs after doing a recent deploy which has to be pulling in urllib3 >=2.0.0 which is coming from https://github.com/boto/botocore/blob/master/botocore/httpsession.py#L52 try:
from urllib3.util.ssl_ import DEFAULT_CIPHERS
except ImportError:
# Defer to system configuration starting with
# urllib3 2.0. This will choose the ciphers provided by
# Openssl 1.1.1+ or secure system defaults.
DEFAULT_CIPHERS = Nonewhich is concerning because urllib3 removed What I'm not understanding is why the |
|
Same as @jplock here. I started seeing this error today on a AWS Lambda Function after I made deployment. Tried rolling back to previous states, but error remained. Lambda uses these libraries: |
|
Hi @pistolilla, if you're still encountering this error you'll want to add this to your dependency closure in the same way you're specifying the dependencies you've listed above. Make sure this is the last entry: One of your dependencies is pulling in the latest version of urllib3 which contains a handful of breaking changes. |
|
|
tim-finnigan
added
feature-request
|
Hi @edgarrmondragon, we're aware of the Requests update. This has surfaced a number of unexpected breakages within AWS infrastructure that we're currently looking at options to remedy. We'd advise users to pin to |
Look like new version of requests is pulling newer versions of urllib3 which are not comptible with botocore. Ref: boto/botocore#2926 Ref: psf/requests#6443 Ref: urllib3/urllib3#2168
Look like new version of requests is pulling newer versions of urllib3 which are not comptible with botocore. Ref: boto/botocore#2926 Ref: psf/requests#6443 Ref: urllib3/urllib3#2168
Storing the SBOM dependecy tree in the database turned out to not be the right decision due to performance issues at scale. Previous changes to improve performance reduced the usage of the dependency table to just generating SBOM reports. This change moves the storage of the dependency tree from the database to S3, removing the need to deconstruct and reconstruct the tree and removes the overhead that goes along with that. The S3 key is structured so that other SBOM file formats, such as SPDX or CycloneDX, could also be stored alongside. - Update engine SBOM processing to write the dependency tree to a JSON file in S3 instead of the database. The dependency tree is still processed in order to store component and license information in the database. - Update sbom_report Lambda to pull the dependency tree JSON file from S3. If the file is not found in S3 it falls back to pulling the tree from the database. This allows for the gradual migration of the dependency tree data from the database to S3 as new scans are run and old scans are purged by the db_cleanup Lambda. - Update the db_cleanup to identify and remove dependency files that were orphaned when their associated scans were deleted. Deleting scans via the ORM will clean up the dependency files from S3. This is a backstop just in case a scan is deleted directly or something else happens that prevents the cleanup at deletion time from succeeding. - Update localstack config to add an S3 bucket that can store dependency tree files during local testing and update AWSConnect in artemislib so that it can be configured to use this S3 bucket for scan data. - Update IAM permssions in Terraform configuration so that the right things can read and write to the scans/ portion of the S3 bucket. - Add sbom_dependency_migration utility to migrate the dependency trees from existing scans from the database to S3. This is useful for testing and also if there are key scans that need the performance improvement and can't wait for the scan replacement and cleanup process. Unrelated to the SBOM dependency changes but included out of necessity: - Pin urllib3 version to 1.x because of compatability issue with botocore: boto/botocore#2926
* Move dependency tree storage from the database to S3 Storing the SBOM dependency tree in the database turned out to not be the right decision due to performance issues at scale. Previous changes to improve performance reduced the usage of the dependency table to just generating SBOM reports. This change moves the storage of the dependency tree from the database to S3, removing the need to deconstruct and reconstruct the tree and removes the overhead that goes along with that. The S3 key is structured so that other SBOM file formats, such as SPDX or CycloneDX, could also be stored alongside. - Update engine SBOM processing to write the dependency tree to a JSON file in S3 instead of the database. The dependency tree is still processed in order to store component and license information in the database. - Update sbom_report Lambda to pull the dependency tree JSON file from S3. If the file is not found in S3 it falls back to pulling the tree from the database. This allows for the gradual migration of the dependency tree data from the database to S3 as new scans are run and old scans are purged by the db_cleanup Lambda. - Update the db_cleanup to identify and remove dependency files that were orphaned when their associated scans were deleted. Deleting scans via the ORM will clean up the dependency files from S3. This is a backstop just in case a scan is deleted directly or something else happens that prevents the cleanup at deletion time from succeeding. - Update localstack config to add an S3 bucket that can store dependency tree files during local testing and update AWSConnect in artemislib so that it can be configured to use this S3 bucket for scan data. - Update IAM permissions in Terraform configuration so that the right things can read and write to the scans/ portion of the S3 bucket. - Add sbom_dependency_migration utility to migrate the dependency trees from existing scans from the database to S3. This is useful for testing and also if there are key scans that need the performance improvement and can't wait for the scan replacement and cleanup process. Unrelated to the SBOM dependency changes but included out of necessity: - Pin urllib3 version to 1.x because of compatibility issue with botocore: boto/botocore#2926
Look like new version of requests is pulling newer versions of urllib3 which are not comptible with botocore. Ref: boto/botocore#2926 Ref: psf/requests#6443 Ref: urllib3/urllib3#2168
We recently received the following error in the lambbda cloudwatch logs.. ``` Runtime.ImportModuleError: Unable to import module 'app': cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_' ``` This seems like a library conflict between botocore and urllib3 so we update according to workaround proposed at these locations: * https://stackoverflow.com/questions/76414514/cannot-import-name-default-ciphers-from-urllib3-util-ssl-on-aws-lambda-us * boto/botocore#2926 * psf/requests#6432
We recently received the following error in the lambbda cloudwatch logs.. ``` Runtime.ImportModuleError: Unable to import module 'app': cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_' ``` This seems like a library conflict between botocore and urllib3 so we update according to workaround proposed at these locations: * https://stackoverflow.com/questions/76414514/cannot-import-name-default-ciphers-from-urllib3-util-ssl-on-aws-lambda-us * boto/botocore#2926 * psf/requests#6432
|
Is there anything we can do to help unblock an upgrade to a more recent version of |
|
Hi @brodybits, botocore currently supports |
|
Given we've resolved the 2.0 move on Python 3.10+ and don't have any immediate plans to add support for earlier versions, it's probably best to close this issue. For tracking catching up to the latest minor release we can have a separate feature request, that will help with some of the confusion this thread has had in the last few months. Thanks everyone for your participation and patience! |
|
This issue is now closed. Comments on closed issues are hard for our team to see. |
|
@nateprewitt can you please give us a link to the feature request for the update to urllib >= 2.1? |
|
@SamStephens I just filed #3138 |
Describe the feature
There is a new version of urllib3 (v2.0.0) that can't be installed in projects that use botocore because, in the setup.py file, it has the version restriction
urllib3>=1.25.4,<1.27.Use Case
So other projects that want to upgrade to urllib3 2.0.0 can without being blocked by the botocore library.
Proposed Solution
No response
Other Information
No response
Acknowledgements
SDK version used
latest
Environment details (OS name and version, etc.)
all
The text was updated successfully, but these errors were encountered: