Add support for urllib3 2.2.1 about botocore

[nateprewitt]

To follow up, this was resolved in today's release (botocore 1.34.63) with #3141. Users on Python 3.10+ should be free to control their urllib3 2.x requirements going forward.

[nateprewitt]

Hi everyone, just to reiterate what was discussed in the original issue since it was left out of the description. Botocore cannot support urllib3 2.0 on Python <3.10, this is covered in the urllib3 docs as well. There are no plans currently to change that as it would result in broad impact for AWS customers until they manually pin themselves.

For Python 3.10+, urllib3 2.2.0 had breaking changes that were impacting botocore until the most recent release on Saturday (Feb 17). We're currently reviewing the latest version to confirm there are no other subtle breakages and will update this ticket once we have a clear path to moving the pin. Thanks for your patience!

[nateprewitt]

Can't you just vendor urllib3?

The short answer is no. We already vendored urllib3 for years (botocore>=1.0.0,<=1.12.253) before ultimately removing it from botocore and similarly, was removed from Requests. The primary downsides are size and longevity of releases. Once there's a vulnerability in urllib3, every version of botocore with an older vendored copy becomes either an active security risk or wasted space on PyPI. Keeping the two projects separated allows for more flexibility in choices, as well as not requiring us to make a release for customers to patch their systems.

The other major issue is the size of botocore. While already large, adding vendored dependencies further increases overall size and create problems when packaged for Lambda. The downsides of vendoring outweigh the majority of the benefits which we already have empirical data on. While it's unfortunate we may not have same day parity with new releases, the trade off is stability at the expense of a moderate delay.

[ThiefMaster]

Can't you just vendor urllib3?

[nateprewitt]

Thanks for the feedback, @robd003. To make sure we're on the same page, all versions of urllib3 we support are actively maintained, stable releases. urllib3 1.x is still receiving updates and we're currently ~3 weeks behind on the first stable release of urllib3 for 2.x beyond what we currently support.

Major versioning botocore would make this packaging problem considerably worse. Anyone using the AWSCLI, PynamoDB, Boto3, aiobotocore, or s3fs in conjunction would start breaking with a new major version and you'd still have a urllib3 pin somewhere in that set. We'd also be detracting time from fixing this issue in the current major version of botocore by having to maintain multiple variants of this package. There's no tangible benefit to the end user in that setup.

Thanks for your continued patience while this is addressed.

[kaleb-keny]

God please yes!

Collecting urllib3<1.27,>=1.25.4 (from botocore<1.35.0,>=1.34.48->boto3<2,>=1.9.253->watchtower)
  Using cached urllib3-1.26.18-py2.py3-none-any.whl.metadata (48 kB)

I keep having to do this

pip install watchtower --no-dependencie
pip install boto3 --no-dependencie
pip install botocore --no-dependencie

[brodybits]

One more thing is that the application I am working on also uses requests, which uses urllib3 in turn. I continue to wish that botocore could leave this up to the higher-level application. This was part of my original reqeust.

P.S. Yes I do totally understand the need to keep the functionality stable, avoiding breaking changes from underlying modules such as urllib3. But there will be users who would rather avoid using outdated versions, especially if they may affect any other libraries. I guess this is a situation we cannot avoid given the design of Python & PIP installations.

[nateprewitt]

Just to clarify, Requests has no requirements to update to urllib3 2.0, it's compatible with both the 1.x and 2.x branches. urllib3 1.26.x is still actively maintained and is receiving continued releases for the foreseeable future.

But yes, you're right, a lot of the defensiveness in our pins is due to how Python's packaging infrastructure works. Without the pins we'd have had ~4-5 large scale breakages for boto3 which is the most downloaded python package. In the rare occassions those do happen, it's felt across a large portion of the python ecosystem. We have to weigh safety over total flexibility since there are breakages with a major version bump in dependencies.

[robd003]

Why not have botocore 1.x for ancient Python versions and botocore 2.x for all the modern cool kids who stick with active stable releases and actually maintain their code base?

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.