When uploading, checking 'Expect' header results in "TypeError: sequence item 0: expected str instance, bytes found" about botocore

[lorenzomicheli]

I am facing the same issue using AWS SAM version 1.108.0 and Python 3.11. When running sam package/deploy on one of the stock SAM template I get:

		Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-1mk1hlsoyqzcb
		A different default S3 bucket can be set in samconfig.toml
		Or by specifying --s3-bucket explicitly.
Error: Unable to upload artifact hello_world referenced by CodeUri parameter of HelloWorldFunction resource.
An HTTP Client raised an unhandled exception: sequence item 0: expected str instance, bytes found

[MehdiPoustchi]

install urllib3==2.0.7 fixed mine.
rest of the packages are still the latest.

[shairubin]

same issue here
but upgrade to 1.34.33 solved it

[konrad0101]

I get this issue when using AWS Lambda (Python 3.12).

[fredlllll]

i noticed this happens when using urllib 2.* instead of 1.26.18 for me with boto3 1.34.33

[nateprewitt]

Hi everyone,

Coming back to our previous responses, we've confirmed this issue is specific to urllib3 2.2.0. We do not support this version of urllib3 and specifically have botocore pinned to urllib3<2.1 due to breaking changes made in that minor version release.

We've confirmed that while the upload_file workflow works as expected with urllib3 2.1.0, it broke with the release of 2.2.0 due to what appeared to be an innocuous type change. Specifically on this line, all headers are now getting coverted to urllib3's HTTPHeaderDict which wasn't the case before. This has implicit requirements that all headers have string values, while previously urllib3 has supported bytes or headers (Requests and botocore prefer bytes as they are more correct when serializing over the wire).

This is the current root cause of the breakage, I've opened a ticket (urllib3/urllib3#3343) in the urllib3 tracker to discuss options. For now, installing any version of boto3 normally will work as expected. The only way to hit this case it to force a urllib3 upgrade beyond botocore's defined support range, or install another project that explicitly requires urllib3>=2.2.0.

For the time being the remediation step is to ensure you are using this in your installation requirements. There is no need to pin boto3 or botocore versions:

urllib3<2.2

If anyone can help identify how urllib3 2.2.0 go into their dependency closure, that may be helpful. A common way users break their installations is using --upgrade independently of installing their dependencies.

[wadefletch]

Uninstalling Homebrew SAM and using the real installer fixed this for me.

[cbsmith]

We hit this one too. Solution was to revert to an older version. This shouldn't have made it out to release.

[wzxu]

Using the latest version here (installed via pymongo[aws] so I don't control the version).
Was working fine up until yesterday and today I also ran into it.

One interesting observation: Because I'm curious what response caused it to break, I launched Proxyman which is a packet inspection tool, but before I went and checked the packet, I noticed the error went away. I tried closing Proxyman, and the error occurred. Launching it again, and I was able to successfully deploy my code without even changing the library versions.

[gihanw]

@gihanw try installing urllib3<2.2.0. We're trying to test if the issue is in the latest urllib3 release.

I was already on 1.26.18. But, it didn't work.

Finally this worked. aws/aws-sam-cli#6629 (comment)

@nateprewitt , the previous installation was via brew and was the same version.
SAM CLI, version 1.108.0

We have multiple stacks and it worked for some stacks. But, not this particular one. if that information is going to help.

[narenderAwen]

Ran into the same issue, force installed boto3==1.34.33 , which reduced urllib3 to 2.0.7, and it worked

[steffenweber]

Just ran into this issue using botocore 1.34.35. Downgrading urllib3 from version 2.2.0 to 2.1.0 fixed the issue.

[pengdadako]

We hit this one too. 1.34.31

[drimal]

I also ran into the same issue. urllib3==2.0.7 seems to fixes it but have another sagemaker dependency conflict. Is this related to the CVE-2024-2626 security updates? https://aws.amazon.com/security/security-bulletins/AWS-2024-001/

[LLMCoder2023]

This worked:
!pip install urllib3==2.0.7 --force-reinstall

Throws this warning in SageMaker Studio thought: which is weird in general

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
distributed 2022.7.0 requires tornado<6.2,>=6.0.3, but you have tornado 6.4 which is incompatible.
sagemaker 2.199.0 requires urllib3<1.27, but you have urllib3 2.0.7 which is incompatible.

[nusantara-self]

Also getting this error with Lambda Python 3.12. What's the best way to solve this in Lambda? Create custom layer with another version of boto3?

Hello @RomainBsb, you need to make sure your Lambda layers do not use urllib3==2.2.0. For example, if you made a layer for requests, requests will use latest 2.2.0 as a dependency. As a result, this will cause your lambda to run into issues with botocore.

To make sure, you don't encounter any issues you can install requests with the --no-dependencies flag and its dependencies manually while making sure urllib3<2.2.0.

For example,

        if [ "$pkg" == "requests" ]; then
        # Force urllib3 to version 2.1.0, as 2.2.0 causes issues with botocore
        pip install --platform manylinux2014_x86_64 \
                --target="$pkg_dir/python" \
                --implementation cp \
                --python-version $python_version \
                --only-binary=:all: --upgrade \
                --no-dependencies  \
                requests
        pip install --platform manylinux2014_x86_64 \
            --target="$pkg_dir/python" \
            --implementation cp \
            --python-version $python_version \
            --only-binary=:all: --upgrade \
            urllib3==2.1.0
        pip install --platform manylinux2014_x86_64 \
            --target="$pkg_dir/python" \
            --implementation cp \
            --python-version $python_version \
            --only-binary=:all: --upgrade \
            idna
        pip install --platform manylinux2014_x86_64 \
            --target="$pkg_dir/python" \
            --implementation cp \
            --python-version $python_version \
            --only-binary=:all: --upgrade \
            charset-normalizer
        pip install --platform manylinux2014_x86_64 \
            --target="$pkg_dir/python" \
            --implementation cp \
            --python-version $python_version \
            --only-binary=:all: --upgrade \
            certifi
        fi

[nateprewitt]

Hi everyone, thanks for reaching out. It's not immediately clear why you're seeing issues here, this code has been in Botocore for ~10 years (ref). The reason we fall back to an empty byte string for comparison is header values should always be bytes, that's the existing typing contract.

We'll need some more information to investigate further. If you can answer these questions, it will help determine what's happening.

1. Did everyone start seeing this recently? If so, can you provide the version you started seeing this issue?
2. In the cases people saw it stop with a rollback, what version is stable?
3. Is everyone using a non-AWS endpoint for communicating with an s3 emulator?
4. What version of Python and urllib3 are you using?

[delabj]

I'm running into this issue as well, seemingly at random when using the R kernels in a Sagemaker Notebook Instance to send a file to S3 using the CLI on AWS.

We don't control the versioning of the base AWS Sagemaker kernels but we're using Amazon Linux 2, Jupyter Lab 3
(notebook-al2-v2)

I used ```conda list```` in an instance that failed today and am sharing

Did everyone start seeing this recently? If so, can you provide the version you started seeing this issue?

This issue started happening very recently, we noticed it yesterday (2024-02-02) It's listing botocore as using 1.34.32

In the cases people saw it stop with a rollback, what version is stable?

Have not attempted.

Is everyone using a non-AWS endpoint for communicating with an s3 emulator?

I believe I'm on an AWS endpoint, as I'm running this in a notebook instance in Sagemaker.

What version of Python and urllib3 are you using?

Python is listed as 3.11.5, but I'm accessing the CLI from the system() command in R.
URLlib is 1.26.18

I had been digging into if this was a CLI change but a coworker found this issue and the error messages match.

[andersha]

I can confirm that I have the same issue and that running Proxyman "solves it". Could it be some sort of certificate issue somewhere since running the call through Proxyman works?

[nateprewitt]

For anyone who's recently been able to produce this and upgrading fixed it, can you try installing urllib3==2.2.0? The only lead we have currently is their recent release on 2024-01-31.

We do not officially support this version yet, but for users upgrading all dependencies to the latest version, they may end up with dependencies outside of the supported range.

[gihanw]

For anyone who's recently been able to produce this and upgrading fixed it, can you try installing urllib3==2.2.0? The only lead we have currently is their recent release on 2024-01-31.

We do not officially support this version yet, but for users upgrading all dependencies to the latest version, they may end up with dependencies outside of the supported range.

I'm having this issue right now. tried installing urllib3=2.2.0. But, the issue is still there :(

[nateprewitt]

@gihanw try installing urllib3<2.2.0. We're trying to test if the issue is in the latest urllib3 release.

[dylan0356]

Same issue here,

Downgrading botocore to 1.34.33 solved it. urlib3 was below version 2.2.0 and it was still broken.

[princebansal]

I am using docker image built by cog. Running this docker image in a GCP VM. Tried multiple solutions above but couldn't fix it. Happy to connect to over discord to resolve this quickly. https://discord.gg/4NyDaSdx

[CneelyLip]

install urllib3==2.0.7 fixed mine. rest of the packages are still the latest.

This worked for me as well

[jbaranski]

Ran sam deploy to deploy a Lambda function. Received error:

error: an http client raised an unhandled exception: sequence item 0: expected str instance, bytes found

Doing #3111 (comment) fixed the issue for me too.

Uninstalling Homebrew SAM and using the real installer fixed this for me.

[zicanl-amazon]

Hi are facing the issue when boto3 1.34.34 and urllib3 1.26.18. Any quick fix?

[a-sane]

I am using docker image built by cog. Running this docker image in a GCP VM. Tried multiple solutions above but couldn't fix it. Happy to connect to over discord to resolve this quickly. https://discord.gg/4NyDaSdx

@princebansal

consider adding something like the following lines to your cog.yaml:

  run:
    - pip3 install boto3==1.33.11
    - pip3 install urllib3==1.26.17

for some reason, specifying the same versions in the python_packages section didn't resolve the issue for me. It seems to be related to a sub-dependency problem.

[zicanl-amazon]

install urllib3==2.0.7 fixed mine. rest of the packages are still the latest.

Could you please tell what is the version of botocore?

[CneelyLip]

install urllib3==2.0.7 fixed mine. rest of the packages are still the latest.

Could you please tell what is the version of botocore?

For me, not botocore, but boto3.
boto3==1.34.31

[stephen-thomas-buildingestimates]

Hit this a few days ago in a conda environment:
boto3 1.34.34 pyhd8ed1ab_0 conda-forge
botocore 1.34.34 pyhd8ed1ab_0 conda-forge
urllib3 2.2.0 pypi_0 pypi
python 3.8.18 hd12c33a_0_cpython conda-forge
Cannot work out why it happens, but if I delete and rebuild the conda environment seems to be ok for a while.

Update: Noticed from coping and pasting above that urllib was coming from pypi, issue was an incompatible urllib version was being installed by mypy with "--install-types" argument. Manually installing type packages and keeping correct urllib version solved the issue.

[karan-mungra]

We are also facing the same issue. I think it is not due to botocore, but because urllib3 updated from using bytes to str

[sinogermany]

I confirm that @drimal 's solution works.

I explicitly forced urllib3==2.0.7 in my dependencies and the error no longer appears.

(Python 3.12, official sam lambda image)

[sravan2366]

install urllib3==2.0.7 fixed mine. rest of the packages are still the latest.

This solution has worked for me.

[smarabattula]

I was running an AWS Sagemaker MLOps workshop notebook on a Data Science 3.0 on Sagemaker Classic notebook and ran into the same issue.
Python version = 3.10
boto3 version = 1.34.33
Sagemaker version = 2.206.0

Then Sagemaker version = 2.207.1 has fixed the issue.

[dudeanurag]

Ran into the same issue, force installed boto3==1.34.33 , which reduced urllib3 to 2.0.7, and it worked

How come? boto3 has conflict with urllib3 >= 2.0

[dudeanurag]

Just ran into this issue using botocore 1.34.35. Downgrading urllib3 from version 2.2.0 to 2.1.0 fixed the issue.

How is this possible.. boto3 (even the latest version) has conflict with urllib3 >= 2.0

[steffenweber]

Seems like Gentoo removes this setup.py version check: https://github.com/gentoo/gentoo/tree/master/dev-python/botocore

# unpin deps
sed -i -e "s:>=.*':':" setup.py || die

[rkmish]

I am also facing same issue: botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: sequence item 0: expected str instance, bytes found

Could anyone help me with exact version of boto3 , botocore and urllib3 to use in order to overcome this issue.

[alexjiao-chrono]

We were able to resolve the issue by pinning the following versions:

urllib3==1.26.13
botocore==1.29.165
boto3==1.26.93

This is in a python3.11-buster Docker image.

[essamgouda97]

We were able to solve this issue by having the following packages:

boto3==1.34.33
botocore==1.34.40
urllib3==1.26.18

for python 3.8.16 debian based docker image.

[RomainBsb]

Also getting this error with Lambda Python 3.12.
What's the best way to solve this in Lambda? Create custom layer with another version of boto3?