ResourceExhausted: 429 received metadata size exceeds soft limit about google-auth-library-python

[arithmetic1728]

@pweglik Could you provide some examples of the large x-goog-api-client header value to help us identify the issue?

[arithmetic1728]

@pweglik I created a fix in metric branch, not sure if that works, could you give it a try? You can install it via pip install git+https://github.com/googleapis/google-auth-library-python.git@metric

[JehangirLBG]

@arithmetic1728 Hi I have the same issue, I've tried your fix and it doesn't seem to have any effect.

In my case the issue is easily reproducible after calling the llm a 1000 times in succession. As I'm trying to evaluate llm outputs on a few different metrics I need to run a call to the LLM on a large amount of outputs and rerun a couple times.

After around 1000 calls I hit the same metadata issue as above with the x-goog-api-client-header continuing to grow with each request and receiving the error after each call when the soft limit had been reached.

[arithmetic1728]

@JehangirLBG any chance to give some examples of the x-goog-api-client header?

[arithmetic1728]

@JehangirLBG can you also provide an easy code sample to repro?

[arithmetic1728]

@JehangirLBG I updated that branch with a new potential fix. Could you try it again? Thanks!

[JehangirLBG]

@arithmetic1728 Hi, I have gave it a go and still having the issue, as you can see error message below:
Retrying langchain_google_vertexai.chat_models._completion with_retry.<locals-_completion-with-retry-inner in 4.0 seconds as it raised ResourceExhausted: 429 received metadata size exceeds soft limit (34425 vs. 16384): path 103B authority 79B method:43 Bischeme:44B content-type:60B te:42B grpc-accept-encoding:75B user-agent:100B grpc-trace-bin: 103B pc-low-fud-bin:77B x-goog-request-params:150B x-goog-api-client:29279B x-goog-api-client:62B authorization: 1076B *-goog-user-project: 65B x-google-gfe-frontline-info:914B x-google-gfe-cloud-client-vnid:69B x-google-gfe-cloud-client-network-project-number.92B x-google-gfe-timestamp-trace:76B x-google-gfe-verified-user-ip:97B x-gfe-signed-request-headers:620B x-google-gfe-location-info:74B x-gfe-ssl:44B x-google-gfe-tls-base64urlclienthelloprotobuf:299B x-user-ip:52B x-google-service: 105B x-google-gfe-service-trace: 115B x-google-gfe-backend-timeout-ms:71B accept-encoding:56B x-google-peer-delegation-chain-bin:92B x-google request-uid:141B x-google-dappertraceinfo: 111B.

I have seen the metadata go up to 50000 with only x-goog-api-client increasing in size.

[arithmetic1728]

@JehangirLBG I see. The error message only tells the size (e.g. x-goog-api-client:29279B), what I need is the x-goog-api-client header string content (i.e. the 29279 bytes). Any x-goog-api-client values are fine.

The header can be added by multiple libraries beside this lib, without knowing the content, it's not possible to figure out the reason. Is there any way to reveal the header string in the error message?

I updated the "fix" to not add the x-goog-api-client header from this lib (This is not what we wanted to do, but just for troubling shooting). Could you try it again?

Also could you provide a simple repro sample?

[JehangirLBG]

• @arithmetic1728 Tried to run again, still getting the issue.

I understand what you mean with the needing the content of the actual header now, I believe I can just run '.response_metadata' on the returned llm object, however I'm still new to Python so unsure how to break from the code once the metadata has exceeded a certain amount and then run this on the last object returned.

If you have any guidance on that I can do it now or I can check with my colleague tomorrow and get back to you. Thanks for your help :)

I can try to get something together to help reproduce this on your end.

[arithmetic1728]

@JehangirLBG You don't need to run this on the last object, just run it for every object, and provide a few x-goog-api-client header values that grow in size. From there I can compare and see which part is repeated, and figure out what library adds it. Thank you for your help!

[arithmetic1728]

Closing the issue due to inactivity, please reopen if the issue persists or if any new information can be provided, thanks!