Nirn returns 401 sometimes on interactions about nirn-proxy

[germanoeich]

Hello!

If I'm not mistaken, Discord may return 401s on interactions. Nirn has a fail-safe in place to lock the queues permanently when we receive a 401 (This is important to prevent a cloudflare ban when a token is reset).

Clearly the 401 checks were too lax and have caused some problems. I've released version v1.1.3 to address these instances. This version prevent the queues from being locked if we are dealing with an interaction and also prevent the unauthorized queues (webhooks, requests with no token attached) from being locked.

Let me know if this version fixes the issues you described!

Note: If something is stopping you from updating at this moment, you can also set the DISABLE_401_LOCK env var to true in order to stop the fail-safe from triggering.

[itsmishra19]

I don't think this is fixed

[itsmishra19]

@germanoeich

[germanoeich]

@hydrox19 Hey, what version are you running?

401s are only returned by the proxy when the proxy itself receives a 401 from discord. Did you try with the DISABLE_401_LOCK flag set to true? Did that help?

It's also wierd that your 429 curve is so high on /gateway/bot, are you spamming that endpoint with multiple bots? Are all your workloads going through the proxy? How many bots or applications that communicate with Discord share the same IP?

[itsmishra19]

@hydrox19 Hey, what version are you running?

401s are only returned by the proxy when the proxy itself receives a 401 from discord. Did you try with the DISABLE_401_LOCK flag set to true? Did that help?

It's also wierd that your 429 curve is so high on /gateway/bot, are you spamming that endpoint with multiple bots? Are all your workloads going through the proxy? How many bots or applications that communicate with Discord share the same IP?

1. I don't know what flag you are talking about
2. 4 nirn replicas, 3 bots (one with max_concurrency of 16)

[germanoeich]

I don't know what flag you are talking about

There is an environment variable called "DISABLE_401_LOCK", can you set that to true and see if this issue stops happening?

4 nirn replicas, 3 bots (one with max_concurrency of 16)

Are all the applications proxied?

[itsmishra19]

I don't know what flag you are talking about

There is an environment variable called "DISABLE_401_LOCK", can you set that to true and see if this issue stops happening?

4 nirn replicas, 3 bots (one with max_concurrency of 16)

Are all the applications proxied?

Yes

[germanoeich]

Please, answer all the questions. It's really difficult to help you if you don't give me context and information.

Have you tried the environment variable? Did it help?

[itsmishra19]

Please, answer all the questions. It's really difficult to help you if you don't give me context and information.

Have you tried the environment variable? Did it help?

Oh sorry for missing that out, no I haven't really tried that. I am not sure should I pipe interactions through or not since a lot of times I get unknown interactions and also 404 on interactions and 401 (idk what causes it seems to be only getting from nirn).

[germanoeich]

We pipe all our requests through nirn, interactions, webhooks, normal requests and we have no issues.

What version are you running? I published a new version that should have fixed the 401 issues with interactions specifically a few days ago.

[itsmishra19]

We pipe all our requests through nirn, interactions, webhooks, normal requests and we have no issues.

What version are you running? I published a new version that should have fixed the 401 issues with interactions specifically a few days ago.

I will try with the flag and get back to you.

[itsmishra19]

This is the result after using the flag.

[itsmishra19]

DELETE /channels/!/messages/! (429 Too Many Requests)
GET /channels/! (429 Too Many Requests)
GET /guilds/!/members/! (429 Too Many Requests)
PATCH /guilds/!/members/! (429 Too Many Requests)
PATCH /webhooks/!/!/messages/@original (429 Too Many Requests)
POST /channels/!/messages (429 Too Many Requests)
POST /interactions/!/!/callback (429 Too Many Requests)
PUT /channels/!/messages/!/reactions/!/! (429 Too Many Requests)

[germanoeich]

I am confused, is the issue with 429s or 401s?

A 401 will stop the proxy dead in it's track if it doesn't come from webhooks or interactions. This is expected, and is what the DISABLE_401_LOCK env var prevents.

A 429 is a recoverable request, and will happen during normal proxy usage, although the volume won't be enough to cause issues.

Per the docs for this project, you are responsible for retrying a 429 if the proxy happens to return one. Discord.js should handle this automatically.

What is the issue here exactly?

[itsmishra19]

I am confused, is the issue with 429s or 401s?

A 401 will stop the proxy dead in it's track if it doesn't come from webhooks or interactions. This is expected, and is what the DISABLE_401_LOCK env var prevents.

A 429 is a recoverable request, and will happen during normal proxy usage, although the volume won't be enough to cause issues.

Per the docs for this project, you are responsible for retrying a 429 if the proxy happens to return one. Discord.js should handle this automatically.

What is the issue here exactly?

My question is why I am getting 429 on interactions, even tho interactions don't have a ratelimit afaik. I know other routes are my issue.

[germanoeich]

Well what I can say is that the proxy is not generating those 429s, they are coming from discord directly.

Check if you are calling /callback on multiple places, hook your request to a logger to check the body for those requests, etc and try to check for useful error messages from Discord