proxy: grpc client should retry connections to services on failure about pomerium

[victornoel]

@desimone so I finally got back to this.

The problem is still present:

• I generate new cookie/shared secret to be sure this is behaving as a fresh start
• I (re)start all the services
• I connect to a protected app
• I get redirected to the authenticate service which redirect me to google signin
• I get redirected back to the protected app and the error below appears
• I can click on session and see my session
• I can refresh the page and access the protected app

Furthermore:

• if I remove the cookies and reconnect to the app, I can signin and get redirected successfully to the app.
• if I restart the service (but do not delete the cookies), I can signin and get redirected successfully to the app.
• if I restart the service and delete the cookies, I can signin but get the error above.

[victornoel]

@travisgroth @desimone this is fixed in v3.0.0, thanks for your patience.

[victornoel]

@desimone I'm out of my depth in there :) I was hoping for some kind of very simple solution that would retry once on connection failure in this situation or something like that.

Let's also note that this is not a severe bug, even though it's not nice to experience.

[victornoel]

@desimone I will try v0.0.4 very soon and get back to you on this

[travisgroth]

@victornoel We started handling transient GRPC issues a bit more gracefully with #261. Can you check on reproducing when you have a moment?

[1rootgear]

I've also received a very similar error. I've used Google as the identity provider and after logging in, I'm redirected to a 500 server error.

The logs in the proxy display:
ERR proxy: error redeeming authorization code error="rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: <nil>"

Environment:

• Pomerium version: v0.0.3+7bf257f

• Architecture: Synology/Docker

[desimone]

I've seen this error when I've down an (unsafe) restart of one of the services. all subconns are in transientFailure usually means one of the services is down... but this this is especially odd in the context of using it with Synology because these connections would all be over localhost.

@1rootgear Can you tell me a little bit about what was going on prior to the error? Is it intermittent? Any surrounding errors?

[1rootgear]

The issue is not intermittent, it consistently provides that error when I try to access the page or refresh after authenticating. So everything starts out appearing as if it will work but after authenticating it redirects to the Pomerium 500 internal server error page.

An error and warning do appear when first accessing the page - before authenticating.

DBG proxy: starting auth flow error="http: named cookie not present" fwd_ip=10.0.1.1 ip=172.17.0.1

ERR authenticate: failed to load session error="internal/sessions: invalid session"
WRN authenticate: authenticate error error="internal/sessions: invalid session"

Then the TransientFailure happens.

I created a debian docker container and compiled Pomperium, however I'm getting similar errors.

ERR authenticate: failed to load session error="http: named cookie not present"
ERR proxy: error redeeming authorization code error="rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: authentication handshake failed: EOF\"

[desimone]

The issue is not intermittent, it consistently provides that error when I try to access the page or refresh after authenticating. So everything starts out appearing as if it will work but after authenticating it redirects to the Pomerium 500 internal server error page.

Ahh, okay. My hunch is that there is some sort of configuration error.

DBG proxy: starting auth flow error="http: named cookie not present" fwd_ip=10.0.1.1 ip=172.17.0.1

Means your session is invalid. This is normal if you have not let authenticated. The error could be more clear so I'll fix that.

ERR proxy: error redeeming authorization code error="rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: authentication handshake failed: EOF"

This makes me think the underlying error is certificate related. Authentication and authorization related actions are made of remote procedure calls over the network via gRPC. In your case, this is a little goofy because the calls are being made over localhost since we aren't running pomerium in a distributed fashion. It looks like the proxy service is trying to make a TLS connection to the authenticate/authorize services on localhost but does not trust the certificate and thus blocks the connection. Are you using

Would you be able to post links to your complete logs / configuration? As you mentioned you are using synology, you can do the following.
For settings,
Docker > Container > pomerium > right click pomerium > export container settings > export.
For logs,
Docker > Container > pomerium > Log > Export > CSV

Be sure to remove any sensitive data like passwords and secret keys before posting. You can also send this to me directly at bobby{my github user name}@gmail.com.

[1rootgear]

So the initial DBG proxy: starting auth flow error= I mentioned is before authenticating, that's when you first attempt to go to httpbin.my-domain.com, for example. If the error is more specific as you mention that could be helpful.

I've tried using AUTHENTICATE_INTERNAL_URL and AUTHORIZE_INTERNAL_URL as localhost:443, in conjunction with OVERRIDE_CERTIFICATE_NAME - but still cannot get rpc error to go away.

Looks like you did not finish your sentence "are you using", what were you going to ask?

Synology Docker Pomerium config linked:
https://pastebin.com/VZxne6js

And here are the logs:
https://pastebin.com/faeD8Bji

I should note, this Synology is sitting behind haproxy... however, I've successfully accessed the Synology directly through the reverse proxy while using these certs.

[desimone]

@1rootgear thank you for attaching your logs and config settings.

I meant to ask if you were using a LetsEncrypt (or universally trusted CA) wild card subdomain tls certificate?

Try:

• (Re)-set AUTHENTICATE_INTERNAL_URL and AUTHORIZE_INTERNAL_URL to localhost:443 again and OVERRIDE_CERTIFICATE_NAME to whatever your wildcard domain is.
• Set CERTIFICATE_AUTHORITY or FILE to be whatever your CA's cert chain is.

I'm pretty confident this is certificate and ingress related. Just to explain my thinking a bit...

*_INTERNAL_URL settings need to be set behind any ingress (haproxy vs nginx shouldn't matter pomerium needs to be told to connect to itself directly over the local socket instead of trying to transverse the network again from the dns domain). Otherwise -- and I've had this happen -- pomerium is going to try to connect back out through the network and then be load balanced by whatever Ingress you are using. Unfortunately few balancers currently natively support HTTP2/gRPC so the request will fail and you'll usually give a vague error of a bad handshake or bad content type depending on your ingress type.

For example, if I follow the synology guide but omit those settings I see something like....

{  
   "level":"error",
   "fwd_ip":"192.168.1.1",
   "ip":"172.17.0.1",
   "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36",
   "req_id":"f35b128e-97fe-e696-79e3-704cf5635f7f",
   "error":"rpc error: code = Internal desc = transport: received the unexpected content-type \"text/html; charset=utf-8\"",
   "time":"2019-04-14T19:32:38Z",
   "message":"proxy: error redeeming authorization code"
}

Let me know if that helps; or even if we can get a slightly different error.

[1rootgear]

Great! Utilizing CERTIFICATE_AUTHORITY_FILE worked! I'm using a wildcard Let's Encrypt certificate and now using this CA - Let’s Encrypt Authority X3 (IdenTrust cross-signed)

Just to note, I am also utilizing AUTHENTICATE_INTERNAL_URL and AUTHORIZE_INTERNAL_URL to localhost:443 again and OVERRIDE_CERTIFICATE_NAME as you mentioned.

Now the last weird thing - and I almost commented on it earlier but had edited it out, as I thought it may have been a fluke. My referrer is going through - referer=https://accounts.youtube.com/accounts/SetSID even after setting IDP_PROVIDER_URL to "https://accounts.google.com". So right after logging in, the URL is briefly changing to accounts.youtube.com as you'd imagine. This is more of a formality at this point. However, I've checked in GCP and it does not appear that anything is pointing to youtube. Any ideas?

[desimone]

Glad to hear it's working!

As for why the referrer is going through youtube, I admit that sounds strange and is definitely specific to google. I'm not sure but my best guess is that google logs you into multiple other products including youtube when you complete a SSO request and youtube is the final stop before hitting the pomerioum redirect URL.

See:

• http://teacher-seo.blogspot.com/2012/06/google-is-redirecting-to.html
• https://webapps.stackexchange.com/questions/30254/why-does-gmail-login-go-through-youtube-com
• https://support.google.com/mail/forum/AAAAK7un8RUwWlLUvoO-wU/?hl=en&gpf=%23!msg%2Fgmail%2FwWlLUvoO-wU%2F5xdymIDG5rMJ&msgid=5xdymIDG5rMJ

[victornoel]

@desimone @1rootgear glad you could find a solution. Do we agree that the original description is actually not related to the discussion you just had?

[desimone]

@victornoel That's right. Sorry to muddy-the-waters.

I think the original reported (though I haven't reproduce it yet) you reported is due to the connection between grpc server/clients being severed and the solution would be to implement layer 4 load balancing [2] to prevent service interruption on restart or upgrade.

[desimone]

Totally agree. There's probably an opportunity for an 80/20 solution (like you said, maybe just to do a backoff retry) built into the client connection.

…

On Mon, Apr 15, 2019 at 9:27 AM Victor Noël ***@***.***> wrote: @desimone <https://github.com/desimone> I'm out of my depth in there :) I was hoping for some kind of very simple solution that would retry once on connection failure in this situation or something like that. Let's also note that this is not a severe bug, even though it's not nice to experience. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#69 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABeSsVSEGtEc49lBIPspvTjz8LqRP2KTks5vhKiIgaJpZM4cGhFy> .

[1rootgear]

@desimone Thanks for your assistance, and it looks like according to those links you mentioned that its normal to be logged into many google services at once.

@victornoel Sorry for stealing your thread, I was not sure if it was related in the beginning or not.

[desimone]

@victornoel Have you seen this recently?

[victornoel]

@travisgroth I may wait for v0.3.0, but if I can I will try to test it yes

[travisgroth]

Great. Thanks for confirming!