v4.1.5: oauth redirect loop about jupyterhub

[minrk]

However I suspect this is defeating the point of check_xsrf_cookie()?

Yes, that is precisely defeating the purpose. The goal of the XSRF token is establishing that the sender has access to the xsrf cookie. The XSRF token must always be sent in two places:

1. the _xsrf cookie, which is controlled by the browser, and
2. another location (such as the _xsrf argument or X-Xsrftoken header), controlled by the requesting code

The XSRF check passes if these two match. The presence of the _xsrf cookie itself is not meaningful on its own.

found that this function is now called in 4.1.5 on the GET requests for the JS and CSS static files, when it wasn't in 4.1.4

This generally shouldn't happen on static files, and doesn't in my tests. I will try to investigate why yours are getting caught up in this.

@ibh1127 it sounds like your login form doesn't set the xsrf token input. The explanation is above, that presence in _xsrf is not sufficient, it must be sent by your login form as well, e.g. via hidden input, as done here.

[minrk]

One thing that would clean up and simplify things is if we only did the login redirect on Sec-Fetch-Mode: navigate + Sec-Fetch-Dest: document requests. I don't think other requests are going to complete the login redirect process, so they shouldn't try. The concurrent login redirects is what's causing the oauth state messages, but that's a misdirection from the real issue, which is that the initial requests aren't accepted - what exactly happens after the failed request is less relevant.

The underlying cause is that 4.1 applies consistent XSRF checks to authenticated GET requests, which is required to protect user servers from each other, whereas 4.0 did not strictly protect GET requests, only POST and others.

In general, these are the changes required to work in this situation:

1. disable XSRF checks on handlers where you don't care to protect the endpoint from other JupyterHub users:

    def check_xsrf_cookie(self):
        return

This makes sense e.g. for shared static assets, but probably not userprofile (removing @web.authenticated would also avoid the xsrf check), and
2. make API requests with the JupyterHub API token in the Authorization header. Token-authenticated requests do not have XSRF checks applied. XSRF checks only apply to requests that rely on implicit auth with cookies (the source of the XSRF problem in general), OR
3. add xsrf header to GET API requests like this one for userprofile, which you already have for the graphql post, since it's been required there for longer.

I would recommend going with token approach, if possible, but copying the xsrf header code you already have to the requests missing them is probably the smallest change to get things working in the short term.

I've made the following PRs:

• cylc/cylc-uiserver#592 (fixes static asset requests)
• cylc/cylc-ui#1778 (fixes userprofile request by adding xsrf token)

which together made cylc work for me in 4.1.5 and should be fully backward compatible.

[minrk]

Ah, the missing xsrf token is probably because the request to serve index.html doesn't set the xsrf token because that page is also served by the StaticFileHandler (in most Jupyter pages, HTML is served by a template, and the xsrf token is computed for the template namespace). I believe that is fixed by the latest commit in cylc/cylc-uiserver#592, which accesses the xsrf_token for any static page GET.

That way, when GET /user/:name/cylc/ completes, you definitely have a currently valid xsrf cookie

[MetRonnie]

That's almost got it, however I think when the index.html is cached by the browser, the GET never goes through our static handler and so the XSRF cookie is not set 😢. This seems more prevalent on Firefox than Chrome

Nothing that can't be solved by refreshing the page, however.

Edit: see cylc/cylc-uiserver#592 (comment):

I think setting Cache-Control: no-cache does the trick 🤞

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[MetRonnie]

Update: I added some print statements in here:

jupyterhub/jupyterhub/_xsrf_utils.py

Lines 203 to 204 in b405361

	def check_xsrf_cookie(handler):
	"""Check that xsrf cookie matches xsrf token in request"""

and found that this function is now called in 4.1.5 on the GET requests for the JS and CSS static files, when it wasn't in 4.1.4

(Note to self: only difference between the two versions is this diff f395acd from #4771)

[MetRonnie]

Further update: When inspecting the requests that were getting redirected and ultimately failing, they included the XSRF token in the Cookie header, but didn't include the X-Xsrftoken header, so check_xsrf_cookie() was raising a (swallowed) 403

If I make this edit then everything works as expected:

Hidden

 def check_xsrf_cookie(handler):
     """Check that xsrf cookie matches xsrf token in request"""
     # overrides tornado's implementation
     # because we changed what a correct value should be in xsrf_token
     if not _needs_check_xsrf(handler):
         # don't require XSRF for regular page views
         return


     token = (
         handler.get_argument("_xsrf", None)
         or handler.request.headers.get("X-Xsrftoken")
         or handler.request.headers.get("X-Csrftoken")
+        or handler.get_cookie("_xsrf")
     )

However I suspect this is defeating the point of check_xsrf_cookie()?

[ibh1127]

I am having a very similar issue where check_xsrf_cookie returns a 403 during login.

Our cookie header contains the correct _xsrf token but the X-Xsrftoken/X-Csrftoken header is not set so the check_xsrf_cookie method returns the 403 '_xsrf' argument missing from POST.

[minrk]

@MetRonnie can you explain why you have added crossorigin to your js/css? I believe this is what's causing it to take a different path, because it appears to be explciitly attempting a crossorigin-style request without the required crossorigin credentials, when a 'standard' style/script tag would work fine.

[MetRonnie]

Many thanks for looking into this.

can you explain why you have added crossorigin to your js/css?

This seems to be done by Vite when building the web app. Removing them from the built HTML file does not solve the problem unfortunately.

However, a colleague suggested we could remove Tornado's @web.authenticated from our static handler's get method in the server: https://github.com/cylc/cylc-uiserver/blob/b007b5f91491a2be97fa16313dfbbbe57ce7e933/cylc/uiserver/handlers.py#L187-L188

and this does allow the JS/CSS to be fetched without a problem.

But unfortunately we still get the redirect loop problem for a XHR GET to our userprofile.json endpoint, just without the "oauth state does not match" bit:

Log

[I JupyterHub users:776] Server rdutta is ready
[I JupyterHub log:192] 200 GET /hub/api/users/rdutta/server/progress?_xsrf=[secret] (rdutta@::ffff:XX.XXX.XXX.XX) 6968.92ms
[I JupyterHub log:192] 302 GET /hub/spawn-pending/rdutta -> /user/rdutta/ (rdutta@::ffff:XX.XXX.XXX.XX) 11.80ms
[I CylcHubApp log:192] 302 GET /user/rdutta/ -> /user/rdutta/cylc? (@::ffff:XX.XXX.XXX.XX) 0.70ms
[I CylcHubApp log:192] 301 GET /user/rdutta/cylc? -> cylc/ (@::ffff:XX.XXX.XXX.XX) 0.58ms
[W CylcHubApp web:1873] 403 POST /user/rdutta/cylc/graphql (::ffff:XX.XXX.XXX.XX): XSRF cookie does not match POST argument
[W CylcHubApp log:192] 403 POST /user/rdutta/cylc/graphql (@::ffff:XX.XXX.XXX.XX) 22.77ms
[I CylcHubApp _xsrf_utils:128] Setting new xsrf cookie for b'0ff7b4bd:O63Zfgd=' {'path': '/user/rdutta/', 'max_age': 3600}
[W CylcHubApp log:192] 403 GET /user/rdutta/cylc/subscriptions (@::ffff:XX.XXX.XXX.XX) 3.63ms
[I CylcHubApp log:192] 302 GET /user/rdutta/cylc/userprofile -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-rdutta&redirect_uri=%2Fuser%2Frdutta%2Foauth_callback&response_type=code&state=[secret] (@::ffff:XX.XXX.XXX.XX) 5.29ms
[I JupyterHub log:192] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-rdutta&redirect_uri=%2Fuser%2Frdutta%2Foauth_callback&response_type=code&state=[secret] -> /user/rdutta/oauth_callback?code=[secret]&state=[secret] (rdutta@::ffff:XX.XXX.XXX.XX) 94.55ms
[I CylcHubApp _xsrf_utils:128] Setting new xsrf cookie for b'0ff7b4bd:O63Zfgd=' {'path': '/user/rdutta/', 'max_age': 3600}
[W CylcHubApp log:192] 403 GET /user/rdutta/cylc/subscriptions (@::ffff:XX.XXX.XXX.XX) 2.31ms
[I JupyterHub log:192] 200 POST /hub/api/oauth2/token ([email protected]) 98.51ms
[I JupyterHub log:192] 200 GET /hub/api/user ([email protected]) 17.42ms
[I CylcHubApp auth:1510] Logged-in user {'groups': [], 'kind': 'user', 'admin': False, 'name': 'rdutta', 'session_id': '0ff7b4bd', 'scopes': ['access:servers!server=rdutta/', 'read:users:groups!user=rdutta', 'read:users:name!user=rdutta']}
[I CylcHubApp _xsrf_utils:128] Setting new xsrf cookie for b'0ff7b4bd:64a3bbe' {'path': '/user/rdutta/'}
[I CylcHubApp log:192] 302 GET /user/rdutta/oauth_callback?code=[secret]&state=[secret] -> /user/rdutta/cylc/userprofile (@::ffff:XX.XXX.XXX.XX) 123.97ms
[I CylcHubApp log:192] 302 GET /user/rdutta/cylc/userprofile -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-rdutta&redirect_uri=%2Fuser%2Frdutta%2Foauth_callback&response_type=code&state=[secret] (@::ffff:XX.XXX.XXX.XX) 1.64ms
[W CylcHubApp _xsrf_utils:198] Skipping XSRF check for insecure request GET /user/rdutta/cylc/subscriptions
[I CylcHubApp log:192] 101 GET /user/rdutta/cylc/subscriptions (rdutta@::ffff:XX.XXX.XXX.XX) 1.72ms
[I JupyterHub log:192] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-rdutta&redirect_uri=%2Fuser%2Frdutta%2Foauth_callback&response_type=code&state=[secret] -> /user/rdutta/oauth_callback?code=[secret]&state=[secret] (rdutta@::ffff:XX.XXX.XXX.XX) 120.60ms
[I JupyterHub log:192] 200 POST /hub/api/oauth2/token ([email protected]) 55.49ms
[I JupyterHub log:192] 200 GET /hub/api/user ([email protected]) 16.81ms
[I CylcHubApp auth:1510] Logged-in user {'groups': [], 'kind': 'user', 'admin': False, 'name': 'rdutta', 'session_id': '0ff7b4bd', 'scopes': ['access:servers!server=rdutta/', 'read:users:groups!user=rdutta', 'read:users:name!user=rdutta']}
[I CylcHubApp _xsrf_utils:128] Setting new xsrf cookie for b'0ff7b4bd:c5c009d' {'path': '/user/rdutta/'}
[I CylcHubApp log:192] 302 GET /user/rdutta/oauth_callback?code=[secret]&state=[secret] -> /user/rdutta/cylc/userprofile (@::ffff:XX.XXX.XXX.XX) 79.93ms
[I CylcHubApp log:192] 302 GET /user/rdutta/cylc/userprofile -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-rdutta&redirect_uri=%2Fuser%2Frdutta%2Foauth_callback&response_type=code&state=[secret] (@::ffff:XX.XXX.XXX.XX) 1.83ms
[I JupyterHub log:192] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-rdutta&redirect_uri=%2Fuser%2Frdutta%2Foauth_callback&response_type=code&state=[secret] -> /user/rdutta/oauth_callback?code=[secret]&state=[secret] (rdutta@::ffff:XX.XXX.XXX.XX) 114.53ms
[I JupyterHub log:192] 200 POST /hub/api/oauth2/token ([email protected]) 62.33ms
[I JupyterHub log:192] 200 GET /hub/api/user ([email protected]) 19.52ms
[I CylcHubApp auth:1510] Logged-in user {'groups': [], 'kind': 'user', 'admin': False, 'name': 'rdutta', 'session_id': '0ff7b4bd', 'scopes': ['access:servers!server=rdutta/', 'read:users:groups!user=rdutta', 'read:users:name!user=rdutta']}
[I CylcHubApp _xsrf_utils:128] Setting new xsrf cookie for b'0ff7b4bd:3ebc13e' {'path': '/user/rdutta/'}
[I CylcHubApp log:192] 302 GET /user/rdutta/oauth_callback?code=[secret]&state=[secret] -> /user/rdutta/cylc/userprofile (@::ffff:XX.XXX.XXX.XX) 94.40ms
[I CylcHubApp log:192] 302 GET /user/rdutta/cylc/userprofile -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-rdutta&redirect_uri=%2Fuser%2Frdutta%2Foauth_callback&response_type=code&state=[secret] (@::ffff:XX.XXX.XXX.XX) 1.82ms
[I JupyterHub log:192] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-rdutta&redirect_uri=%2Fuser%2Frdutta%2Foauth_callback&response_type=code&state=[secret] -> /user/rdutta/oauth_callback?code=[secret]&state=[secret] (rdutta@::ffff:XX.XXX.XXX.XX) 80.20ms
[I JupyterHub log:192] 200 POST /hub/api/oauth2/token ([email protected]) 77.35ms
[I JupyterHub log:192] 200 GET /hub/api/user ([email protected]) 14.40ms
[I CylcHubApp auth:1510] Logged-in user {'groups': [], 'kind': 'user', 'admin': False, 'name': 'rdutta', 'session_id': '0ff7b4bd', 'scopes': ['access:servers!server=rdutta/', 'read:users:groups!user=rdutta', 'read:users:name!user=rdutta']}
[I CylcHubApp _xsrf_utils:128] Setting new xsrf cookie for b'0ff7b4bd:ecb92d0' {'path': '/user/rdutta/'}
[I CylcHubApp log:192] 302 GET /user/rdutta/oauth_callback?code=[secret]&state=[secret] -> /user/rdutta/cylc/userprofile (@::ffff:XX.XXX.XXX.XX) 106.67ms
[I CylcHubApp log:192] 302 GET /user/rdutta/cylc/userprofile -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-rdutta&redirect_uri=%2Fuser%2Frdutta%2Foauth_callback&response_type=code&state=[secret] (@::ffff:XX.XXX.XXX.XX) 10.07ms
[I JupyterHub log:192] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-rdutta&redirect_uri=%2Fuser%2Frdutta%2Foauth_callback&response_type=code&state=[secret] -> /user/rdutta/oauth_callback?code=[secret]&state=[secret] (rdutta@::ffff:XX.XXX.XXX.XX) 48.52ms
[I JupyterHub log:192] 200 POST /hub/api/oauth2/token ([email protected]) 67.26ms
[I JupyterHub log:192] 200 GET /hub/api/user ([email protected]) 22.32ms
[I CylcHubApp auth:1510] Logged-in user {'groups': [], 'kind': 'user', 'admin': False, 'name': 'rdutta', 'session_id': '0ff7b4bd', 'scopes': ['access:servers!server=rdutta/', 'read:users:groups!user=rdutta', 'read:users:name!user=rdutta']}
[I CylcHubApp _xsrf_utils:128] Setting new xsrf cookie for b'0ff7b4bd:577781e' {'path': '/user/rdutta/'}
[I CylcHubApp log:192] 302 GET /user/rdutta/oauth_callback?code=[secret]&state=[secret] -> /user/rdutta/cylc/userprofile (@::ffff:XX.XXX.XXX.XX) 99.30ms
[I CylcHubApp log:192] 302 GET /user/rdutta/cylc/userprofile -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-rdutta&redirect_uri=%2Fuser%2Frdutta%2Foauth_callback&response_type=code&state=[secret] (@::ffff:XX.XXX.XXX.XX) 1.80ms
[I JupyterHub log:192] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-rdutta&redirect_uri=%2Fuser%2Frdutta%2Foauth_callback&response_type=code&state=[secret] -> /user/rdutta/oauth_callback?code=[secret]&state=[secret] (rdutta@::ffff:XX.XXX.XXX.XX) 90.53ms
[I JupyterHub log:192] 200 POST /hub/api/oauth2/token ([email protected]) 107.84ms
[I JupyterHub log:192] 200 GET /hub/api/user ([email protected]) 58.80ms
[I CylcHubApp auth:1510] Logged-in user {'groups': [], 'kind': 'user', 'admin': False, 'name': 'rdutta', 'session_id': '0ff7b4bd', 'scopes': ['access:servers!server=rdutta/', 'read:users:groups!user=rdutta', 'read:users:name!user=rdutta']}
[I CylcHubApp _xsrf_utils:128] Setting new xsrf cookie for b'0ff7b4bd:2a4b09d' {'path': '/user/rdutta/'}
[I CylcHubApp log:192] 302 GET /user/rdutta/oauth_callback?code=[secret]&state=[secret] -> /user/rdutta/cylc/userprofile (@::ffff:XX.XXX.XXX.XX) 2104.00ms
[I CylcHubApp log:192] 200 POST /user/rdutta/cylc/graphql (rdutta@::ffff:XX.XXX.XXX.XX) 2029.59ms
[I CylcHubApp log:192] 302 GET /user/rdutta/cylc/userprofile -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-rdutta&redirect_uri=%2Fuser%2Frdutta%2Foauth_callback&response_type=code&state=[secret] (@::ffff:XX.XXX.XXX.XX) 11.25ms
[I JupyterHub log:192] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-rdutta&redirect_uri=%2Fuser%2Frdutta%2Foauth_callback&response_type=code&state=[secret] -> /user/rdutta/oauth_callback?code=[secret]&state=[secret] (rdutta@::ffff:XX.XXX.XXX.XX) 98.74ms
[I JupyterHub log:192] 200 POST /hub/api/oauth2/token ([email protected]) 81.43ms
[I JupyterHub log:192] 200 GET /hub/api/user ([email protected]) 17.16ms
[I CylcHubApp auth:1510] Logged-in user {'groups': [], 'kind': 'user', 'admin': False, 'name': 'rdutta', 'session_id': '0ff7b4bd', 'scopes': ['access:servers!server=rdutta/', 'read:users:groups!user=rdutta', 'read:users:name!user=rdutta']}
[I CylcHubApp _xsrf_utils:128] Setting new xsrf cookie for b'0ff7b4bd:344d52e' {'path': '/user/rdutta/'}
[I CylcHubApp log:192] 302 GET /user/rdutta/oauth_callback?code=[secret]&state=[secret] -> /user/rdutta/cylc/userprofile (@::ffff:XX.XXX.XXX.XX) 116.27ms

Edit: just noticed on page refresh after this the app loads successfully. But after closing the browser and re-opening, I get the problem again. Seems like the XRSF cookie has not been set by the time the the GET happens. Sometimes the XSRF cookie has been set and the header is included in the GET but it still gets in a redirect loop.

[MetRonnie]

Much appreciated! And thank you for the explanation!

It now works after logging in for the first time. However, if I close the browser, then re-open the browser, I find that I get the oauth redirect loop again. I think this is because the XSRF cookie has session lifetime, and when re-opening the browser it is no-longer set in time for the userprofile GET request.

I think this is why the redirect loop is happening: After the first redirect to the oauth URLs, the XSRF cookie is set, but when it redirects back to the userprofile, the GET request still does not contain the XSRF token header (because it is still the original request? (It is not re-executing our axios.get())). Hence it redirects back to oauth and so on.

This can be worked around by retrying the userprofile GET after the redirect loop fails. But it sounds like this could be solved by using the JHub API token that you've mentioned, is there somewhere in the docs you can point me to for this? Or is there another way to solve this by somehow going through the oauth before the userprofile GET request?

[MetRonnie]

Awesome, thanks for the fixes!