robots.txt Disallow / for public routes about pomerium

[kenjenkins]

Hi @stuba,

It looks like currently the logic from #1361 is based on the allow_public_unauthenticated_access option. Please try updating your route configuration to set allow_public_unauthenticated_access instead of policy, something like this:

  - name: Default route
    from: https://my_service.url
    to: http://nginx
    preserve_host_header: true
    allow_public_unauthenticated_access: true

[stuba]

Surely this works. I do not know why I had understood this allow_public_unauthenticated_access is all ready deprecated in flavour of new policy. Thank you for clarifying this!

[calebdoxsey]

I agree we should update the logic to also work with PPL.

[kenjenkins]

Per Caleb: we could refactor this logic to move it to the authorizer service-- if we get a request for /robots.txt and the request is not authorized, we could directly return the "disallow" response. (This should avoid some of the complication of recognizing equivalent PPL policies.)

[calebdoxsey]

The logic around robots.txt handling has been updated to happen in the authorize service so that any policy that allows access without requiring a login should behave the same.

The example route:

 - name: Default route
    from: https://my_service.url
    to: http://nginx
    preserve_host_header: true
    policy:
      allow:
        and:
          - accept: true

Should forward requests to the backend. Whereas a route with an authorization policy should return

User-agent: *
Disallow: /

[stuba]

Perfect, thanks!