Improve docs on preferred ways to route to K8S Services and Pods about gloo

[Nadine2016]

#9194

[sam-heilbron]

Option A: Routing to the Kubernetes Service directly

This works by routing traffic from the client, through the gateway-proxy, directly to the Kubernetes Service. As @DuncanDoyle points out, this means that we are relying on the kube-proxy as opposed to our gateway-proxy for upstream load balancing.

This option is configured via the API:

routes:
- matchers:
   - prefix: /petstore
  routeAction:
    single:
      kube:
        ref:
          name: petstore
          namespace: default
        port: 8080

The ControlPlane relies on a Gloo primitive, the Upstream CR, to define upstream workloads. This is meant as a generic API to apply to upstream workloads as this may be a kubernetes service, or any other type of destination. Because of this API, we have logic to validate that upstream destinations are in fact valid. This works by performing the following steps:

1. Convert a "destination" into the Upstream reference (name/namespace)
2. Inspect the API Snapshot (the full set of resources that Gloo is aware of) to see if that Upstream ref exists
3. Error if not

As a result, to allow for routing to a Kubernetes Service directly, Gloo must create an Upstream CR to represent the config (and therefore it is in the API Snapshot, and the above linked code does not return an error).
In order to generate the Upstream, Gloo must:

1. Poll the list of Kubernetes Services (https://github.com/solo-io/gloo/blob/main/projects/gloo/pkg/upstreams/hybrid_client.go#L50)
2. Update the internal memory to store those Upstream representations

This action has been proven to be more resource exhaustive

Option B: Routing to the Kubernetes Service via the Standard Upstream API

Instead of routing to a Kubernetes Service, you can route to the endpoints that a Kubernetes Service represents.

This is done via the Upstream API:

gloo/projects/gloo/api/v1/options/kubernetes/kubernetes.proto

Line 18 in ce7474c

message UpstreamSpec {

The ControlPlane will then execute an EDS loop (https://github.com/solo-io/gloo/blob/main/projects/gloo/pkg/plugins/kubernetes/eds.go#L88) in which the Upstreams are processed, and the ControlPlane continually resolves the available pods/endpoints to pass along to the gateway-proxy.

[Nadine2016]

#9196

[Nadine2016]

this is in prod https://docs.solo.io/gloo-edge/latest/guides/traffic_management/destination_types/kubernetes_services/ closing