Coder Social home page Coder Social logo

datawire / edge-stack Goto Github PK

View Code? Open in Web Editor NEW
24.0 16.0 13.0 1.22 MB

Kubernetes-native API gateway for microservices built on the Envoy Proxy with built-in features for securing and managing traffic

Home Page: https://www.getambassador.io

License: Apache License 2.0

docker kubernetes api-gateway envoy-proxy gateway-api microservice api-management cloud-native kubernetes-ingress ambassador

edge-stack's Introduction

Ambassador Edge Stack

Docker Repository Docker Pulls Join Slack

Disclaimer: This repository is in a read only state. We do not monitor issues or pull requests. Questions and issues should can be directed to the following places:

Information regarding specific Ambassador Edge Stack releases can be found in matching rel/{release version} branches. For example Ambassador Edge Stack v3.2.0 information is at rel/v3.2.0.

edge-stack's People

Contributors

acookin avatar aidanhahn avatar aliceproxy avatar ddymko avatar ivankovnatsky avatar kflynn avatar lanceea avatar lukeshu avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

edge-stack's Issues

Ambassador pods displaying log level info DESPITE setting AES_LOG_LEVEL to error and envoy_log_path to null

I am trying to set the log level of my Ambassador deployment to error.

  1. I first tried setting the environment variable AES_LOG_LEVEL to error as specified in the documentation (https://www.getambassador.io/docs/edge-stack/latest/topics/running/aes-extensions/#aes_log_level) but the logs still remained of log level info.
env:
  AES_LOG_LEVEL: error
  AMBASSADOR_CERTS_SINGLE_NAMESPACE:

I even went inside the pod to make sure that environment variable AES_LOG_LEVEL was set to error and it was.

  1. So then I believed I had to change the Envoy debug logs and so by following (https://www.getambassador.io/docs/edge-stack/latest/topics/running/ambassador/#envoy), I set "envoy_log_path: /dev/null" for my module. Yet, my logs still display logs of level info.
service:
  externalTrafficPolicy: 'Local'
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: 'true'
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v1
      kind: Module
      name: ambassador
      config:
        service_port: 8443
        use_remote_address: true
        buffer:  
          max_request_bytes: 10485760
        x_num_trusted_hops: 1
        x_forwarded_proto_redirect: true
        grpc_stats:
          upstream_stats: true
          all_methods: true
        envoy_log_path: /dev/null
    external-dns.alpha.kubernetes.io/hostname: '${serviceVariable.ambassador_domain}'

Despite both of these changes, my logs appear as so:

2021-08-05 18:29:17 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_29] INFO: 0CE05326-EE6E-4B9D-8613-9A9D2FAD71C1: 10.58.131.14 "GET /metrics" 16ms 200 success
2021-08-05 18:29:21 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_28] INFO: BA21CC1A-BC73-4096-8EA6-2708AA4F6089: 10.58.131.18 "GET /metrics" 13ms 200 success
2021-08-05 18:29:22 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_3] INFO: 05288778-EF15-4A89-B6BC-B5F4AD0DB762: 10.58.131.14 "GET /metrics" 16ms 200 success
2021-08-05 18:29:26 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_22] INFO: 9B2F7A08-FBA8-4CEA-8558-95C938F65B2F: 10.58.131.18 "GET /metrics" 13ms 200 success
2021-08-05 18:29:27 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_24] INFO: F7BA0E95-BCA9-40BD-839B-CDC992CAD796: 10.58.131.14 "GET /metrics" 16ms 200 success
2021-08-05 18:29:31 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_15] INFO: 8CFBC403-7579-4421-86FA-095D4609DC70: 10.58.131.18 "GET /metrics" 14ms 200 success
2021-08-05 18:29:32 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_23] INFO: 53E9D092-58C4-4D0A-9A04-D14B7604C4F4: 10.58.131.14 "GET /metrics" 15ms 200 success
2021-08-05 18:29:36 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_8] INFO: 6F401082-76C2-4A1A-B3FB-7BDE8495D4C0: 10.58.131.18 "GET /metrics" 16ms 200 success
2021-08-05 18:29:37 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_28] INFO: 51C0FE1F-A92F-4C2F-8F8B-2607810D6A35: 10.58.131.14 "GET /metrics" 17ms 200 success
2021-08-05 18:29:40 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_18] INFO: 4DCAE8C6-C0FD-4C40-9CDA-2C325CC01984: 10.58.131.18 "GET /metrics" 14ms 200 success
2021-08-05 18:29:42 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_20] INFO: 4E89C44C-5793-45C4-91B4-EE3205C1902E: 10.58.131.14 "GET /metrics" 17ms 200 success

I believe I did everything specified from the documentation, can you please help me change the logs to level error.

passing annotations to underlying emissary ingress service

I would love to be able to create my AWS load balancer with accompanying SSL cert as described here while installing this helm chart but the values do not allow annotations to be passed to the underlying emissary ingress chart. Can this be added? Additionally this will allow folks using ArgoCD to deploy the edge stack as an Application using helm config like the below:

source:
    helm:
      releaseName: edge-stack
      values: |
        emissaryIngress:
          service:
            annotations:
              external-dns.alpha.kubernetes.io/hostname: XX
              service.beta.kubernetes.io/aws-load-balancer-backend-protocol: XX
              service.beta.kubernetes.io/aws-load-balancer-ssl-cert: XX
              service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: XX
              service.beta.kubernetes.io/aws-load-balancer-ssl-ports: XX
    repoURL: https://app.getambassador.io
    chart: edge-stack

what does this actually mean 0/2 nodes are available: 2 node(s) didn't have free ports for the requested pod ports ?

Recently i have upgraded my cluster to 1.21 from 1.19. I am unable to schedule the ambassador pods. Here are my logs

Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  86s (x2 over 87s)  default-scheduler  0/2 nodes are available: 2 node(s) didn't have free ports for the requested pod ports.
  Normal   Scheduled         80s                default-scheduler  Successfully assigned default/ambassador-77b44684fd-mphc5 to dev-master
  Normal   Pulled            78s                kubelet            Successfully pulled image "quay.io/datawire/ambassador:1.0.0" in 446.491739ms
  Normal   Pulled            62s                kubelet            Successfully pulled image "quay.io/datawire/ambassador:1.0.0" in 432.624481ms
  Normal   Pulling           30s (x3 over 79s)  kubelet            Pulling image "quay.io/datawire/ambassador:1.0.0"
  Normal   Created           29s (x3 over 78s)  kubelet            Created container ambassador
  Normal   Started           29s (x3 over 78s)  kubelet            Started container ambassador
  Normal   Pulled            29s                kubelet            Successfully pulled image "quay.io/datawire/ambassador:1.0.0" in 446.673825ms
  Warning  BackOff           11s (x4 over 46s)  kubelet            Back-off restarting failed container

I have checked with taints, since node doesnot have taints, there wont be issue with taints. Any help would be appreciated.

Can't configure mapping.docs with istio

My cluster configuration:

  • GKE v.1.22.6-gke.300
  • Istio v.1.13
  • Edge-stack chart v.7.3.2
    My mapping:
apiVersion: getambassador.io/v2
kind: Mapping
metadata:
  annotations:
    argocd.argoproj.io/tracking-id: dev-assessor-svc:getambassador.io/Mapping:services/dev-assessor-svc
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"getambassador.io/v3alpha1","kind":"Mapping","metadata":{"annotations":{"argocd.argoproj.io/tracking-id":"dev-assessor-svc:getambassador.io/Mapping:services/dev-assessor-svc"},"labels"
:{"argocd.argoproj.io/instance":"dev-assessor-svc"},"name":"dev-assessor-svc","namespace":"services"},"spec":{"circuit_breakers":[{"max_connections":1024,"max_pending_requests":1024,"max_retries":3,"prior
ity":"default"}],"docs":{"display_name":"assessor","path":"/openapi/"},"hostname":"saas-core-dev.haut.ai","prefix":"/service/assessor/","retry_policy":{"num_retries":10,"retry_on":"5xx"},"rewrite":"/servi
ce/assessor/","service":"assessor.services:80","timeout_ms":300000,"tls":"istio-upstream"}}
  creationTimestamp: "2022-02-22T08:27:06Z"
  generation: 7
  labels:
    argocd.argoproj.io/instance: dev-assessor-svc
  name: dev-assessor-svc
  namespace: services
  resourceVersion: "142119937"
  uid: 4680e784-6ce0-432e-b56b-926649811adb
spec:
  ambassador_id:
  - --apiVersion-v3alpha1-only--default
  circuit_breakers:
  - max_connections: 1024
    max_pending_requests: 1024
    max_retries: 3
    priority: default
  docs:
    display_name: assessor
    path: /openapi/
  host: saas-core-dev.haut.ai
  prefix: /service/assessor/
  retry_policy:
    num_retries: 10
    retry_on: 5xx
  rewrite: /service/assessor/
  service: assessor.services:80
  timeout_ms: 300000
  tls: istio-upstream

The issue I'm encountering can be visible in these 2 log lines from edge-stack pod:

emissary-ingress time="2022-03-22 15:00:27.2968" level=error msg="Get \"https://127.0.0.1:8443/service/assessor/openapi/\": http: server gave HTTP response to HTTPS client" func="github.com/datawire/apro/
v2/cmd/amb-sidecar/devportal/server.(*DevPortalHTTPClient).Get" file="github.com/datawire/apro/v2/cmd/amb-sidecar/devportal/server/http.go:82" CMD=amb-sidecar PID=15 THREAD=/devportal_fetcher URL="https:/
/127.0.0.1:8443/service/assessor/openapi/" clusterName=cluster_assessor_services_80_otls_istio_-0 component=devportal mapping=dev-assessor-svc.services mhost=saas-core-dev.haut.ai subsystem=fetcher url="h
ttps://127.0.0.1:8443/service/assessor/openapi/"
emissary-ingress time="2022-03-22 15:00:27.2969" level=error msg="GET failed https://127.0.0.1:8443/service/assessor/openapi/" func="github.com/datawire/apro/v2/cmd/amb-sidecar/devportal/server.(*Fetcher)
.retrieve.func2" file="github.com/datawire/apro/v2/cmd/amb-sidecar/devportal/server/fetcher.go:387" CMD=amb-sidecar PID=15 THREAD=/devportal_fetcher URL="https://127.0.0.1:8443/service/assessor/openapi/"
clusterName=cluster_assessor_services_80_otls_istio_-0 mapping=dev-assessor-svc.services subsystem=fetcher

I tried to set url instead of path: path: http://assessor.services/service/assessor/openapi/ but my service just disappears from DevPortal, however I don't get any errors.

Cannot access services mapped to Edge Stack: connection refused

I have applied the edge stack, mappings to the example quote service, host and listener. I cannot reach the services through the edge stack, attempts to connect to the nodeport connected to the API gateway result in connection refused.

I tried experimenting with metallb, but the IP that was assigned in the network was not reachable. So instead I tried to use a Nodeport to connect the gateway to a port on the machine running the cluster.

The nodeport is defined like this:


Name:                     ambassador-nodeport
Namespace:                ambassador
Labels:                   <none>
Annotations:              <none>
Selector:                 app.kubernetes.io/instance=edge-stack,app.kubernetes.io/name=edge-stack,profile=main
Type:                     NodePort
IP:                       10.106.8.192
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  30000/TCP
Endpoints:                10.244.0.162:8080,10.244.0.163:8080,10.244.0.164:8080
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  30443/TCP
Endpoints:                10.244.0.162:8443,10.244.0.163:8443,10.244.0.164:8443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

In the ambassador namespace, I have:

> kc get host
NAME           HOSTNAME   STATE   PHASE COMPLETED   PHASE PENDING   AGE
minimal-host   *  

> kc get listeners
NAME                       PORT    PROTOCOL   STACK   STATSPREFIX   SECURITY   L7DEPTH
edge-stack-listener-8080   8080    HTTPS                            XFP
edge-stack-listener-8443   30443   HTTPS                            XFP
http-listener  

> kc get svc
NAME                  TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
ambassador-nodeport   NodePort       10.106.8.192     <none>        80:30000/TCP,443:30443/TCP   16d
edge-stack            LoadBalancer   10.108.133.185   192.168.0.3   80:31976/TCP,443:31611/TCP   16d
edge-stack-admin      ClusterIP      10.96.105.37     <none>        8877/TCP,8005/TCP            16d
edge-stack-agent      ClusterIP      10.108.57.148    <none>        80/TCP                       16d
edge-stack-redis      ClusterIP      10.103.23.175    <none>        6379/TCP                     16d
quote-service         ClusterIP      10.97.220.80     <none>        80/TCP                       14d

my mapping in this namespace is defined like this:

> kc get mapping
NAME                          SOURCE HOST   SOURCE PREFIX                               DEST SERVICE     STATE   REASON
edge-stack-devportal                        /documentation/                             127.0.0.1:8500
edge-stack-devportal-api                    /openapi/                                   127.0.0.1:8500
edge-stack-devportal-assets                 /documentation/(assets|styles)/(.*)(.css)   127.0.0.1:8500
edge-stack-devportal-demo                   /docs/                                      127.0.0.1:8500
quote-backend                               /backend/                                   quote

If SSH into the node, and try to reach this quote service, I get:

node@node:~$ curl 10.97.220.80
{
    "server": "adorable-lemon-q36h3huq",
    "quote": "Nihilism gambles with lives, happiness, and even destiny itself!",
    "time": "2022-05-10T21:19:59.178675473Z"}
 

My expectation is now that, based on the above, if I curl localhost:30000/backend from within the machine, I should be able to get a quote. This is because there is a listener, a host, and a mapping applied.
However, this is the response:

node@node:~$ curl localhost:30000/backend
curl: (7) Failed to connect to localhost port 30000: Connection refused

I notice also that the LoadBalancer service above has a NodePort:

node@node:~$ curl localhost:31976/backend
curl: (7) Failed to connect to localhost port 31976: Connection refused

I hope this is enough information. My question is why I cannot connect to the nodeports from the localhost of the machine running the cluster, and access the quote service .
Thank you for your time.

Helm chart fails when referencing licenseKey secret - pods stuck in ContainerCreating

I'm running into an issue with the latest edge-stack helm chart. The pods get stuck in ContainerCreating due to a missing secret. This seems to happen when configuring the "licenseKey" attributes in values passed to the chart in order to reference a secret we're using for the license key we purchased.

Steps to reproduce:

  1. Create namespace and a secret with license key purchased
  2. Install helm chart for edge-stack - pass in values to reference secret name and key where license key is stored:
licenseKey:
  annotations: {}
  createSecret: false
  secretName: edge-stack-license
  value: license-key

Results:

  • Chart deploys successfully
  • Pods are stuck in ContainerCreating state
%
kubectl get pods -n edge-stack
NAME                                READY   STATUS              RESTARTS   AGE
edge-stack-58444544bb-bnfzd         0/1     ContainerCreating   0          10m
edge-stack-58444544bb-hp87z         0/1     ContainerCreating   0          10m
edge-stack-58444544bb-jnq74         0/1     ContainerCreating   0          10m
edge-stack-58444544bb-ltlqt         0/1     ContainerCreating   0          10m
edge-stack-58444544bb-pd9cp         0/1     ContainerCreating   0          10m
edge-stack-58444544bb-sktf4         0/1     ContainerCreating   0          10m
edge-stack-agent-5d5879b568-lk4k8   1/1     Running             0          10m
edge-stack-redis-56cdf99d69-zxhxk   1/1     Running             0          10m
  • Events that show up on the pods:
Events:
  Type     Reason       Age                   From               Message
  ----     ------       ----                  ----               -------
  Normal   Scheduled    7m42s                 default-scheduler  Successfully assigned edge-stack/edge-stack-58444544bb-bnfzd to ip-10-64-80-135.us-west-2.compute.internal
  Warning  FailedMount  7m41s                 kubelet            MountVolume.SetUp failed for volume "edge-stack-secrets" : failed to sync secret cache: timed out waiting for the condition
  Warning  FailedMount  5m39s                 kubelet            Unable to attach or mount volumes: unmounted volumes=[edge-stack-secrets], unattached volumes=[kube-api-access-2dn86 ambassador-pod-info edge-stack-secrets]: timed out waiting for the condition
  Warning  FailedMount  89s (x10 over 7m40s)  kubelet            MountVolume.SetUp failed for volume "edge-stack-secrets" : secret "edge-stack" not found
  Warning  FailedMount  69s (x2 over 3m24s)   kubelet            Unable to attach or mount volumes: unmounted volumes=[edge-stack-secrets], unattached volumes=[ambassador-pod-info edge-stack-secrets kube-api-access-2dn86]: timed out waiting for the condition
  • Secrets in namespace (the secret ("edge-stack") that the pods are trying to use for a volumeMount is missing):
%
kubectl get secrets -n edge-stack
NAME                                       TYPE                                  DATA   AGE
default-token-fpjzt                        kubernetes.io/service-account-token   3      23m
edge-stack-agent-token-284tp               kubernetes.io/service-account-token   3      10m
edge-stack-license                         Opaque                                1      23m
edge-stack-token-fswfj                     kubernetes.io/service-account-token   3      10m
sh.helm.release.v1.edge-stack-license.v1   helm.sh/release.v1                    1      23m
sh.helm.release.v1.edge-stack.v1           helm.sh/release.v1                    1      10m

Expected results:

  • Chart deploys successfully
  • Pods show up in a running state and secret should be present:
%
kubectl get pods -n edge-stack
NAME                                READY   STATUS    RESTARTS   AGE
edge-stack-58444544bb-29fch         1/1     Running   0          13m
edge-stack-58444544bb-5zh6f         1/1     Running   0          13m
edge-stack-58444544bb-qclf9         1/1     Running   0          13m
edge-stack-58444544bb-r98w6         1/1     Running   0          13m
edge-stack-58444544bb-tqgpz         1/1     Running   0          13m
edge-stack-58444544bb-wz6hf         1/1     Running   0          13m
edge-stack-agent-5d5879b568-xvnsf   1/1     Running   0          13m
edge-stack-redis-56cdf99d69-wl9h5   1/1     Running   0          13m

%
kubectl get secrets -n edge-stack
NAME                               TYPE                                  DATA   AGE
ambassador-internal                Opaque                                2      14m
default-token-fpjzt                kubernetes.io/service-account-token   3      65m
edge-stack                         Opaque                                1      14m
edge-stack-agent-token-9q8lf       kubernetes.io/service-account-token   3      14m
edge-stack-token-xbsrc             kubernetes.io/service-account-token   3      14m
fallback-self-signed-cert          kubernetes.io/tls                     2      14m
sh.helm.release.v1.edge-stack.v1   helm.sh/release.v1                    1      14m

Versions

  • Kubernetes - EKS (AWS) 1.21
  • Chart version: 7.3.2
  • Edge Stack Version: 2.2.2

Other Notes

  • When I remove the licenseKey values passed in, the chart installs properly and the pods show up in a running state but it doesn't use our license key.
  • In our automation we're inserting the license key in a secret before the helm chart is run, which is why we need to be able to use these in the values passed into the chart.

Host - mapping associating not working.

Hi,
I've tried to make my configuration in edge-stack like here:
https://www.getambassador.io/docs/edge-stack/latest/topics/running/host-crd/

But despite my label selector, all hosts gets all mappings.

mappings:

apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
  generation: 1
  labels:
    host-nb_outlet: nb_outlet
  name: isearch-wasiol
  namespace: test-edgestack
spec:
  ambassador_id:
  - --apiVersion-v3alpha1-only--default
  precedence: 2
  prefix: /wasiol
  rewrite: ""
  service: isearch.test-edgestack
  timeout_ms: 0
  hostname: a2d461472a026efc.ci.merce.cloud

---

apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
  generation: 1
  labels:
    host-raccoon: raccoon
  name: iuploader-wasiol
  namespace: test-edgestack
spec:
  ambassador_id:
  - --apiVersion-v3alpha1-only--default
  precedence: 2
  prefix: /wasiol
  rewrite: ""
  service: iuploader.test-edgestack
  timeout_ms: 0
  hostname: a3e23aacf7a776c2.ci.merce.cloud

and host:

apiVersion: getambassador.io/v2
kind: Host
metadata:
  name: raccoon
  namespace: test-edgestack
spec:
  ambassador_id:
  - --apiVersion-v3alpha1-only--default
  hostname: a3e23aacf7a776c2.ci.merce.cloud
  requestPolicy:
    insecure:
      action: Route
      additionalPort: -1
  selector:
    matchLabels:
      host-raccoon: raccoon
---
apiVersion: getambassador.io/v2
kind: Host
metadata:
  name: nb-outlet
  namespace: test-edgestack
spec:
  ambassador_id:
  - --apiVersion-v3alpha1-only--default
  hostname: a2d461472a026efc.ci.merce.cloud
  requestPolicy:
    insecure:
      action: Route
      additionalPort: -1
  selector:
    matchLabels:
      host-nb_outlet: nb_outlet

Here the mapping for one domains:
foo
foo

Is there any misconfiguration been done? or it's just not working?

oauth2 filter disable mulitcookie

I used oauth2 filter and configured multiple domain names in protectedOrigins. When using one of the domain names for single sign-on, the mulitcookie will automatically redirect the request to another domain name. Can I turn this off? We do not need to share cookies.

Error: unable to build kubernetes objects from release manifest

Error: unable to build kubernetes objects from release manifest: [unable to recognize "": no matches for kind "AuthService" in version "getambassador.io/v3alpha1", unable to recognize "": no matches for kind "Mapping" in version "getambassador.io/v3alpha1", unable to recognize "": no matches for kind "RateLimitService" in version "getambassador.io/v3alpha1"]

about

I'm submitting this more as a resolved issue we encountered when installing via helm than an actual issue. It would probably be better to have this as a section of an FAQ, but I couldn't find a better place to put it that's discoverable via search engine. A few months ago we trialed out Ambassador and determined it was something we wanted to use. We uninstalled the old trial version likely in an improper fashion by deleting the namespace rather than helm uninstall or kubectl delete, and that left behind the lint of the old Ambassador customResourceDefinitions. When we went to reinstall, we followed the directions for a new installation since we thought that the old installation was completely gone, but the presence of the old CRDs gave us the aforementioned error message.

solution

Deleting all the older Ambassador CRDs and re-running the helm install command resolved the issue for us.

edge-stack is failing to route traffic on port 8443

edge-stack is failing to route traffic on port 8443 after exposing it, with HTTP 404 error.

Casey did a little bit of digging, and according to him the virtual host and the server names are not propagated correctly for some reason.

The port is exposed to Kubernetes:

kubectl get services
NAME               TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                                     AGE
ambassador         LoadBalancer   11.111.222.111   localhost     80:31769/TCP,443:31982/TCP,8443:31695/TCP   54d

We do use the lua script in the Module that strips the port from authority.

apiVersion: getambassador.io/v3alpha1
kind:  Module
metadata:
  name:  ambassador
  namespace: alpha-ambassador
spec:
  config:
  # Use the items below for config fields
    lua_scripts: |
      function envoy_on_request(request_handle)
        local authority = request_handle:headers():get(":authority")
        if(string.find(authority, ":") ~= nil)
        then
          local authority_index = string.find(authority, ":")
          local stripped_authority = string.sub(authority, 1, authority_index - 1)
          request_handle:headers():replace(":authority", stripped_authority)
        end
      end

This is the Mapping object:

apiVersion: getambassador.io/v2
kind: Mapping
metadata:
  name: guestbook-mapping
spec:
  grpc: false
  host: "^example\\.com:8443$"
  host_regex: true
  prefix: /
  rewrite: ""
  service: guestbook:80

We also tried to add this Host object, but it didn't help:

apiVersion: getambassador.io/v2
kind: Host
metadata:
  name: example.com-8443
spec:
  acmeProvider:
    authority: none
  hostname: '*.example.com:8443'
  selector:
    matchLabels:
      hostname: example.com:8443
  tls:
    alpn_protocols: h2
  tlsSecret:
    name: example-com-server-secret

Please sync the Ambassador Edge Stack version with the Helm CHART/APP VERSION

Upgrades are coming soon so I need to install a specific version of AES: 2.1.1, then upgrade to the latest from there.

No matter what is done, this seems to be the result:

% helm list --namespace ambassador
NAME      NAMESPACE 	UPDATED                 STATUS  	CHART           	APP VERSION
aes-test  ambassador	2022-07-06 14:35	deployed	edge-stack-8.0.0	3.0.0

Used the CRDs for that version: https://app.getambassador.io/yaml/edge-stack/2.1.1/aes-crds.yaml

Since this is not latest version of aes, it appears I need to install from the official Helm repo so I can specify a version. I've selected Helm chart v7.2.1, which should deploy APPLICATION VERSION 2.1.1.

When that didn't work, I downloaded/pointed to a values file that specifies:

  image:
    repository: docker.io/datawire/aes
    tag: 2.1.1
    pullPolicy: IfNotPresent

After spending way too much time on this, I finally exec'd into the shell for one of the deployed containers:

$ ambassador --version
Ambassador 2.1.1
Ambassador Scout version 2.1.1
Ambassador Scout semver  2.1.1

It was installing the correct version all along.

Please fix this :-)

Edge-stack-agent is unable to get Ambassador Cloud token when AGENT_CONFIG_RESOURCE_NAME points to a secret.

AES Version: 3.0.0

When the AGENT_CONFIG_RESOURCE_NAME environment variable points to a ConfigMap, edge-stack-agent is able to get the token to connect to Ambassadro Cloud. However, when AGENT_CONFIG_RESOURCE_NAME points to a Secret, AES agent does not get the token, which will break the integration with Ambassador Cloud.

The following log is printed by AES agent when the token is stored in a ConfigMap:

time="2022-07-12 20:47:53.1228" level=info msg="/usr/bin/python3 /ambassador/kubewatch.py --debug failed with exit status 1\n\n" func=github.com/emissary-ingress/emissary/v3/pkg/environment.EnvironmentSetupEntrypoint file="/go/pkg/environment/helper.go:45" CMD=agent PID=1
time="2022-07-12 20:47:53.1234" level=info msg="metrics service listening on :8080" func=github.com/emissary-ingress/emissary/v3/cmd/agent.run file="/go/cmd/agent/main.go:59" CMD=agent PID=1
time="2022-07-12 20:47:53.1247" level=info msg="Agent is running..." func="github.com/emissary-ingress/emissary/v3/pkg/agent.(*Agent).Watch" file="/go/pkg/agent/agent.go:290" CMD=agent PID=1 THREAD=/watch
time="2022-07-12 20:47:53.4365" level=info msg="Setting cloud connect token from configmap" func="github.com/emissary-ingress/emissary/v3/pkg/agent.(*Agent).handleAPIKeyConfigChange" file="/go/pkg/agent/agent.go:264" CMD=agent PID=1 THREAD=/watch
time="2022-07-12 20:47:53.5745" level=info msg="WatchGeneric: Listening for events from resouce \"argoproj.io/v1alpha1, Resource=rollouts\"" func="github.com/emissary-ingress/emissary/v3/pkg/agent.(*DynamicClient).WatchGeneric" file="/go/pkg/agent/k8s.go:138" CMD=agent PID=1 THREAD=/watch
time="2022-07-12 20:47:53.5748" level=info msg="WatchGeneric: Listening for events from resouce \"argoproj.io/v1alpha1, Resource=applications\"" func="github.com/emissary-ingress/emissary/v3/pkg/agent.(*DynamicClient).WatchGeneric" file="/go/pkg/agent/k8s.go:138" CMD=agent PID=1 THREAD=/watch
time="2022-07-12 20:47:53.5749" level=info msg="Beginning to watch and report resources to ambassador cloud" func="github.com/emissary-ingress/emissary/v3/pkg/agent.(*Agent).watch" file="/go/pkg/agent/agent.go:408" CMD=agent PID=1 THREAD=/watch
time="2022-07-12 20:47:53.8178" level=error msg="failed to validate OpenAPI spec: invalid components: unsupported 'format' value \"uuid\"" func=github.com/emissary-ingress/emissary/v3/pkg/agent.newOpenAPI file="/go/pkg/agent/api_docs.go:252" CMD=agent PID=1 THREAD=/watch
time="2022-07-12 20:47:54.0161" level=info msg="Connected to the CEPC Director" func="github.com/emissary-ingress/emissary/v3/pkg/agent.(*BasicDirectiveHandler).HandleDirective" file="/go/pkg/agent/directive_handler.go:47" CMD=agent PID=1 THREAD=/watch directive=1657658874-e974421e-8d4a-4792-827e-4ff0a1c807ce
time="2022-07-12 20:47:55.7512" level=info msg="Received 1307 metric(s)" func="github.com/emissary-ingress/emissary/v3/pkg/agent.(*Agent).MetricsRelayHandler" file="/go/pkg/agent/agent.go:655" CMD=agent PID=1 THREAD="/metrics-server/conn=10.56.8.107:8080"
time="2022-07-12 20:47:55.7514" level=info msg="Relaying 80 metric(s)" func="github.com/emissary-ingress/emissary/v3/pkg/agent.(*Agent).MetricsRelayHandler" file="/go/pkg/agent/agent.go:679" CMD=agent PID=1 THREAD="/metrics-server/conn=10.56.8.107:8080"
time="2022-07-12 20:47:55.7530" level=info msg="Next metrics relay scheduled for 2022-07-12 20:48:25.753060963 +0000 UTC" func="github.com/emissary-ingress/emissary/v3/pkg/agent.(*Agent).MetricsRelayHandler" file="/go/pkg/agent/agent.go:687" CMD=agent PID=1 THREAD="/metrics-server/conn=10.56.8.107:8080"

These are the agent logs when token is in a secret:

time="2022-07-12 20:49:32.2591" level=info msg="/usr/bin/python3 /ambassador/kubewatch.py --debug failed with exit status 1\n\n" func=github.com/emissary-ingress/emissary/v3/pkg/environment.EnvironmentSetupEntrypoint file="/go/pkg/environment/helper.go:45" CMD=agent PID=1
time="2022-07-12 20:49:32.2598" level=info msg="metrics service listening on :8080" func=github.com/emissary-ingress/emissary/v3/cmd/agent.run file="/go/cmd/agent/main.go:59" CMD=agent PID=1
time="2022-07-12 20:49:32.2608" level=info msg="Agent is running..." func="github.com/emissary-ingress/emissary/v3/pkg/agent.(*Agent).Watch" file="/go/pkg/agent/agent.go:290" CMD=agent PID=1 THREAD=/watch
time="2022-07-12 20:49:32.5251" level=info msg="Setting cloud connect token from environment" func="github.com/emissary-ingress/emissary/v3/pkg/agent.(*Agent).handleAPIKeyConfigChange" file="/go/pkg/agent/agent.go:275" CMD=agent PID=1 THREAD=/watch

This is how the egde-stack-agent environment configuration looks like:

  {
    "name": "AGENT_CONFIG_RESOURCE_NAME",
    "value": "edge-stack-agent-cloud-token"
  },

Note: When token is stored in a secret, it's base64 encoded.

Least Privilege on RBAC Permissions

Please describe your use case / problem.

A review of the RBAC permissions given to the edge-stack pods seems overly permissive. Here are some examples:

https://github.com/emissary-ingress/emissary/blob/5e03b912c048c2db25763dbf77265792199ebbad/charts/emissary-ingress/templates/rbac.yaml#L87-L90

https://github.com/datawire/edge-stack/blob/main/charts/edge-stack/templates/rbac.yaml#L27-L29

Does this actually need to read every secret in every namespace?

Similarly, its allowed to delete any CRD.

rules:
  - apiGroups: [ "apiextensions.k8s.io" ]
    resources: [ "customresourcedefinitions" ]
    verbs: ["get", "list", "watch", "delete"]

Describe the solution you'd like

At minimum, it would be nice to have an explanation of what's going on that seems to require these permissions. Better would be to be a bit more verbose about what secrets, CRDs, etc that actually need to be managed.

Describe alternatives you've considered

Disabling unsavory permissions until I've had a chance to review the code and/or see errors in the logs.

Additional context
n/a

Originate-TLS context ambassador-consul is not defined

After upgrading from 7.2.2 to 7.3.1 all of our services are unreachable with errors like

2022-08-30 20:24:29 diagd 2.2.1 [P30TAEW] INFO: cluster_goldilocks_dashboard_sidecar_proxy_goldilocks_dashboard_sidecar_proxy_goldilocks_er_round_robin: <RichStatus BAD error='Originate-TLS context ambassador-consul is not defined' hostname='ambassador-fb6d6967f-x8csz' version='2.2.1'>

Is there something new in the chart we need to define or something else?

Issue with Lets Encrypt and Host CRD

I have AWS EKS behind AWS Load Balancer
Lets Encrypt doesn't work at all with AWS NLB, and by using AWS Classic Load Balancer, i am able to register ONLY one host, after that for every other host i am getting the error ACME 403 Unauthenticated
this same error i am getting also when i use AWS NLB
error:

obtaining tlsSecret "test1.mydomain.com"."ambassador"
    (hostnames=["test1.mydomain.com"]): acme: Error -> One or more domains had
    a problem:

    [test1.mydomain.com] acme: error: 403 ::
    urn:ietf:params:acme:error:unauthorized :: Invalid response from
    http://test1.mydomain.com/.well-known/acme-challenge/NM0XccervQ1Ldjm-50dsdf2F5qrZ2fdfsXqjyiuvium0V-tI

 authority: https://acme-v02.api.letsencrypt.org/directory

The single validated host (test.mydomain.com), with AWS Classic Load Balancer, is reachable and doesn't have any other issue
Setup:

apiVersion: getambassador.io/v3alpha1
kind: Host
metadata:
  name: test
  namespace: ambassador  
spec:
  hostname: "test.mydomain.com"
  acmeProvider:
    email: [email protected]
    authority: https://acme-v02.api.letsencrypt.org/directory
  requestPolicy:
    insecure:
      action: Redirect
      additionalPort: 8080
---
apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
  name: test
  namespace: ambassador
spec:
  host: "test.mydomain.com"
  prefix: "/"
  service: "nginx.default:80" 
---
apiVersion: getambassador.io/v3alpha1
kind: Host
metadata:
  name: test1
  namespace: ambassador  
spec:
  hostname: "test1.mydomain.com"
  acmeProvider:
    email: [email protected]
    authority: https://acme-v02.api.letsencrypt.org/directory
  requestPolicy:
    insecure:
      action: Redirect
      additionalPort: 8080
---
apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
  name: test1
  namespace: ambassador
spec:
  host: "test1.mydomain.com"
  prefix: "/"
  service: "nginx1.default:80" 

EKS 1.21 (newly created - Edge Stack is the first resource)
Edge-stack 2.0.5

Custom licenseKey.secretName not found

Problem

Providing custom licenseKey.secretName makes pods unable to start due to secret not found.

How to reproduce

helm repo add edge-stack https://s3.amazonaws.com/datawire-static-files/charts
helm repo update
helm install edge-stack edge-stack/edge-stack --set licenseKey.secretName=license-secret

Reason

Here .Values.licenseKey.secretName is resolved in the context of the subchart (emissary-ingress), where this value doesn't exist.

volumesRaw: |
- name: {{ include "ambassador.fullname" . }}-secrets
secret:
{{- if and .Values.licenseKey .Values.licenseKey.secretName }}
secretName: {{ .Values.licenseKey.secretName }}
{{- else }}
secretName: {{ include "ambassador.fullname" . }}
{{- end }}

It is possible to get this deployed correctly if both licenseKey.secretName and emissary-ingress.licenseKey.secretName are set like so:

helm upgrade edge-stack edge-stack/edge-stack --set licenseKey.secretName=license-secret --set emissary-ingress.licenseKey.secretName=license-secret

Possible solution

This can be fixed by moving licenseKey parameter to global: block. This will make it available in both edge-stack and emissary-ingress charts context.

tolerations for agent deployment missing

We use a system node pool and user node pool on aks. The system node pool is tainted. The edge stack helm chart only supports tolerations for the redis pods but not for the agent deployment. I saw, the old ambassador helm chart did support tolerations. Is there a reason this was removed? When will we expect this feature to reappear so the ambassador agent also can run on tainted nodes?

Why the need to `update` CRDs?

I'd like to disable ambassador from making changes to CRDs (even it's own). If this is part of auto-updating, I'd like to delete that.

External IP returning 502

I have a testing cluster with ambassador 1.9 version in GCP. As per the docs, I have updated the same to ambassador version 1.14 and everything is working fine. But on upgrading from 1.14 to 2.3.2 according to the docs eliminating downtime(running both the versions in parallel), my ingress backends have become unhealthy and external ip is now returning an 502.

I have applied all the resources according to the docs including the listener, host, etc.

ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.global-static-ip-name: testapp-api-ip
networking.gke.io/managed-certificates: api-cert
name: api-services
spec:
defaultBackend:
service:
name: edge-stack
port:
number: 80
status:
loadBalancer:
ingress:
- ip: xx.xxx.xx.xx

Update:
Since upgrading from 1.14 is not working. I have tried a different approach to remove the current ambassador resources and applied a fresh version of ambassador 2.3.4, still it's not working. NEG is unhealthy, health check path is /ambassador/v0/check_ready

Any sort of help is appreciated. Please comment for more details.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.