Plugin annotations do not work with multiple hosts about kubernetes-ingress-controller

[hbagdi]

@jaygorrell,

I can't provide a timeline for this feature yet but this is something that would solve multiple problems.

Alternatively, is there a way to add a global plugin with Kubernetes Ingress? That would help a lot.

Unfortunately no. There is no mechanism to apply global plugins via the ingress controller yet. Stay tuned, this might be a smaller problem to solve.

[hbagdi]

Last question (because I'm derailing here): until labels are added, which resources are unsafe to manually add? I assume I can probably manually configure the global plugins at least and save 100s of resources here.

I'm sorry to say but you shouldn't manage any resource manually in Kong when you're using Kong as an Ingress controller. At this point, the only resort here is to temporarily create all KongPlugin resources. We will be trying to figure out an approach to make it possible to create global plugins via the ingress controller for the short term but in future, KongPlugin resources will be reusable.

[hbagdi]

So the re-use of plugins works with Service annotations, but not Ingress? That seems to be what @Dag24 is saying but I would have expected similar behavior based on the problem's description.

You can't reuse plugin configurations in services either but you can apply the plugin at the service level instead of route level using an annotation in the k8s service.

[jaygorrell]

Ah, that makes sense. I didn't realize they'd actually be applied at the service level in kong. That might be more fitting for my use-case anyway from a logical perspective since each domain/entrypoint never uses different plugins.

[jaygorrell]

A bit more information on this:
I tried splitting the example into 3 different Ingress resources and got the same result... the plugin would only create on one of the routes. I thought for sure that would be an ugly workaround at least.

I then tried duplicating the KongPlugin resource with a different name for each route and using that. This worked, but is obviously even more messy than the duplicate Ingress idea because it requires duplicating both.

Here's an example showing this

apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: add-header
config:
  header_name: X-Request-ID

---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: mytest-foo
  annotations:
    kubernetes.io/ingress.class: kong
    correlation-id.plugin.konghq.com: add-header
spec:
  rules:
  - host: foo.com
    http:
      paths:
      - path: /foo
        backend:
          serviceName: myservice
          servicePort: 80

---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: mytest-bar
  annotations:
    correlation-id.plugin.konghq.com: add-header
spec:
  rules:
  - host: bar.com
    http:
      paths:
      - path: /bar
        backend:
          serviceName: myservice
          servicePort: 80

The result from the above is that 2 routes and 1 service are created, as expected. But unfortunately only one plugin is created and it is assigned to one of the two routes.

[hbagdi]

Hello @jaygorrell,

The duplication of KongPlugin resource is a known limitation currently which is necessitated by the fact that UUID of the KongPlugin k8s resource is used as the UUID of the plugin entity in Kong.
We've on our roadmap to avoid this duplication. But as you found out, the temporary workaround is to create a dedicated Ingress and KongPlugin resource.

[jaygorrell]

Eek, I was afraid of that.

Thanks for the response but we essentially have 6 global plugins in our current Kong setup so to expose all 50 services, it sounds like we would need to have 300 plugin resources. Do you have a rough timeline on that being addressed? If it's relatively soon (month or two) I could set things up for how we expect it to work and just wait on the fixes.

Alternatively, is there a way to add a global plugin with Kubernetes Ingress? That would help a lot.

[jaygorrell]

Last question (because I'm derailing here): until labels are added, which resources are unsafe to manually add? I assume I can probably manually configure the global plugins at least and save 100s of resources here.

[3dbrows]

Not sure if this helps @jaygorrell but I have a very similar setup to yours, and it's also possible to set plugin annotations on the Service itself which is how I achieve a plugin being applied to multiple hosts, as follows. (I don't work for Kong, I'm just a user.)

apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: add-header
config:
  header_name: X-Request-ID

---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: mytest-foo
  annotations:
    kubernetes.io/ingress.class: kong
spec:
  rules:
  - host: foo.com
    http: &http_rules
      paths:
      - path: /foo
        backend:
          serviceName: myservice
          servicePort: 80
  - host: bar.com
    http: *http_rules

---

apiVersion: v1
kind: Service
metadata:
  name: myservice
  annotations:
    correlation-id.plugin.konghq.com: add-header
spec:
  selector:
    app: myservice
  ports:
  - port: 80
    targetPort: 3000

Obviously, this requires that you are in a position to apply such an annotation on your Service. But now, myservice will get the add-header plugin on foo.com/foo and bar.com/foo.

[jaygorrell]

@Dag24 Thanks, that's pretty interesting but I wonder if it's intentional or not. The documentation indicates that annotations are for the Ingress resource. It feels a little disconnected in a weird way, too.

If this is intentional though, it really isn't a bad workaround at all and could fit in my use-case.

[hbagdi]

@jaygorrell @Dag24
As documented here, this is indeed intentional.

Having said that, the underlying problem here is the controller currently doesn't reuse KongPlugin objects. I've been thinking a little bit about the problem.
It seems it is possible to avoid this duplication if we rewrite the plugin sync logic in the controller.
The controller would not compare IDs of the plugin but compare the plugin names and their configs.
This would complicate the logic a little bit but is worth it.

Once the reuse of KongPlugin problem is solved, it will open up doors for a lot of usability enhancements like this current issue.

Stay tuned for updates!

[jaygorrell]

I stand corrected! I missed that plugins can be applied to services instead of Ingress. Thanks @hbagdi!

So the re-use of plugins works with Service annotations, but not Ingress? That seems to be what @Dag24 is saying but I would have expected similar behavior based on the problem's description.

[hbagdi]

After #121, KongPlugin resource can be reused across multiple Ingress and/or Service objects.
This feature is now merged into master.

@jaygorrell Next release of Ingress Controller will introduce support for Global plugins as well allow reusing KongPlugin resources across services.

Closing this, please re-open if otherwise.

Thank you!

[jaygorrell]

@hbagdi Awesome. Does that also address the multiple host entry issue in Ingress resources? That was the scope of this issue originally.

[hbagdi]

@hbagdi Awesome. Does that also address the multiple host entry issue in Ingress resources? That was the scope of this issue originally.

Yes, it will create a plugin for each host entry.