Coder Social home page Coder Social logo

Comments (10)

chrisohaver avatar chrisohaver commented on September 25, 2024

As separate server blocks, and without the reverse zone fallthrough, this looks something like...

    <CLUSTER_DOMAIN> <REVERSE_ZONES...> {
        health
        ready
        errors
        kubernetes . {
          pods insecure
        }
        loadbalance
    }

    . {
        prometheus :9153
        forward . /etc/resolv.conf
        loop
        cache 30
        reload
    }

Although it has been urged that we do so, it's not clear to me what the concrete benefit of splitting up into multiple server blocks is. There is some efficiency gained, i think, since the split up plugin chains are shorter than the single chain. Each query traverses less plugins (though I think that in itself would be a fairly minor gain). I'm fuzzy about how this relates to plugins like prometheus... would we log metrics for queries routed to both server blocks in the above example?

I think this would leave kubernetes records uncached. But I think is good, since kubernetes already caches data in its client-go connection to the k8s API.

Splitting into multiple blocks means we cannot use fallthrough in kubernetes to forward queries to the upstream name servers. This requires us to define the reverse zones as only those which we are authoritative for.

One detail that makes this impractical to deploy is that we don't have a programatic way get all the reverse zones of the kubernetes cluster. In some clusters pod CIDR ranges are defined per node, so the list of zones can change as nodes are added/removed. We could instead ...

  • make the server block authoritative for all reverse zones as we did before but without fallthrough, not allowing any reverse lookups upstream. This technically would not violate the dns spec, although I don't think it's a very usable solution. (I think this is how kube-dns did it, though i'm not 100% certain)
  • make the server block not authoritative for any reverse zones, but this would violate the DNS spec.

@fturib, you also had a concern about multiple zones for a single server block. Can you elaborate on that here?

from deployment.

chrisohaver avatar chrisohaver commented on September 25, 2024

FYI ... @miekg @johnbelamaric @fturib @rajansandeep

from deployment.

fturib avatar fturib commented on September 25, 2024

Yes, from Caddy behavior, the setup function of each plugin will be called as many keys are defined in the ServerBlock. It is defined in executeDirectives here

In other words, with such configuration we would have as many instance of Kubernetes structure as Zones that are defined in this server block:

<CLUSTER_DOMAIN> <REVERSE_ZONES...> {
       ...
        kubernetes . {
          pods insecure
        }
        ...
    }

And each of the kubernetes structure will contains the object caches of k8s Client API.

I guess we cannot go this way because of scalability issue depending size of the cluster

We may want also to fix that useless (I think so far - but not completely sure) duplication by revising the function InspectServerBlocks here

from deployment.

chrisohaver avatar chrisohaver commented on September 25, 2024

@fturib, How do other plugins cope with that, e.g. health, metrics. Multiple instances of those would collide on the listening port.

from deployment.

fturib avatar fturib commented on September 25, 2024

It is not manage by other plugins.
For metrics the problem is larger than only the KEY at the ServerBloc. The same listening port could be share between several plugin instance on different servers. So anyway there is already a mechanism to share the same listener for several instances of the same plugin.

For health I am not sure how that works with multiple KEY ... does it?
looks like it does not:

francois.com:5553 francois.fr:5553{
    health
 }

log of CoreDNS:

listen tcp :8080: bind: address already in use
Process finished with exit code 1

from deployment.

chrisohaver avatar chrisohaver commented on September 25, 2024

So it seems that at least until the plugin instance duplication for multi zone blocks is fixed, we will have to stick to a single stanza for now.

The question remains whether or not we should cease using fallthrough for reverse domains. It becomes a matter of choosing the lesser evil...

  1. Continue to use fallthrough, and suffer performance impact for reverse zone queries of IPs that are not in the kubernetes cluster.
  2. Stop using fallthrough, but still be authoritative for all reverse zones, thus disabling reverse zone queries of IPs that are not in the kubernetes cluster.
  3. Stop using fallthrough, and do not be authoritative for any reverse zones, violating the k8s dns spec.

I think the lesser evil is 1. I don't think the performance impact is great enough to warrant going with 2 or 3.

from deployment.

chrisohaver avatar chrisohaver commented on September 25, 2024

per @johnbelamaric's experiments, it looks like client-go employs some form of connection sharing - so the memory thing may not be an issue. However it depends on which level the connections are shared (e.g. is there a single network connection, but multiple cache/stores, or is there a single cache store, etc). Further testing/experimentation required here.

from deployment.

johnbelamaric avatar johnbelamaric commented on September 25, 2024

I think just the connection is shared. Meaning the caches are duplicated.

from deployment.

chrisohaver avatar chrisohaver commented on September 25, 2024

I think just the connection is shared. Meaning the caches are duplicated.

I have more or less confirmed this when testing coredns/coredns#3862. With coredns/coredns#3862, the caches are not duplicated for each zone in the server block, and therefore the memory footprint drops (considerably).

from deployment.

johnbelamaric avatar johnbelamaric commented on September 25, 2024

Nice, I will try to review that soon.

from deployment.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.