feature: add ability to filter additional host info about fleet

[zwass]

The recommendation exists because the implementation of additional_queries is not optimized or profiled for high volumes of data.

Are you using it successfully with a large volume of data/hosts? It would be interesting to understand (1) what your use case is and (2) how your infrastructure is configured and how it is performing.

Overall I'm not opposed to the idea of allowing filtering on the additional info, but I think it may be necessary to continue iterating on the concept of additional_queries based on what we learn of real-world uses.

[billcobbler]

We're considering a use case to integrate Fleet with a vuln scanner and provide the vuln scanner with detailed host info such as packages installed on hosts. The goal is to remove the need to run another security agent on our hosts. We haven't enabled additional queries on our real fleet, but we did a load test with 20k ubuntu clients to get feel for performance impact. We used additional queries to pull deb packages with deb_packages: SELECT * FROM deb_packages.

Rough performance we saw at 20k clients (5m config refresh) including API calls to pull ~1k hosts at a time (without this filtering feature):
EC2: 2x c5.large, each at ~15% cpu at steady state
RDS: db.m5.large, at ~22% cpu at steady state
Redis: cache.t2.micro at < 5% cpu at steady state

That lead us to feel it was feasible to use additional queries to gather this data, but that filtering the results would be nice to limit the amount of data sent/received by the frontend (or anything else that doesn't care about additional info).