kolide / fleet

A flexible control server for osquery fleets

Fleet is Retired

This notice is to inform you that on Wednesday, November 4th, 2020 Kolide has officially retired Fleet. This repository will be left up in an archived state for posterity, and to assist existing Fleet users.

Please refer to the FAQ below for additional information including migration options.

FAQ

Q: What was Kolide Fleet?

Kolide Fleet was an open-source Osquery Fleet Manager written in Go and Javascript.

Q: What does retiring entail?

Effective immediately, Kolide will no longer promote, endorse, support, or update Fleet. Any infrastructure outside of Github supporting downloads or packages will also be retired.

Since we will not be updating Fleet with any security patches or accepting bug bounty reports, it is important that you find a suitable alternative right away.

Q: Is Kolide retiring Kolide Launcher?

No. Kolide Launcher is a major component of our SaaS product, and we fervently believe in keeping the endpoint agent technology open-source.

Q: Why is Kolide retiring Fleet?

We are proud and humbled by all of the organizations using Fleet today. We think it’s a great solution to monitor server-based infrastructure using Osquery. We recognize many organizations depend on Fleet so retiring it was not a decision we took lightly.

Since 2019, Kolide has found success building a SaaS endpoint security product around the philosophy of User Focused Security. This philosophy is about building a foundation of trust between the security team and the end-users of a given organization.

When Fleet is used to obtain visibility on end-user devices, it is not software that enables an honest and accountable relationship between the security team and the end-users who are subject to the data collection Fleet enables.

Specifically, Fleet allows administrators to vacuum up large quantities of end-user data without providing those users with any visibility or tools to understand or control that process. Fleet does not hold security practitioners accountable for the data they collect or how it will be ultimately used. In fact, many folks reach out to us asking how to use Fleet and Osquery together to collect things like web browser history, device geolocation, and other personal information from devices. These objectives are antithetical to our company’s values. At Kolide these values are non-negotiable and we no longer want to enable software under our name and branding that enables those use-cases.

While we could change Fleet to meet these ideals and be compatible with our philosophy, we would rather invest that same energy and resources toward improving our SaaS app.

Given the facts above, we have decided it is in our best interest to retire the project and thoroughly explain our rationale.

Q: Why retire Fleet now?

Kolide has not substantially contributed to Fleet for over 2 years. Recently, an active contributor reached out to let us know they were planning on forking it and asked for our blessing. This recent reach-out caused us to re-evaluate if it’s something Kolide wants to continue to directly invest in or endorse.

After going through that exercise, we determined that Fleet was built by a different Kolide; one that embodied different values and had different goals. Due to the mismatch in ideals and the opportunity to distance ourselves from Fleet, we have decided now is the best time to retire the project and let the community decide on a suitable fork to follow.

Q: What should I use instead of Fleet?

Kolide no longer endorses the use of Fleet for monitoring end-user devices.

If you are looking to get additional visibility on end-user devices and want to do it honestly, you should check out Kolide K2. It’s a refreshing take on how to use technologies like Osquery and Slack to achieve the security team’s goals while simultaneously respecting end-users and their privacy.

If you use Fleet to monitor servers and are looking for a direct migration path https://github.com/fleetdm/fleet appears to be the first of many promising forks.

We will update this space with forks that best support that use case.

Also, since Fleet is open source under the permissive MIT license, you are of course free to fork it and build your own version. Our only ask is you do not associate Kolide or our branding as a means of promoting or endorsing any derivative of Fleet you create.

Q: Where is the fleet source code

Security vulnerabilities have been reported since we formally retired Fleet. As such, we have deleted the code. Older version can be found in the commit history and the past releases.

Kolide SaaS

Want to go directly to insights?

Try our osquery SaaS platform providing insights, alerting, fleet management and user-focused security tools. We also support advanced aggregation of osquery results for power users. Get started immediately, with your 14-day free trial today. Launcher packages customized for your organization can be downloaded in-app after signup.

fleet's Issues

Write benchmarks and tests for JWT auto-renewal middleware

Record osqueryd TLS requests for use in future testing

Per @groob's suggestion.

https://golang.org/pkg/net/http/httputil/#DumpRequest

Create a handler that dumps the request to a file. Upload these requests in a zip file to Drive.

Implement more robust dependency management solution

Right now, most dependencies pull from the master of each repository when they're imported / go got. We should look into the Go dependency management ecosystem and use a tool, like godep or something similar, to manage our dependencies reliably and offer 100% repeatable builds.

Create a session store backed by MySQL

Right now, sessions are stored on the client in a cookie. Change this to use MySQL instead. One day, when this is too slow, we'll probably have to add memcached/redis here.

Add admin APIs for managing the sessions in MySQL

Once sessions are backed in MySQL, we will need to expose HTTP APIs so that the application can have session management functionality.

For example:

• Delete all sessions for a user (DELETE /api/v1/kolide/user/session)
• Delete a single session for a user (DELETE /api/v1/kolide/user/session/{id})

config.LoadConfig ignores default config values

Because LoadConfig assigns to the config objects, the default values are eliminated. We should either fix this so it is not the case, or remove the default values entirely.

https://github.com/kolide/kolide-ose/blob/master/config/config.go#L86

Serve HTTP as well as HTTPS servers on kolide serve

So that users can serve the HTTP server on 80 and the HTTPS on 443 and not have to configure vhosts at all. Ideally the HTTP site should just redirect.

Dockerize Apps and Deps

As a developer I want the ability to be able to bootstrap the app and deps instantly with something like docker.

Figure out the transition from travis-ci.com to travis-ci.org

The moment we open source this repository, build will stop happening on the paid travis infrastructure and start happening on the open source travis infrastructure. This happened to us with osquery and, because travis-ci.org was SO much slower at the time than travis-ci.com, we had to disable travis and build our own CI solution. Lets make sure we won't get burned by this and figure out a plan.

User Management and Login Tasks

Tasks

• Login Page Component
• Admin Page Component
• Create User Component
• Disable User Component
• Reset Password Component
• Promote to Admin Component
• Demote to User Component

Add a sqlx datastore implementation

Following the patterns laid out here, lets move to sqlx isolated behind a subpackage.

Docker Fake Host Launcher

As a developer, I want the ability to launch N osqueryd (where N 1 - 30) on my local machine so that I can simulate various development conditions (like for enrollment, or various scales of queries, etc)

Create an email subpackage

Create subpackage which abstractly allows for

• sending emails in the application
• defining HTML templates and sending HTML email
• DKIM / email security settings, if available
• verifying an email address for a user
• sending emails like "click here to verify that you wanted to do X"

Status & Run Log Management

As a ops or support person I need to know the status of each of my endpoints and their connectivity to Kolide for troubleshooting purposes.

As a user I want to be able to be able to aggregate my query results so they are consumable in an aggregation/log index product (ex: Splunk) so that I can perform useful lookups using those tools.

• Support collecting and aggregating status Log (separate file)
• Support collecting and aggregating result Logs (separate file)
• Add sensible options for disabling/changing locations of these logs

Kolide Inc. deployment pipeline

I'm going through setting up a deployment pipeline like this
dev => ci => run tests => build docker image => push to registry => spin up kubernetes pod

While I'm going through the process, there are things that come up that need to be either fixed, added, documented or improved. This issue is for creating a list of things to track.

In no particular order:

• Create Docker Image
• Add --build flag to kolide-builder which creates a binary version of OSE
• version flag should return the commit sha
• figure out a good way of loading certs into container, creating certs
  k8 can have these preloaded with secrets object
• test that docker image can be fully configured with env variables
• add a /healthz endpoint (or create a sidecar)

going to move tracking this into https://github.com/kolide/kolide-ose/projects/1

Add `kolide check` subcommand

Kolide depends on a few key configurations to even start running. We should add a kolide check command which someone could run with a Kolide configuration to ensure that their settings are sane and that all of the configured infrastructure is available. Maybe kolide check would instruct users to run kolide prepare-db to prepare databases or kolide migrate if we ever get around to that.

Improve Integration Tests

From sprint planning - Mike to improve the tests to get better coverage, depends on #65.

Distributed query should respond to distributed write endpoint with errors

osqueryd should respond to the distributed write endpoint of the remote TLS API with any errors that occur when executing a distributed query.

User controllers return database error when user ID not found

The controllers should probably check whether the error returned from the DB call is a not found error, and take appropriate action.

No bootup output to reflect that kolide binary is serving

When I was first setup this project, I ran the kolide binary without and flags and received a very helpful usage message.

When I ran ./kolide serve I received no output at all, which was disconcerting. As a user I expect at minimum the following information:

1. A notice that server has started successfully
2. The current IP address the server is bound to
3. The current port the server is bound to
4. The current version of kolide
5. The escape sequence needed to kill the server

Here is an example of something that I think would be nice:

=> Kolide 0.0.1 application starting on https://localhost:8080
=> Run `kolide serve -h` for more startup options
=> Listening on tcp://localhost:8080
Use Ctrl-C to stop

Redirect http to https

As a user setting up the app for the first time, I ran the server and thenI typed "localhost:8080" into Chrome's address bar. When I did that, instead of showing the page, my browser downloaded a garbage file and I got the following error in the logs:

http: TLS handshake error from [::1]:59012: tls: first record does not look like a TLS handshake

This happened because my browser began negotiating an http connection, which the kolide binary does not support (which causes some weird download to occur). We should support a standard http to https redirect, even on non-standard port.

Steps to reproduce

1. Build Kolode
2. Run ./kolide serve
3. In an ingonito chrome type "http://localhost:8080"
   Expected Behavior: Kolide redirects the user to "https://localhost:8080"
   Actual Behavior: "Chrome auto-downloads a junk file and an error appears on the Kolide server logs"

Become more consistent and flexible with logging

Right now we use logrus for application logs and these can only be logged to stdout. We also use lumberjack for osquery result and status logs and these can only be logged to rotating files.

Logrus has an internesting hooks API and go-kit also has an interesting logging library.

It would be nice if all logs (osquery result logs, osquery status logs, kolide application logs, kolide alert logs, etc) could be picked and mixed and matched. For example, I might want to send one set of logs to ES, one set of logs to Kafka, one set of logs to a local rotating file and one set of logs to a syslog server. It would be nice if the kolide binary supported this level of configuration natively. Ideally, this would all happen with the same library as well.

Add a boltdb datastore implementation

Let's add a BoltDB datastore implementation for the use-case of running small to medium scale kolide instances without external database dependencies.

Finish the mock datastore implementation

Right now the in-memory mock datastore implementation is incomplete. It's very useful for testing, so let's finish developing it.

Create deployment documentation for different platforms

The OSE project should have good docs for getting started on different platforms.

• heroku
  Heroku should just have a deploy button right away, with some additional instructions.
• GKE- Kubernetes
  Should be similar to the tutorial for setting up wordpress
  https://cloud.google.com/container-engine/docs/tutorials/persistent-disk/
• Mac:
  This one should be docker compose up with an optional darwin binary for people to try on their system. The darwin binary should just be for demo purposes and run with an inmem sql version.
• Windows: I have no clue yet, but probably important to figure out. CI should at least build an exe...
• ESXI: I can build a template.

Expose a way to skip integration tests

http://stackoverflow.com/a/28007631/3204535

Back-End User & Session Management

Master task for: #10, #11, #12, #13, #14, #15, #16

Clean up salt generation function

Use KolideError for error detail in authorization code

#50 enables more detailed error messages to be provided from Kolide code. Getting these extra details from the authorization framework is a worthy refactor.

Accept ENV vars for Kolide config

As a infrastructure person running Kolide in a context like docker or other containers-based deployment solutions or PaSSes, I want the ability to specify my config using ENV vars vs a config file as it makes it much easier to use those solutions in the way their creators intended.

See: http://12factor.net/config & https://docs.docker.com/engine/reference/commandline/run/ (specifically env files)

meta_test.go should print compile/go vet errors

Currently there is just a "failed" message with no other information, and you must run the command on the appropriate package (or all packages if you don't know) to see the error. It would be nice if one go test was sufficient.

Isolate configuration parsing

Everything else should waterfall in via injection on instantiated methods.

Add modern security flags to session cookies

https://www.owasp.org/index.php/Session_Management_Cheat_Sheet#Cookies

Get Email Pipeline Working

Build out osquery config endpoint

This endpoint should be able to do the following:

• Given a config request from an osqueryd, return a list of label queries for that host to execute
• Save results of label queries to the DB
• Generate a config from the scheduled queries attached to the labels for the host

Work through fresh setup flow from a user management perspective

Who will create the first user if there are no admins created yet?

Configuration endpoint for osqueryd clients

Hard dependency for user configuration screen but can start by pushing a basic config.

Write integration tests for all user/account management HTTP endpoints

Support DKIM mail signing

Allow DKIM keys to be specified so that messages can be signed via https://github.com/toorop/go-dkim

Integrate go cover into test executions

Go has a tool which will track how much of a codebase is tested. Perhaps we can use this to ensure we don't start adding untested code?

https://godoc.org/golang.org/x/tools/cmd/cover

Asset Pipeline Setup

As a developer I want to be able to have a solid asset pipeline so that I can modify front-end code and have it correctly compile and serve on the fly.

Add documentation for how to use Godeps

How do I use Godeps? When am I supposed to run it? What if I add a new dependency? What if I've already installed dependencies via go get? etc etc etc

Implement a password reset flow on-top of email pipeline

Basic Enrollment API

As a user I need the ability to enroll my osqueryd agent with Kolide installation.

Tasks

• Figure out how Kolide and the Kolide w/ enroll secret works
• DB management with enroll keys
• Authorizing osqueryds and handling errors with invalid node keys
• Kolide Server Endpoint /api/v1/osquery/enroll See: https://github.com/kolide/kolide/blob/master/osquery.go#L87

Plan "directory structure" of the app

This is a placeholder issue to serve as an area where we discuss a good layout or structure for the various components of the app.

Outcome of this will be a decision and possibly a PR that makes any recommended changes to the app.

Harden the session management APIs

Ensure that the session management APIs are resilient and handle various error cases, like cookies disabled, etc.

consider using a docker image for CI tests

Ran into a few issues with the CI build today because the circle-ci environment is somewhat different from what I was expecting:

• used version 1.5.3 by default, even if the build machine was set to ubuntu 14:04, the go version
  would be 1.6.2. 1.7.0 is scheduled to be released on monday
• cached with master

switching to a docker build image would give us better control over the build environment and avoid issues with differences between CI, production and developer environments.

For reference, see the prometheus setup and corresponding docker image.

• create a new repo for builder
• update circle.yml to use new image
• add read-only bot account for docker hub that has access to pull images
• add builds for multiple go versions
• add linter to image
• add documentation

Expose application metrics in a /metrics endpoint

We should expose metrics about the state of the application at an endpoint that can be picked up by a monitoring solution.

Go provides expvar for doing this in the stdlib, but I'd like to propose exposing metrics in a Prometheus format.

Example metrics that the app would expose:

• request latency
• error/request counters

Having metrics from the start would allow not only to monitor for issues but to also see how new features and code refactors affect the performance of the app.

Validation errors should return helpful JSON

Currently when binding a struct from JSON fails, a 500 error is returned, but there is an empty JSON response. We need a way to get useful error messages to clients (osqueryd, frontend, etc.).

A good idea is available in this issue: gin-gonic/gin#430 (comment)

Add drone.io badge to the README