kolide / fleet Goto Github PK

Ensure that the session management APIs are resilient and handle various error cases, like cookies disabled, etc.

As a user I need the ability to enroll my osqueryd agent with Kolide installation.

Tasks

• Figure out how Kolide and the Kolide w/ enroll secret works
• DB management with enroll keys
• Authorizing osqueryds and handling errors with invalid node keys
• Kolide Server Endpoint /api/v1/osquery/enroll See: https://github.com/kolide/kolide/blob/master/osquery.go#L87

As a ops or support person I need to know the status of each of my endpoints and their connectivity to Kolide for troubleshooting purposes.

As a user I want to be able to be able to aggregate my query results so they are consumable in an aggregation/log index product (ex: Splunk) so that I can perform useful lookups using those tools.

• Support collecting and aggregating status Log (separate file)
• Support collecting and aggregating result Logs (separate file)
• Add sensible options for disabling/changing locations of these logs

Tasks

• Login Page Component
• Admin Page Component
• Create User Component
• Disable User Component
• Reset Password Component
• Promote to Admin Component
• Demote to User Component

So that users can serve the HTTP server on 80 and the HTTPS on 443 and not have to configure vhosts at all. Ideally the HTTP site should just redirect.

We should expose metrics about the state of the application at an endpoint that can be picked up by a monitoring solution.

Go provides expvar for doing this in the stdlib, but I'd like to propose exposing metrics in a Prometheus format.

Example metrics that the app would expose:

• request latency
• error/request counters

Having metrics from the start would allow not only to monitor for issues but to also see how new features and code refactors affect the performance of the app.

Kolide depends on a few key configurations to even start running. We should add a kolide check command which someone could run with a Kolide configuration to ensure that their settings are sane and that all of the configured infrastructure is available. Maybe kolide check would instruct users to run kolide prepare-db to prepare databases or kolide migrate if we ever get around to that.

https://www.owasp.org/index.php/Session_Management_Cheat_Sheet#Cookies

Ran into a few issues with the CI build today because the circle-ci environment is somewhat different from what I was expecting:

• used version 1.5.3 by default, even if the build machine was set to ubuntu 14:04, the go version
  would be 1.6.2. 1.7.0 is scheduled to be released on monday
• cached with master

switching to a docker build image would give us better control over the build environment and avoid issues with differences between CI, production and developer environments.

For reference, see the prometheus setup and corresponding docker image.

• create a new repo for builder
• update circle.yml to use new image
• add read-only bot account for docker hub that has access to pull images
• add builds for multiple go versions
• add linter to image
• add documentation

The OSE project should have good docs for getting started on different platforms.

• heroku
  Heroku should just have a deploy button right away, with some additional instructions.
• GKE- Kubernetes
  Should be similar to the tutorial for setting up wordpress
  https://cloud.google.com/container-engine/docs/tutorials/persistent-disk/
• Mac:
  This one should be docker compose up with an optional darwin binary for people to try on their system. The darwin binary should just be for demo purposes and run with an inmem sql version.
• Windows: I have no clue yet, but probably important to figure out. CI should at least build an exe...
• ESXI: I can build a template.

How do I use Godeps? When am I supposed to run it? What if I add a new dependency? What if I've already installed dependencies via go get? etc etc etc

Hard dependency for user configuration screen but can start by pushing a basic config.

Go has a tool which will track how much of a codebase is tested. Perhaps we can use this to ensure we don't start adding untested code?

https://godoc.org/golang.org/x/tools/cmd/cover

Right now, sessions are stored on the client in a cookie. Change this to use MySQL instead. One day, when this is too slow, we'll probably have to add memcached/redis here.

The moment we open source this repository, build will stop happening on the paid travis infrastructure and start happening on the open source travis infrastructure. This happened to us with osquery and, because travis-ci.org was SO much slower at the time than travis-ci.com, we had to disable travis and build our own CI solution. Lets make sure we won't get burned by this and figure out a plan.

When I was first setup this project, I ran the kolide binary without and flags and received a very helpful usage message.

When I ran ./kolide serve I received no output at all, which was disconcerting. As a user I expect at minimum the following information:

1. A notice that server has started successfully
2. The current IP address the server is bound to
3. The current port the server is bound to
4. The current version of kolide
5. The escape sequence needed to kill the server

Here is an example of something that I think would be nice:

=> Kolide 0.0.1 application starting on https://localhost:8080
=> Run `kolide serve -h` for more startup options
=> Listening on tcp://localhost:8080
Use Ctrl-C to stop

As a developer I want to be able to have a solid asset pipeline so that I can modify front-end code and have it correctly compile and serve on the fly.

osqueryd should respond to the distributed write endpoint of the remote TLS API with any errors that occur when executing a distributed query.

Everything else should waterfall in via injection on instantiated methods.

The controllers should probably check whether the error returned from the DB call is a not found error, and take appropriate action.

Following the patterns laid out here, lets move to sqlx isolated behind a subpackage.

http://stackoverflow.com/a/28007631/3204535

Create subpackage which abstractly allows for

• sending emails in the application
• defining HTML templates and sending HTML email
• DKIM / email security settings, if available
• verifying an email address for a user
• sending emails like "click here to verify that you wanted to do X"

As a infrastructure person running Kolide in a context like docker or other containers-based deployment solutions or PaSSes, I want the ability to specify my config using ENV vars vs a config file as it makes it much easier to use those solutions in the way their creators intended.

See: http://12factor.net/config & https://docs.docker.com/engine/reference/commandline/run/ (specifically env files)

Currently when binding a struct from JSON fails, a 500 error is returned, but there is an empty JSON response. We need a way to get useful error messages to clients (osqueryd, frontend, etc.).

A good idea is available in this issue: gin-gonic/gin#430 (comment)

Currently there is just a "failed" message with no other information, and you must run the command on the appropriate package (or all packages if you don't know) to see the error. It would be nice if one go test was sufficient.

This is a placeholder issue to serve as an area where we discuss a good layout or structure for the various components of the app.

Outcome of this will be a decision and possibly a PR that makes any recommended changes to the app.

Allow DKIM keys to be specified so that messages can be signed via https://github.com/toorop/go-dkim

Per @groob's suggestion.

https://golang.org/pkg/net/http/httputil/#DumpRequest

Create a handler that dumps the request to a file. Upload these requests in a zip file to Drive.

#50 enables more detailed error messages to be provided from Kolide code. Getting these extra details from the authorization framework is a worthy refactor.

Right now we use logrus for application logs and these can only be logged to stdout. We also use lumberjack for osquery result and status logs and these can only be logged to rotating files.

Logrus has an internesting hooks API and go-kit also has an interesting logging library.

It would be nice if all logs (osquery result logs, osquery status logs, kolide application logs, kolide alert logs, etc) could be picked and mixed and matched. For example, I might want to send one set of logs to ES, one set of logs to Kafka, one set of logs to a local rotating file and one set of logs to a syslog server. It would be nice if the kolide binary supported this level of configuration natively. Ideally, this would all happen with the same library as well.

Let's add a BoltDB datastore implementation for the use-case of running small to medium scale kolide instances without external database dependencies.

As a developer, I want the ability to launch N osqueryd (where N 1 - 30) on my local machine so that I can simulate various development conditions (like for enrollment, or various scales of queries, etc)

This endpoint should be able to do the following:

• Given a config request from an osqueryd, return a list of label queries for that host to execute
• Save results of label queries to the DB
• Generate a config from the scheduled queries attached to the labels for the host

Right now the in-memory mock datastore implementation is incomplete. It's very useful for testing, so let's finish developing it.

Who will create the first user if there are no admins created yet?

As a developer I want the ability to be able to bootstrap the app and deps instantly with something like docker.

Because LoadConfig assigns to the config objects, the default values are eliminated. We should either fix this so it is not the case, or remove the default values entirely.

https://github.com/kolide/kolide-ose/blob/master/config/config.go#L86

I'm going through setting up a deployment pipeline like this
dev => ci => run tests => build docker image => push to registry => spin up kubernetes pod

While I'm going through the process, there are things that come up that need to be either fixed, added, documented or improved. This issue is for creating a list of things to track.

In no particular order:

• Create Docker Image
• Add --build flag to kolide-builder which creates a binary version of OSE
• version flag should return the commit sha
• figure out a good way of loading certs into container, creating certs
  k8 can have these preloaded with secrets object
• test that docker image can be fully configured with env variables
• add a /healthz endpoint (or create a sidecar)

going to move tracking this into https://github.com/kolide/kolide-ose/projects/1

Master task for: #10, #11, #12, #13, #14, #15, #16

Right now, most dependencies pull from the master of each repository when they're imported / go got. We should look into the Go dependency management ecosystem and use a tool, like godep or something similar, to manage our dependencies reliably and offer 100% repeatable builds.

Once sessions are backed in MySQL, we will need to expose HTTP APIs so that the application can have session management functionality.

For example:

• Delete all sessions for a user (DELETE /api/v1/kolide/user/session)
• Delete a single session for a user (DELETE /api/v1/kolide/user/session/{id})

As a user setting up the app for the first time, I ran the server and thenI typed "localhost:8080" into Chrome's address bar. When I did that, instead of showing the page, my browser downloaded a garbage file and I got the following error in the logs:

http: TLS handshake error from [::1]:59012: tls: first record does not look like a TLS handshake

This happened because my browser began negotiating an http connection, which the kolide binary does not support (which causes some weird download to occur). We should support a standard http to https redirect, even on non-standard port.

Steps to reproduce

1. Build Kolode
2. Run ./kolide serve
3. In an ingonito chrome type "http://localhost:8080"
   Expected Behavior: Kolide redirects the user to "https://localhost:8080"
   Actual Behavior: "Chrome auto-downloads a junk file and an error appears on the Kolide server logs"

From sprint planning - Mike to improve the tests to get better coverage, depends on #65.