kolide / kolide-quickstart

[DEPRECATED] A quickstart demo for Kolide tools

This repository was created to demo Kolide Fleet before it was available as an open source tool. Because Fleet is now open source, please refer to the Fleet docs for information on getting started with Fleet.

---

Kolide Quickstart Demo

The scripts and config files in this repository will enable you to quickly get a demo Kolide Fleet installation up and running. If you would like to try Fleet without setting up a production testing environment, this demo is for you. For guidance on installing a production Fleet environment, please see the infrastructure documentation.

We're available to help with this script, or deploying Kolide in your environment. You can contact us by email at [email protected] or by joining #kolide on the osquery slack team.

If you would like to contribute to the script, you can open an Issue or Pull Request.

Dependencies

• Bash compatible shell with standard unix commands
• Git, or a way to download and unzip these scripts
• Docker and Docker Compose (installed by default with Docker on Mac and Windows)

All other necessary dependencies will be installed via Docker by the scripts in this repository.

Quickest Setup

git clone https://github.com/kolide/kolide-quickstart.git
cd kolide-quickstart
./demo.sh up simple
./demo.sh add_hosts 10 # Will add 10 containerized hosts to your installation

At this point you can navigate to https://localhost:8412 (or the IP/DNS name of the server running Kolide) and log in with the credentials supplied in the output of the above script.

More advanced setup is explained below.

Usage

Start Fleet (and Dependencies)

git clone https://github.com/kolide/kolide-quickstart.git # or download and unzip https://github.com/kolide/kolide-quickstart/archive/master.zip
cd kolide-quickstart
./demo.sh up

On the first run, a self-signed TLS certificate will be generated to be used with your demo instance of Fleet. Please enter a CN for this certificate that osquery hosts will be able to use to connect. If you already have a trusted TLS certificate, you can provide it in this step.

./demo.sh up /path/to/server.key /path/to/server.crt

When startup completes successfully, a message will be printed with a link to the Kolide instance. At this URL you will be walked through final setup.

Stop Fleet (and Dependencies)

./demo.sh down

This will terminate the containers running Fleet and its dependencies, but data will persist across restarts. Use ./demo.sh up to start again.

Reset Fleet Instance

./demo.sh reset

This will terminate the containers, and remove the MySQL data and generated TLS certificate. Use ./demo.sh up to start again from scratch.

Enroll Endpoints

This Fleet demo comes with various methods for adding hosts. It can easily be tested with containerized fake hosts in Docker, but testing with real hosts will help you understand the true value Fleet can bring to your infrastructure.

Add Docker-based Hosts

These Docker-based hosts can be added immediately with no additional setup. Because the containers are all built from the same image, they will return similar results for most queries. To enroll docker-based hosts:

./demo.sh add_hosts <number of hosts>

You can run the command multiple times to scale the number of enrolled osqueryd containers up or down.

Add macOS Hosts

This demo can generate an installer (.pkg) that will configure a macOS osquery installation to work with the Fleet server. To build this package:

./demo.sh enroll mac

The generated installer will be located in out/kolide-enroll-1.0.0.pkg.

Now, ensure that osquery is installed on the target host, and run the generated installer package to configure the osquery installation.

Note: If you want to enroll the macOS host that this demo is running on you may have to edit the /etc/hosts file as specified in the output when generating the installer.

Add Linux Hosts

Soon we will introduce package generation for configuring Linux osquery hosts to operate with this demo.

Testing with Email (Optional)

Email setup is not required to demo Fleet. For those who would like to demo Fleet with a simulated email server, ./demo.sh up starts a Mailhog container that facilitates this. In a production Fleet deployment, you would use your normal SMTP server.

Set Up Email

To configure Fleet with this demo email server:

1. In Fleet, navigate to Admin -> App Settings (/admin/settings).
2. Make up a Sender Address (eg. [email protected]).
3. Enter SMTP server mailhog and port 1025.
4. Set Authentication Type to None.
5. Click "Update Settings"

When completed, the configuration should look like this:

Viewing Emails

Mailhog starts a UI available at port 8025 on your docker host (http://localhost:8025 if you are on the docker host) for viewing the emails "sent" through its SMTP server. If email is properly configured, you should see a test message from Fleet in this UI.

kolide-quickstart's Issues

Error: Kolide failed to start up. Exiting.

$ docker info
Containers: 12
 Running: 3
 Paused: 0
 Stopped: 9
Images: 5
Server Version: 18.03.1-ce
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-862.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.5 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.696GiB
Name: kolide
ID: 5ZPS:DWAF:HTG6:EZUT:2WLZ:PVRT:NCEV:LPVU:WSSF:HTUS:UVWE:3IPH
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

$ sudo ./demo.sh up simple
Using default tag: latest
latest: Pulling from kolide/fleet
Digest: sha256:2c0d0b92f364a665702a62351bc1fa880637fe58d8ebbdc47abc8598952f7243
Status: Image is up to date for kolide/fleet:latest
Using default tag: latest
latest: Pulling from kolide/openssl
Digest: sha256:9d86215afdd5d76faae9de32b7be862a64542ff7ddcc753c43b34118f42aead8
Status: Image is up to date for kolide/openssl:latest
quickstart_mailhog_1 is up-to-date
quickstart_redis_1 is up-to-date
quickstart_mysql_1 is up-to-date
Starting quickstart_fleet_1 ... done
Waiting for MySQL to accept connections...
Waiting for Fleet server to accept connections....</snip> ..Error: Kolide failed to start up. Exiting.

Problems running quickstart on Ubuntu 16.04

~/kolide-quickstart$ sudo ./demo.sh up simple
Using default tag: latest
latest: Pulling from kolide/fleet
88286f41530e: Pull complete
a9e9fd40d73c: Pull complete
8c10e85f0f5c: Pull complete
Digest: sha256:7261b45141c58e04f749f03b853e237ad705f240472a2a13e33ce80a24114450
Status: Downloaded newer image for kolide/fleet:latest
Using default tag: latest
latest: Pulling from kolide/openssl
b7f33cc0b48e: Pull complete
d952d1e7feba: Pull complete
Digest: sha256:9d86215afdd5d76faae9de32b7be862a64542ff7ddcc753c43b34118f42aead8
Status: Downloaded newer image for kolide/openssl:latest
Generating RSA private key, 2048 bit long modulus
..................................................+++
...............+++
e is 65537 (0x10001)
writing RSA key
Signature ok
subject=/CN=kolide
Getting Private key
Creating network "kolidequickstart_default" with the default driver
Pulling redis (redis:3.2.4)...
3.2.4: Pulling from library/redis
43c265008fae: Pull complete
2738f760012a: Pull complete
a3b2771d56b8: Pull complete
5d98f21a4432: Pull complete
eebb159250d6: Pull complete
723650aaa26f: Pull complete
a7712c26a3f2: Pull complete
Digest: sha256:5af2446f774075f2b5bc14738368dc111c357c2a11dd304a90b098e27ff1a50e
Status: Downloaded newer image for redis:3.2.4
Pulling mailhog (mailhog/mailhog:latest)...
latest: Pulling from mailhog/mailhog
709515475419: Pull complete
6f91489e9e5b: Pull complete
6500075800d5: Pull complete
c3d9e42bac5e: Pull complete
Digest: sha256:5be1ae7cf894b58fffbe48f84a0541a3595e53124d460d3fb71b3450a8d11189
Status: Downloaded newer image for mailhog/mailhog:latest
Pulling mysql (mysql:5.7)...
5.7: Pulling from library/mysql
2a72cbf407d6: Pull complete
38680a9b47a8: Pull complete
4c732aa0eb1b: Pull complete
c5317a34eddd: Pull complete
f92be680366c: Pull complete
e8ecd8bec5ab: Pull complete
2a650284a6a8: Pull complete
5b5108d08c6d: Pull complete
beaff1261757: Pull complete
c1a55c6375b5: Pull complete
8181cde51c65: Pull complete
Digest: sha256:691c55aabb3c4e3b89b953dd2f022f7ea845e5443954767d321d5f5fa394e28c
Status: Downloaded newer image for mysql:5.7
Creating kolidequickstart_mailhog_1 ...
Creating kolidequickstart_redis_1   ...
Creating kolidequickstart_mysql_1   ...

ERROR: for kolidequickstart_mailhog_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for kolidequickstart_redis_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for kolidequickstart_mysql_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for redis  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for mailhog  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for mysql  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).
Waiting for MySQL to accept connections...

Then it just sits there until forever.

Please tell me what I am missing, forgetting, not doing and then let me know where in the documentation those missed, forgotten, not done tasks are detailed.

Thanksss

Ubuntu: FAIL 20161118212641_CreateTablePasswordResetRequests.go

Hi,

I'm following install guide provided on your web page, and I'm installing from ppa repo latest available version 1.0.2 of kolide.

When I'm running the prepare db, I get

2017/03/23 10:13:45 FAIL 20161118212641_CreateTablePasswordResetRequests.go (Error 1067: Invalid default value for 'expires_at'), quitting migration.

I see that the same error was already reported on docker install in #25, however I'm not running on docker but on clean ubuntu bare metal install.

generate secure config

We default to some silly settings like toor for the mysql password.

Instead, we should have demo.sh up initialize some secure configurations and store these in a secrets.yml file for the user, unique for each user, and reuse that file.

fail to load on mac over docker

2017-04-07 17:43:35 ⌚ FNIGI-M-Q01R in ~/dev/kolide-quickstart
± |master ✓| → ./demo.sh up simple #########blablakety####r3flE0A
Using default tag: latest
latest: Pulling from kolide/kolide
709515475419: Pull complete
a3ed95caeb02: Pull complete
4d5debf068cf: Pull complete
c82e57e701cd: Pull complete
Digest: sha256:c154e71826c4f4f876a1c618ff995232a6b004c0baf8f827106b8d51dde3f266
Status: Downloaded newer image for kolide/kolide:latest
Using default tag: latest
latest: Pulling from kolide/openssl
b7f33cc0b48e: Pull complete
d952d1e7feba: Pull complete
Digest: sha256:9d86215afdd5d76faae9de32b7be862a64542ff7ddcc753c43b34118f42aead8
Status: Downloaded newer image for kolide/openssl:latest
Generating RSA private key, 2048 bit long modulus
.........................................+++
.....................+++
e is 65537 (0x10001)
writing RSA key
Signature ok
subject=/CN=kolide
Getting Private key
Creating network "kolidequickstart_default" with the default driver
Pulling redis (redis:3.2.4)...
3.2.4: Pulling from library/redis
43c265008fae: Pull complete
2738f760012a: Pull complete
a3b2771d56b8: Pull complete
5d98f21a4432: Pull complete
eebb159250d6: Pull complete
723650aaa26f: Pull complete
a7712c26a3f2: Pull complete
Digest: sha256:5af2446f774075f2b5bc14738368dc111c357c2a11dd304a90b098e27ff1a50e
Status: Downloaded newer image for redis:3.2.4
Pulling mailhog (mailhog/mailhog:latest)...
latest: Pulling from mailhog/mailhog
b7f33cc0b48e: Already exists
b17e1dd5b935: Pull complete
fa5b96f79f47: Pull complete
1a3f7f919f7e: Pull complete
Digest: sha256:618b44697cf947cc03decfee05883fd1ae67b277c9e07a5c9f3f3421eacfaf89
Status: Downloaded newer image for mailhog/mailhog:latest
Pulling mysql (mysql:5.7)...
5.7: Pulling from library/mysql
6d827a3ef358: Pull complete
ed0929eb7dfe: Pull complete
03f348dc3b9d: Pull complete
fd337761ca76: Pull complete
7e6cc16d464a: Pull complete
ca3d380bc018: Pull complete
3fe11378d5c0: Pull complete
2b5dfd325645: Pull complete
b54281d17fbe: Pull complete
7eae4db8eea5: Pull complete
76cf68e17b09: Pull complete
Digest: sha256:49b7d6d8d45f8c3300cba056e8cdf36c714d99e0b40f7005b9e6e75e64ecdf7c
Status: Downloaded newer image for mysql:5.7
Creating kolidequickstart_mailhog_1
Creating kolidequickstart_redis_1
Creating kolidequickstart_mysql_1
Creating kolidequickstart_kolide_1
Waiting for MySQL to accept connections...\c

Waiting for Kolide server to accept connections...\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
Error: Kolide failed to start up. Exiting.

Rename project to kolide/fleet-quickstart

There may be unforeseen consequences of this move, so we should thoroughly test and rename it when we feel confident.

demo.sh error and Running the Kolide server error

1. I had an error when running quickstart script on both CentOS and Ubuntu.

[root@localhost kolide-quickstart]# ./demo.sh up simple [license string]
Using default tag: latest
latest: Pulling from kolide/kolide
Digest: sha256:9a802751690fe3a93f7c11d084128fa631cba964d22c772ff58b153d01034e91
Status: Image is up to date for kolide/kolide:latest
Using default tag: latest
latest: Pulling from kolide/openssl
Digest: sha256:9d86215afdd5d76faae9de32b7be862a64542ff7ddcc753c43b34118f42aead8
Status: Image is up to date for kolide/openssl:latest
kolidequickstart_redis_1 is up-to-date
Starting kolidequickstart_mysql_1
kolidequickstart_mailhog_1 is up-to-date
Starting kolidequickstart_kolide_1
Waiting for MySQL to accept connections...
.\C
Error: MySQL failed to start up. Exiting.
[root@localhost kolide-quickstart]# 

2. CentOS error when preparing db (CentOS/Ubuntu):

/usr/bin/kolide prepare db \
>     --mysql_address=127.0.0.1:3306 \
>     --mysql_database=kolide \
>     --mysql_username=root \
>     --mysql_password=toor
2017/02/21 15:04:36 FAIL 20161118212641_CreateTablePasswordResetRequests.go (Error 1067: Invalid default value for 'expires_at'), quitting migration.

Errors starting on Ubuntu 16

I'm getting this output when I attempt to start on Ubuntu 16

Starting kolidequickstart_redis_1
Starting kolidequickstart_mailhog_1
Starting kolidequickstart_mysql_1
Starting kolidequickstart_fleet_1
Waiting for MySQL to accept connections......
Waiting for Fleet server to accept connections.....................................................Error: Kolide failed to start up. Exiting.

Any advice on troubleshooting?

Thanks

Image Uploads

This issue exists solely to upload images for use in the README.md

Rename repo to "bootstrap"

IMO this repo is useful for much more than a simple demo. With some work it could become the way a user provisions kolide in their environment. "bootstrap" sounds appropriate imo.

demon.sh install error

It can not be installed via kolide-quickstart script.
What is the problem?

Command

./demon.sh simple up

Error Log

Creating kolidequickstart_redis_1 ...
Creating kolidequickstart_mailhog_1 ...
Creating kolidequickstart_mysql_1 ...
Creating kolidequickstart_redis_1
Creating kolidequickstart_mysql_1
Creating kolidequickstart_mysql_1 ... done
Creating kolidequickstart_fleet_1 ...
Creating kolidequickstart_fleet_1 ... done
Waiting for MySQL to accept connections...

Waiting for Fleet server to accept connections.....................................................Error: Kolide failed to start up. Exiting.

docker ps -a

6fba3efc9c97        /kolide/fleet:latest      "sh -c 'echo '\\n' | /"   About a minute ago   Exited (1) About a minute ago                                      kolidequickstart_fleet_1
5fb4f8838bd1        /mysql:5.7                "docker-entrypoint.sh"    About a minute ago   Up About a minute               3306/tcp                           kolidequickstart_mysql_1
f7e9b6ca1784        /mailhog/mailhog:latest   "MailHog"                 About a minute ago   Up About a minute               1025/tcp, 0.0.0.0:8025->8025/tcp   kolidequickstart_mailhog_1
ad174071632d        /redis:3.2.4              "docker-entrypoint.sh"    About a minute ago   Up About a minute               6379/tcp                           kolidequickstart_redis_1

IP address and mac address info

Hey Guys,

is there a way that we can add info for Windows IP and Mac address on kolide, somehow it is not showing on the hosts page. In any case I missed something in relation how to enable this please let me know.

Initial password missing

On master, commit cc6d35b I tried following the instructions.

It did not create the initial user. I went to https://127.0.0.1:8412 and was given brand new install. I created admin/admin123#

I don't know if this is a bug in the instructions or the demo script.

If a port is already bound, script should abort

I had 8025 bound already and run demo.sh up and got an appropriate error message for the bound port, but the script continued and the webserver did not ultimately bind to the claimed port.

$ ./demo.sh up                                                                                                                                                                                                               3:18PM 02/14/17
Enter CN for self-signed SSL certificate [default 'kolide']:
Unable to find image 'kolide/openssl:latest' locally
latest: Pulling from kolide/openssl
b7f33cc0b48e: Already exists
e7284c3d8862: Pull complete
Digest: sha256:f88c701a7a92b4791626cc44b7aad365d63b34c94f98e4838bb1ba205444cc39
Status: Downloaded newer image for kolide/openssl:latest
Generating RSA private key, 2048 bit long modulus
........................................................................................................................................+++
..............................+++
e is 65537 (0x10001)
writing RSA key
Signature ok
subject=/CN=kolide
Getting Private key
Creating network "kolidequickstart_default" with the default driver
Creating kolidequickstart_mailhog_1
Creating kolidequickstart_redis_1
Creating kolidequickstart_mysql_1

ERROR: for mailhog  Cannot start service mailhog: driver failed programming external connectivity on endpoint kolidequickstart_mailhog_1 (0bf73a6e630c7bd1d4631b52b5def2b858e2d1f01d4f93037307869d180c743d): Bind for 0.0.0.0:8025 failed: port is already allocated
ERROR: Encountered errors while bringing up the project.
Waiting for MySQL to accept connections...........
Kolide server should now be accessible at https://127.0.0.1:8412 or https://kolide:8412.
Note that a self-signed SSL certificate will generate a warning in the browser.
To allow other hosts to enroll, you may want to create a DNS entry mapping kolide to the IP of this host.

Windows support

With docker for windows being a thing, there's no reason we can't support this process on windows.
We should make sure the demo can be spun up by a windows user.

@marpaia want to take this on?

Permission problems when installing quickstart on Ubuntu VM.

I installed the quickstart on a ubuntu vm but had to make the following changes to succeed:

diff --git a/demo.sh b/demo.sh
index 00b351e..69197bb 100755
--- a/demo.sh
+++ b/demo.sh
@@ -238,7 +238,7 @@ function up() {
docker run --rm -v $(pwd):/certs kolide/openssl rsa -in /certs/server.key -out /certs/server.key
docker run --rm -v $(pwd):/certs kolide/openssl req -sha256 -new -key /certs/server.key -out /certs/server.csr -subj "/CN=$CN"
docker run --rm -v $(pwd):/certs kolide/openssl x509 -req -sha256 -days 365 -in /certs/server.csr -signkey /certs/server.key -out /certs/server.crt

-        rm server.csr
+        rm -f server.csr

 else
     CN=$(get_cn)
 fi

AND ALSO:

diff --git a/docker-compose.yml b/docker-compose.yml
index d6018c2..3086762 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -9,7 +9,7 @@ services:
MYSQL_USER: kolide
MYSQL_PASSWORD: kolide
volumes:

-      - ./mysqldata:/tmp
+      - mysqldata:/tmp

 expose:
   - "3306"

@@ -51,3 +51,12 @@ services:
core:
hard: 1000000000
soft: 1000000000

+
+volumes:
+  mysqldata:
+

Note: Using a named volume instead of a relative path gets around permission problems re some hosts and also avoids seeing differences in host file systems on macOS, Windows and Linux.

Does Kolide fleet supports DNS?

Hi, I am thinking of substituting the current hostname with DNS, example from 127.0.0.1:8412 to testing.com:8412. Does Kolide fleet supports such implementation?

Issue creating Mac package: pkgbuild: command not found

Using Quickstart on Rhel 7 in AWS.

When running "./demo enroll mac" I am getting this error:
./demo.sh: line 95: pkgbuild: command not found