ops-center / searchlight

🔦 Alerts for Kubernetes

Searchlight

Searchlight by AppsCode is a Kubernetes operator for Icinga. If you are running production workloads in Kubernetes, you probably want to be alerted when things go wrong. Icinga periodically runs various checks on a Kubernetes cluster and sends notifications if detects an issue. It also nicely supplements whitebox monitoring tools like, Prometheus with blackbox monitoring can catch problems that are otherwise invisible, and also serves as a fallback in case internal systems completely fail. Searchlight is a CRD controller for Kubernetes built around Icinga to address these issues. Searchlight can do the following things for you:

• Periodically run various checks on a Kubernetes cluster and its nodes or pods.
• Includes a suite of check commands written specifically for Kubernetes.
• Searchlight can send notifications via Email, SMS or Chat.
• Supplements the whitebox monitoring tools like Prometheus.

Supported Versions

Please pick a version of Searchlight that matches your Kubernetes installation.

Searchlight Version	Docs	Kubernetes Version
8.0.0 (uses CRD)	User Guide	1.9.x+ (test/qa clusters)
7.0.0 (uses CRD)	User Guide	1.8.x
5.1.1 (uses CRD)	User Guide	1.7.x+
3.0.1 (uses TPR)	User Guide	1.5.x - 1.7.x

Installation

To install Searchlight, please follow the guide here.

Using Searchlight

Want to learn how to use Searchlight? Please start here.

Searchlight API Clients

You can use Searchlight api clients to programmatically access its CRD objects. Here are the supported clients:

• Go: https://github.com/appscode/searchlight
• Java: https://github.com/searchlight-client/java

Contribution guidelines

Want to help improve Searchlight? Please start here.

---

Searchlight binaries collects anonymous usage statistics to help us learn how the software is being used and how we can improve it. To disable stats collection, run the operator with the flag --enable-analytics=false.

---

Acknowledgement

• Many thanks to Icinga project.

Support

We use Slack for public discussions. To chit chat with us or the rest of the community, join us in the AppsCode Slack team channel #searchlight_. To sign up, use our Slack inviter.

If you have found a bug with Searchlight or want to request for new features, please file an issue.

searchlight's Issues

Support preferred api group kinds

In Kube 1.6,

• Deployments changed from extensions/v1beta1 to apps/v1beta1.
• Jobs changes from v1alpha2 -> v1

These needs to be supported.

stashed/stash#50

Fix alert commands

• Remove prometheus_metric from https://github.com/appscode/searchlight/blob/master/api/icinga.go#L39

Automate CA cert generation process

Test updated controller manually

Docs: Hightlight Notifier docs

Notifier docs need to be highlighted in the front readme.

Add analytics

UA-62096468-14

Change User "appscode_user" to "searchlight_receiver"

https://github.com/appscode/searchlight/blob/master/pkg/icinga/common.go#L149

https://github.com/appscode/searchlight/blob/master/hack/docker/icinga/config/icinga2/users.conf#L1

Observability in a Dynamically Scheduled World

https://www.slideshare.net/SnehaInguva/observability-in-a-dynamically-scheduled-world

Use HTTP endpoint for alert ack

Icinga2 feature 'perfdata' doesn't work in 2.4.8

This bug is fixed in 2.4.9.

See bug issue here

Convert tests to Ginkgo

https://github.com/appscode/stash/tree/master/test/e2e

Use webhook notifier for appscode api

Support reverse index

Deployment -> PodAlert, etc.
Node -> NodeAlert

Add tests for check_volume

check_kube_event - allow specifying involved object

https://github.com/kubernetes/kubernetes/search?utf8=%E2%9C%93&q=GetFieldSelector&type=

Switch to CustomResourceDefinitions

https://github.com/kubernetes/community/blob/master/contributors/design-proposals/thirdpartyresources.md

kubernetes/enhancements#95 (comment)

Use unified notifiers.

Avoid writing to docker filesystem during init

I don't see a clean way to avoid this.

New Features

• out-of-pod
• Custom configmaps
• pushgateway

Fix plugin

• Some flag descriptions have copy/paste error.
• pod_status, post-exists have new flags not present in data files. + icinga commands

Use unified notifier

https://github.com/appscode/kubed/blob/master/pkg/cert/cert_watcher.go#L104

Add warning event if missing notifierSecret

Review storage metrics

kubernetes/enhancements#167

Fix RBAC

Example: https://github.com/appscode/stash/blob/master/hack/deploy/stash-with-rbac.yaml

Turn alert.appscode.com/objectType annotations into selectors

We create 3 separate TPR:

• Alert
• ClusterAlert
• NodeAlert

Should support using notifierSecret from another namespace?

Make http endpoint a UAS

This is give us trusted info on the caller who ACK-ed.

Cleanup documentation

• Move the instructions for creating TPR to developer section. Less task for users, less confusion.
• Explain to users that they need to wait after Searchlight pod is deployed before creating alerts.

Secret namespace clarification

For json_path & influx_query, always use alert namespace as secret namespace.

Volume plugin is special because the secret is created when the cluster is provisioned. For hostfacts,
use 2 parameters

• secretName
• secretNamespace

ENV

https://docs.icinga.com/icinga2/latest/doc/module/icinga2/chapter/monitoring-basics#command-environment-variables

Notify cluster admin about soon to be expired certs

https://github.com/appscode/kubed/issues/15

We want to move this as a ClusterAlert here, instead of a Kubed thing. This is already implemented in Kubed.

Generate performance data for custom plugins

Icinga2 nagios plugins usually generate performance data.
This features are missing from AppsCode custom plugins

https://docs.icinga.com/icinga2/latest/doc/module/icinga2/chapter/icinga2-features
16.4. Performance Data

https://docs.icinga.com/icinga2/latest/doc/module/icinga2/chapter/object-types#objecttype-perfdatawriter

Enable feature
https://docs.icinga.com/icinga2/latest/doc/module/icinga2/chapter/addons

Support multiple receivers for each state

To []string

hostalive command is not working

-bash-4.3$ ./check_ping -H pfe27 -w 100.0,20% -c 500.0,60% -p 1 -t 30 -4
CRITICAL - You need more args!!!
Could not open pipe: 
-bash-4.3$

https://dan.langille.org/2013/06/01/ngaios-check_ping-critical-you-need-more-args-could-not-open-pipe/

iputils:
https://monitoring-portal.org/index.php?thread/36650-ping4-check-uses-bin-ping-instead-of-check-ping/
net.ipv4.ping_group_range=999 59999

gliderlabs/docker-alpine#253 (comment)

https://support.nagios.com/forum/viewtopic.php?f=35&t=44004

Time Series Anomaly Detection Algorithms

https://blog.statsbot.co/time-series-anomaly-detection-algorithms-1cef5519aef2

Fix secret namespace

Design page for slack bot_token info

Handle non-responsive icinga

There is one problem with this. If the sidecar icinga has not started yet, this results in crashed operator container and eventual crashloop. We need a way to block for this, if we open connection in the beginning. Also, we need to be able to reconnect incase icinga pod crashed and restarted.

This resulted in our Chart failing tests. helm/charts#953 (comment)

Support field selectors

Install as critical addon

https://kubernetes.io/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/

check_component_status - allow specifying component

Retry api calls to Icinga

• On timeout

Sometimes e2e tests fail due to connection issue.

Fix hash_function php.ini

user.warn: Jul 24 17:06:34 icingaweb2[59]: Could not set php.ini setting hash_function = true. This might affect your sessions behaviour.
user.warn: Jul 24 17:06:34 icingaweb2[59]: Could not set php.ini setting hash_bits_per_character = 5. This might affect your sessions behaviour.

Cleanup Status of Alerts

Current fields does not make sense.

AlertStatus should keep track of last applied time

Add slack support as notifier

We can now support slack as notifier.

Supported ENV:

1. SLACK_AUTH_TOKEN
2. SLACK_TO (Channel)

Replace hostfacts with official metrics

kubernetes/kube-state-metrics#179

prometheus-operator/prometheus-operator#485

Add Health check for icinga container

https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#define-a-liveness-http-request

Uniform name for check_**_volume

Upgrade Icinga and IcingaWeb2

• Reduce image size
• Check api compatibility
• SSL seg fault issue
• Perf data