A Docker container for testing and observing Panoptes Functionality.
Panoptes is a Python based network telemetry ecosystem that implements discovery, enrichment and polling. To learn more about Panoptes, check out the project's GitHub repository. This repository provides a workbench version of Panoptes inside a Docker container that is intended to be used for testing concepts and observing code operations.
This does not follow best practices for deployment and shouldn't be used for production deployment. It's a very quick and dirty method to get a Panoptes instance up and running in a minimal amount of time with plenty of redundancy in place to allow enterprise scaling. While this isn't an efficient deployment, it does provide an accurate and operational build of the larger structure.
If you're impatient, and simply want to get this up and running, check out the quickstart guide.
Where to Get Help
Where to file Issues
Issues related to Panoptes_docker
Image Updates
Source of this Description
Maintained by
James Diss: [email protected]
Supported Docker Versions
Tested on 19.03.5
Panoptes is typically run in a distributed system, but it can be difficult to set up a working hive of the various moving parts. Panoptes can be horizontally scaled for metric collection by adding more capacity to run collection, this docker image simply cuts the total number of nodes down to 1. This version also runs from a json file, but we use a command database internally and are working to extend this to CSV files.
The following image provides a visual representation of the services running inside this container.
This container is based on Ubuntu:18.04, and includes Python3.6, open-jdk 8, redis, zookeeper, and influxdb. There are also a few tools that are included for testing purposes, but are not necessary for operation of Panoptes:
- Netcat - Used to check functionality of the services
- tar - Used to view log archives.
- snmpd - Provides an easy target to poll
- nano - Makes it easier to open and modify configuration files and scripts.
- daemontools - Handles output and schedules the various subcomponents of Panoptes. Panoptes typically relies on its granular logging system to redirect to logging servers, but daemontools solves this requirement here.
- Grafana - Included as a visualization tool. See the section on Grafana for details.
The container also installs a specific version of Kafka, because of a hard dependency. This is accomplished by downloading and building the proper version from the ASF archives.
The configuration files for the services included in this image are available to browse under the resources/ directory
structure. Of particular interest will be the resources/daemontools/panoptes_*.run files, as these are the calls to
the various Panoptes components.
First, you will need to have docker installed on your machine.
To build the image, first clone the source code and run docker build
from inside the directory
git clone https://github.com/yahoo/panoptes_docker.git && cd panoptes_docker
docker build . -t panoptes_docker
This container requires quite a lot of processing power since it contains a distributed system that has been packed into a small container. Because of this, you should consider limiting the resources available to the container.
The following command will run panoptes_docker inside a container:
docker run -d \
--sysctl net.core.somaxconn=511 \
--name="panoptes_docker" \
--shm-size=2G \
-p 127.0.0.1:8080:3000/tcp \
panoptes_docker
There is a five minute delay until the first metrics will show up. Due to an issue getting the snmp_community strings into the container's redis, we do need you to run a script on the running container;
docker exec -it panoptes_docker bash
/etc/redis/populate_redis.sh
exitThis effectively writes the environment variables SNMP_SITE and SNMP_COMM_STRING into Redis. These default to local
and public respectively, and can be overriden in the docker run command by supplying -e <VAR> value.
After launching the panoptes_docker container, you should be able to point a browser at http://127.0.0.1:8080 and see the Grafana instance.
Use the auth pair admin/admin to get in, and don't worry about a new password, it won't last longer than the
container. Just skip it.
The datasource and dashboard should be available immediately, however, metrics gathering is delayed by 5 minutes. Your dashboard should look a little like this:
Note that any changes you make to the dashboard won't survive the death of the container. If you want to learn more about Grafana, check out the docs on getting started.
Use docker exec to access the container for examination.
docker exec -it panoptes_docker bash
If you downloaded the prebuilt image, use panoptes/panoptes_docker in this example. This will drop you into the
container at /home/panoptes, and there will be some scripts available that expose the different services running in
the container and split the various log files into meaningful groups.
All of these scripts are copied into the docker container from resources/misc. If you're going to interact with the
python scripts in the container, you need to operate in the proper a virtual environment. To do so, run the following
after you login to the container:
source /home/panoptes_v/bin/activate
conf holds the configuration scripts for Panoptes. logs holds, well, logs and run contains lock files.
show-services will dump the PID and the uptime of each component.
show-influxdb.sh first dumps the data that we're interested in, then tails the logs to show the following calls.
This is generally the first step in troubleshooting because InfluxDB holds the measurement data. Note: there is a 5
minute delay in the collection of stats, and the output can get a little large. To help with this, run:
/usr/bin/influx -database 'Panoptes' -format=csv -execute 'SHOW MEASUREMENTS' | more
show-logs.sh will tail all the relevant logs, but is of limited utility unless you really need a scrolling list of
log updates. Then again, perhaps you run a TV crime lab.
show-panoptes.sh, show-panoptes-agents.sh, show-panoptes-discovery.sh, show-panoptes-polling.sh and
show-panoptes-schedulers.sh show different aspects of the Panoptes system in operation by tailing the relevant logs.
Finally, show-snmpwalk.sh effectively queries localhost for it's SNMP OIDs. This is for checking, but also provides
a glimpse into the world of OIDs.
Once you're done, stop and remove the container before trying to run another.
docker stop panoptes_docker && docker rm panoptes_docker
Panoptes is run as a python module, so the relevant code is under site-packages in the
venv (/home/panoptes_v/lib/site-packages). You can activate this environment from within the docker container.
You can override the default localhost.json by supplying one externally and adding
a flag during the container runtime. The -v join effectively overlays the default localhost.json built into the
container. This example uses a localhost.json at /data/servers/panoptes/conf on the host.
Use -v <source_location>:/home/panoptes/conf/localhost.json as a template.
Both -e variables are optional and default during build time to the values shown.
SNMP_SITE defaults to local, and is one pair of the SNMP Secrets.
SNMP_COMM_STRING defaults to public, but would be the 'community string' for the SNMP_SITE.
docker run -d \
--sysctl net.core.somaxconn=511 \
--name="panoptes_docker" \
--shm-size=2G \
-e SNMP_SITE="local" \
-e SNMP_COMM_STRING="public" \
-v /data/servers/panoptes/conf/localhost.json:/home/panoptes/conf/localhost.json \
-p 127.0.0.1:8080:3000/tcp \
panoptes_docker
During the build of the container, the plugins from each subdirectory of /resource/panoptes/plugins are copied into
the container, so this is a good entry point into the structure.
See the Panoptes Plugin page for details. You'll also find instructions on building plugins at the Plugin Guide
- Panoptes Web Site
- Panoptes Source Code
- Panoptes Documentation
- Panoptes Concepts
- Panoptes Plugin Architecture
- Contributing to Panoptes
- Panoptes Configuration Samples
- Panoptes Plugin Examples
- Open Sourcing Panoptes
Please refer to the contributing.md file for information about how to get involved. We welcome issues, questions, and pull requests. Pull Requests are welcome.
This project is licensed under the terms of the Apache 2.0 open source license. Please refer to LICENSE for the full terms.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent