The teler from trendingtechnology

teler

Real-time HTTP Intrusion Detection
Contribute · What's new · Report Bug · Request Feature

teler is an real-time intrusion detection and threat alert based on web log that runs in a terminal with resources that we collect and provide by the community. ❤️

Features
Why teler?
Demo
Installation
Usage
- Flags
  - Config
  - Input
  - Concurrency
  - Output
  - JSON Format
  - Remove Caches
Configuration
- Log formats
  - Apache
  - Nginx
  - Nginx Ingress
  - Amazon S3
  - Elastic LB
  - CloudFront
- Threat rules
  - Excludes
  - Whitelists
- Notification
- Metrics
  - Prometheus
Supporting Materials
Contributors
- Resources
Pronunciation
Changes
License

Features

Real-time: Analyze logs and identify suspicious activity in real-time.
Alerting: teler provides alerting when a threat is detected, push notifications include Slack, Telegram and Discord.
Monitoring: We've our own metrics if you want to monitor threats easily, and we use Prometheus for that.
Latest resources: Collections is continuously up-to-date.
Minimal configuration: You can just run it against your log file, write the log format and let teler analyze the log and show you alerts!
Flexible log formats: teler allows any custom log format string! It all depends on how you write the log format in configuration file.
Incremental log processing: Need data persistence rather than buffer stream? teler has the ability to process logs incrementally through the on-disk persistence options.

Why teler?

teler was designed to be a fast, terminal-based threat analyzer. Its core idea is to quickly analyze and hunt threats in real time!

Demo

Here is a preview of teler with conditions of use as:

Buffer-streams	Incremental

Installation

from Binary

The installation is easy. You can download a prebuilt binary from releases page, unpack and run! or run with:

▶ curl -sSfL 'https://ktbs.dev/get-teler.sh' | sh -s -- -b /usr/local/bin

using Docker

Pull the Docker image by running:

▶ docker pull kitabisa/teler

from Source

If you have go1.14+ compiler installed and configured:

▶ GO111MODULE=on go get -u ktbs.dev/teler/cmd/teler

The same command works for update to the latest version.

from GitHub

▶ git clone https://github.com/kitabisa/teler
▶ cd teler
▶ make build
▶ mv ./bin/teler /usr/local/bin

Usage

Simply, teler can be run with:

▶ [buffers] | teler -c /path/to/config/teler.yaml
# or
▶ teler -i /path/to/access.log -c /path/to/config/teler.yaml

If you've built teler with a Docker image:

▶ [buffers] | docker run -i --rm -e TELER_CONFIG=/path/to/config/teler.yaml kitabisa/teler
# or
▶ docker run -i --rm -e TELER_CONFIG=/path/to/config/teler.yaml kitabisa/teler --input /path/to/access.log

Flags

▶ teler -h

This will display help for the tool.

Here are all the switches it supports.

Flag	Description	Examples
-c, --config	teler configuration file	kubectl logs nginx \| teler -c /path/to/config/teler.yaml
-i, --input	Analyze logs from data persistence rather than buffer stream	teler -i /var/log/nginx/access.log
-x, --concurrent	Set the concurrency level to analyze logs (default: 20)	tail -f /var/log/nginx/access.log \| teler -x 50
-o, --output	Save detected threats to file	teler -i /var/log/nginx/access.log -o /tmp/threats.log
--json	Display threats in the terminal as JSON format	teler -i /var/log/nginx/access.log --json
--rm-cache	Remove all cached resources	teler --rm-cache
-v, --version	Show current teler version	teler -v

Config

The -c flag is to specify teler configuration file.

▶ tail -f /var/log/nginx/access.log | teler -c /path/to/config/teler.yaml

This is required, but if you have defined TELER_CONFIG environment you don't need to use this flag, e.g.:

▶ export TELER_CONFIG="/path/to/config/teler.yaml"
▶ tail -f /var/log/nginx/access.log | teler
# or
▶ tail -f /var/log/nginx/access.log | TELER_CONFIG="/path/to/config/teler.yaml" teler

Input

Need log analysis incrementally? This -i flag is useful for that.

▶ teler -i /var/log/nginx/access.log

Concurrency

Concurrency is the number of logs analyzed at the same time. Default value teler provide is 20, you can change it by using -x flag.

▶ teler -i /var/log/nginx/access.log -x 50

Output

You can also save the detected threats into a file with -o flag.

▶ teler -i /var/log/nginx/access.log -o threats.log

JSON Format

If you want to display the detected threats as JSON format, switch it with --json flag.

▶ teler -i /var/log/nginx/access.log --json

Please note this will also apply if you save it to a file with -o flag.

Remove Caches

It will removes all stored resources in the user-level cache directory, see cache.

▶ teler --rm-cache

Configuration

teler requires a minimum of configuration to process and/or log analysis, and execute threats and/or alerts. See teler.example.yaml for an example.

Log Formats

Because we use gonx package to parse the log, you can write any log format. As an example:

Apache

log_format: |
  $remote_addr - $remote_user [$time_local] "$request_method $request_uri $request_protocol" $status $body_bytes_sent "$http_referer" "$http_user_agent"

Nginx

log_format: |
  $remote_addr $remote_user - [$time_local] "$request_method $request_uri $request_protocol" 
  $status $body_bytes_sent "$http_referer" "$http_user_agent"

Nginx Ingress

log_format: |
  $remote_addr - [$remote_addr] $remote_user - [$time_local] 
  "$request_method $request_uri $request_protocol" $status $body_bytes_sent 
  "$http_referer" "$http_user_agent" $request_length $request_time 
  [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status $req_id

Amazon S3

log_format: |
  $bucket_owner $bucket [$time_local] $remote_addr $requester $req_id $operationration $key 
  "$request_method $request_uri $request_protocol" $status $error_code $body_bytes_sent - 
  $total_time - "$http_referer" "$http_user_agent" $version_id $host_id 
  $signature_version $cipher_suite $http_auth_type $http_host_header $tls_version

Elastic LB

log_format: |
  $time_local $elb_name $remote_addr $upstream_addr $request_processing_time 
  $upstream_processing_time $response_processing_time $status $upstream_status $body_received_bytes $body_bytes_sent 
  "$request_method $request_uri $request_protocol" "$http_user_agent" $cipher_suite $tls_version

CloudFront

log_format: |
  $date $time $edge_location  $body_bytes_sent  $remote_addr  
  $request_method $http_host_header $requst_uri $status 
  $http_referer $http_user_agent  $request_query  $http_cookie  $edge_type  $req_id 
  $http_host_header $ssl_protocol $body_bytes_sent  $response_processing_time $http_host_forwarded  
  $tls_version  $cipher_suite $edge_result_type $request_protocol $fle_status $fle_encrypted_fields 
  $http_port  $time_first_byte  $edge_detail_result_type  
  $http_content_type  $request_length $request_length_start $request_length_end

Threat rules

Cache

By default, teler will fetch external resources every time you run it, but you can switch external resources to be cached or not.

rules:
  cache: true

If you choose to cache resources, it's stored under user-level cache directory of cross-platform and will be updated every day, see resources.

Excludes

We include resources for predetermined threats, including:

Common Web Attack
CVE
Bad IP Address
Bad Referrer
Bad Crawler
Directory Bruteforce

You can disable any type of threat in the excludes configuration (case-sensitive).

rules:
  threat:
    excludes:
      - "Bad IP Address"

The above format detects threats that are not included as bad IP address, and will not analyze logs/ send alerts for that type.

Whitelists

You can also add whitelists to teler configuration.

rules:
  threat:
    whitelists:
      - "(curl|Go-http-client|okhttp)/*"
      - "^/wp-login\\.php"

It covers the entire HTTP request and processed as regExp, please write it with caution!

Notification

We provide alert notification options:

Slack,
Telegram
Discord

Configure the notification alerts needed on:

notifications:
  slack:
    token: "xoxb-..."
    color: "#ffd21a"
    channel: "G30SPKI"

  telegram:
    token: "123456:ABC-DEF1234...-..."
    chat_id: "-111000"

  discord:
    token: "NkWkawkawkawkawka.X0xo.n-kmZwA8aWAA"
    color: "16312092"
    channel: "700000000000000..."

You can also choose to disable alerts or want to be sent where the alerts are.

alert:
  active: true
  provider: "slack"

Metrics

teler also supports metrics using Prometheus.

Prometheus

You can configure the host, port and endpoint to use Prometheus metrics in the configuration file.

prometheus:
  active: true
  host: "localhost"
  port: 9099
  endpoint: "/metrics"

Here are all the metrics we collected & categorized.

Metric	Description
`teler_threats_count_total`	Total number of detected threats
`teler_cwa`	Get lists of Common Web Attacks
`teler_cve`	Get lists of CVE threats
`teler_badcrawler`	Get lists of Bad Crawler requests
`teler_dir_bruteforce`	Get lists of Directories Bruteforced
`teler_bad_referrer`	Get lists of Bad Referrer requests
`teler_badip_count`	Total number of Bad IP Addresses

Supporting Materials

teler - Protect Your WebApp! Talks were brought to the OWASP Jakarta: Virtual AppSec Indonesia 2020 event.
Tutorial: Cyber Threat Hunting - Useful Threat Hunting Tools (Part One), Semi Yulianto gave a brief explanation and how to use teler in the video.

Contributors

This project exists thanks to all the people who contribute. To learn how to setup a development environment and for contribution guidelines, see CONTRIBUTING.md.

_{Dwi Siswanto} 💻 📖 ⚠️ 🤔	_{ProjectDiscovery} 🔧	_{Anton Egorov} 🔧	_0ktavandi 🤔	_Fik 🎨	_{fairyhunter13} ⚠️	_{Zufar Dhiyaulhaq} 💻
_{Aldin Setiawan} 💻	_{Noah Petherbridge} 🔧	_{Zackky Muhammad} 🚇	_Andy 🔧	_{Aliaksandr Valialkin} 🔧	_{Markus Tenghamn} 🐛	_Rick 🚧 💻
_{Michael BOUVY} 📖	_oelnaggar 📖 🐛