Coder Social home page Coder Social logo

netflow-collector's Introduction

Netflow exporter

Have you ever wondered where is your internet traffic going?

traffic destination by country

traffic source by AS

How it works

Simply put, it uses netflow protocol, specifically it uses V5 version for simplicity. In order for your setup to work, you will either need nfdump or dedicated hardware such as Mikrotik RB941 Flows are then fed into collector that aggregates them as metrics. Geolocation info is gathered from Maxmind GeoIP Lite database. Necessary files can be obtained on RHEL OS (or similar) with sudo dnf install geolite2-country geolite2-asn

Example configuration for routerboard

/ip traffic-flow
set enabled=yes interfaces=wan,bridge1
/ip traffic-flow target
add dst-address=192.168.0.10 port=30000 version=5

Note 192.168.0.10 is address of machine where exporter is running

Configurable metrics

Flows are aggregated into metrics in fully configurable manner.

Example metric

  - name: traffic_detail
    description: Traffic detail
    labels:
      - name: sampler
        value: sampler
        converter: ipv4
      - name: protocol
        value: proto_name
        converter: str
      - name: source_country
        value: source_country
        converter: str
      - name: destination_asn_org
        value: destination_asn_org
        converter: str

Full example can be found here

Supported enrichers

  • maxmind_country

    MaxMind GeoLite country data are used to add source and destination country (if applicable)

    • used attributes: source_ip, destination_ip
    • added attributes: source_country, destination_country
    • configuration options:
  • maxmind_asn

    MaxMind GeoLite country data are used to add source and destination autonomous system (if applicable)

    • used attributes: source_ip, destination_ip
    • added attributes: source_asn_org, destination_asn_org
    • configuration options:
  • interface_mapper

  • protocol_name

  • reverse_dns

    Does a reverse DNS lookup for IP and selects the first entry returned. unknown set if none found and ip_as_unknown is not enabled. Results (including missing) cached per cache_duration.

    • used attributes: source_ip, destination_ip
    • added attributes: source_dns, destination_dns
    • configuration options:
      • cache_duration - how long to cache result for. Default 1h.
      • tail_pihole - useful if running on a server which is running pihole. If set, read from pihole -t to populate DNS cache. This cache will be used instead of a reverse DNS lookup if available. By tailing the PiHole log, we can see the original query before CNAME redirection and thus give a more interesting answer. Ensure that additional logging entries are enabled, e.g. echo log-queries=extra | sudo tee /etc/dnsmasq.d/42-add-query-ids.conf ; pihole restartdns
      • lookup_local - enable looking up local addresses. Default false.
      • lookup_remote - enable looking up remote addresses. Default true.
      • ip_as_unknown - if a reverse record is not available, uses the IP address itself rather than "unknown" string. Default false.

    e.g. add reverse_dns under enrich: and the following under labels::

    - name: source_ip
      value: source_ip
      converter: ipv4
    - name: destination_ip
      value: destination_ip
      converter: ipv4
    - name: source_dns
      value: source_dns
      converter: str
    - name: destination_dns
      value: destination_dns
      converter: str

    e.g. gather in/out statistics for hosts on the local network:

    pipline:
    ...
      enrich:
        - reverse_dns
      metrics:
        ...
        items:
          - name: traffic_in
            description: Traffic in per host
            labels:
              - name: host
                value: destination_dns
                converter: str
          - name: traffic_out
            description: Traffic out per host
            labels:
              - name: host
                value: source_dns
                converter: str
    
    ...
    extensions:
      reverse_dns:
        lookup_local: true
        lookup_remote: false
        ip_as_unknown: true

Run using podman/docker

podman run -ti -p 30000:30000/udp -p 30001:30001/tcp -u 1000 \
  -v $(pwd)/config.yaml:/config.yaml:ro \
  -v /usr/share/GeoIP:/usr/share/GeoIP:ro \
  ghcr.io/rkosegi/netflow-collector:latest

netflow-collector's People

Contributors

dependabot[bot] avatar rkosegi avatar aeijdenberg avatar druggeri avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.