signalfx / gateway

A proxy to buffer and forward metrics, events, and traces.

⚠️ End of Support (EoS) Notice

The SignalFx Gateway has reached End of Support.

We recommend using our Splunk Distribution of OpenTelemetry Collector going forward, which offers the same capabilities and fully supports the OpenTelemetry standard.

ℹ️  SignalFx was acquired by Splunk in October 2019. See Splunk SignalFx for more information.

Gateway

The SignalFx Gateway lets you aggregate metrics and send them to SignalFx. It is a multilingual datapoint demultiplexer that can accept time series data from the carbon (Graphite), collectd or SignalFx protocols and emit those datapoints to a series of servers using the carbon, collectd or SignalFx protocols. We recommend placing the gateway either on the same server as another existing metrics aggregator or on a central server that is already receiving datapoints, such as Graphite's carbon database.

The SignalFx Gateway also acts as a simple proxy and forwarder for SignalFx's Events Ingest and Trace Ingest APIs.

Usage and Configuration

Please refer to the usage and configuration documentation located here for more information on operating the SignalFx Gateway.

Development

If you want to submit patches for the gateway, make sure your code passes travis_check.sh with exit code 0. For help setting up your development environment, it should be enough to mirror the install steps of .travis.yml. You may need to make sure your GOPATH env variable is set correctly.

Code layout

You only need to read this if you want to develop the gateway or understand the gateway's code.

The gateway is divided into two main components: forwarder and listener. The forwarder and listener are glued together by the demultiplexer.

When a listener receives a datapoint, it converts the datapoint into a basic datapoint type. This core datapoint type is then sent to the multiplexer that will send a pointer to that datapoint to each forwarder.

Sometimes there is a loss of fidelity during transmission if a listener and forwarder don't support the same options. While it's impossible to make something understand an option it does not, we don't want to forget support for this option when we translate a datapoint through the multiplexer. We work around this by sometimes encoding the raw representation of the datapoint into the Datapoint object we forward. For example, points from carbon are not only translated into our core datapoint format, but also support ToCarbonLine which allows us to directly convert the abstract datapoint into what it looked like for carbon, which allows us to forward the point to another carbon database exactly as we received it.

All message passing between forwarders, multiplexer, and listeners happen on golang's built in channel abstraction.

Config file format

See the example config file for an example of how configuration looks. Configuration is a JSON file with two important fields: ListenFrom and ForwardTo.

License

The SignalFx Gateway is released under the Apache 2.0 license. See LICENSE for more details.

Dependencies

Note: this list of dependencies applies to both the SignalFx Gateway and its Smart Gateway variant.

Name	License
github.com/apache/thrift/lib/go/thrift	Apache Software License v2.0
github.com/beorn7/perks/quantile	MIT
github.com/coreos/bbolt	MIT
github.com/coreos/etcd	Apache Software License v2.0
github.com/coreos/go-semver/semver	Apache Software License v2.0
github.com/coreos/go-systemd/journal	Apache Software License v2.0
github.com/coreos/pkg/capnslog	Apache Software License v2.0
github.com/davecgh/go-spew/spew	ISC
github.com/dgrijalva/jwt-go	MIT
github.com/ghodss/yaml	MIT
github.com/go-logfmt/logfmt	MIT
github.com/go-stack/stack	MIT
github.com/gobwas/glob	MIT
github.com/gogo/protobuf	BSD 3-Clause
github.com/golang/groupcache/lru	Apache Software License v2.0
github.com/golang/protobuf	BSD 3-Clause
github.com/golang/snappy	BSD 3-Clause
github.com/google/btree	Apache Software License v2.0
github.com/google/go-cmp	BSD 3-Clause
github.com/gopherjs/gopherjs/js	BSD 2-Clause
github.com/gorilla/context	BSD 3-Clause
github.com/gorilla/mux	BSD 3-Clause
github.com/gorilla/websocket	BSD 2-Clause
github.com/grpc-ecosystem/go-grpc-middleware	Apache Software License v2.0
github.com/grpc-ecosystem/go-grpc-prometheus	Apache Software License v2.0
github.com/grpc-ecosystem/grpc-gateway	BSD 3-Clause
github.com/inconshreveable/mousetrap	Apache Software License v2.0
github.com/jaegertracing/jaeger/thrift-gen/jaeger	Apache Software License v2.0
github.com/jonboulle/clockwork	Apache Software License v2.0
github.com/jtolds/gls	MIT
github.com/konsorten/go-windows-terminal-sequences	MIT
github.com/kr/logfmt	MIT
github.com/mailru/easyjson	MIT
github.com/matttproud/golang_protobuf_extensions/pbutil	Apache Software License v2.0
github.com/mdubbyap/timespan	MIT
github.com/opentracing/opentracing-go	Apache Software License v2.0
github.com/pkg/errors	BSD 2-Clause
github.com/pmezard/go-difflib/difflib	BSD 3-Clause
github.com/prometheus/client_golang/prometheus	Apache Software License v2.0
github.com/prometheus/client_model/go	Apache Software License v2.0
github.com/prometheus/common	Apache Software License v2.0
github.com/prometheus/procfs	Apache Software License v2.0
github.com/prometheus/prometheus/prompb	Apache Software License v2.0
github.com/signalfx/com_signalfx_metrics_protobuf	Apache Software License v2.0
github.com/signalfx/embetcd/embetcd	Apache Software License v2.0
github.com/signalfx/go-distribute	Apache Software License v2.0
github.com/signalfx/go-metrics	BSD 3-Clause
github.com/signalfx/gohistogram	MIT
github.com/signalfx/gohelpers	Apache Software License v2.0
github.com/signalfx/golib	Apache Software License v2.0
github.com/signalfx/ondiskencoding	Apache Software License v2.0
github.com/mdubbyap/tdigest	Apache Software License v2.0
github.com/signalfx/xdgbasedir	Apache Software License v2.0
github.com/sirupsen/logrus	MIT
github.com/smartystreets/assertions	MIT
github.com/smartystreets/goconvey	MIT
github.com/soheilhy/cmux	Apache Software License v2.0
github.com/spaolacci/murmur3	BSD 3-Clause
github.com/spf13/cobra	Apache Software License v2.0
github.com/spf13/pflag	BSD 3-Clause
github.com/stretchr/testify/assert	MIT
github.com/tmc/grpc-websocket-proxy/wsproxy	MIT
github.com/uber/tchannel-go	MIT
github.com/ugorji/go	MIT
github.com/xiang90/probing	MIT
go.uber.org/atomic	MIT
go.uber.org/multierr	MIT
go.uber.org/zap	MIT
golang.org/x/crypto	BSD 3-Clause
golang.org/x/lint	BSD 3-Clause
golang.org/x/net	BSD 3-Clause
golang.org/x/sys	BSD 3-Clause
golang.org/x/text	BSD 3-Clause
golang.org/x/time	BSD 3-Clause
golang.org/x/tools	BSD 3-Clause
google.golang.org/googleapis	Apache Software License v2.0
google.golang.org/genproto	Apache Software License v2.0
google.golang.org/grpc	Apache Software License v2.0
gopkg.in/logfmt.v0	MIT
gopkg.in/natefinch/lumberjack.v2	MIT
gopkg.in/stack.v1	MIT
gopkg.in/yaml.v2	Apache Software License v2.0
gotest.tools	Apache Software License v2.0
stathat.com/c/consistent	MIT

gateway's Issues

Conflicting configuration file location in readme versus dockerfile

The dockerfile expects by default /var/config/sfproxy/sfdbproxy.conf, however the readme states:

It assumes you will have a sfdbconfig.json file cross mounted to /var/config/sfproxy/sfdbconfig.json for the docker container.

The filenames are not the same which can be misleading.

Please add support for Prometheus remote storage

An example could be found at: https://github.com/prometheus/prometheus/tree/master/documentation/examples/remote_storage/example_write_adapter

Is there plan to implement sub-sampling in forwarder?

E.g., if the incoming metric has a data rate of 10 data point per minute, when forwarding the metric to signalfx server, it does a down sample to 1 dps.

Add support for dynamic dimension configuration?

We run the signalfx metricproxy on each node in our kubernetes cluster, and apps on the node send their custom metrics through the local metricproxy. This works, but requires apps to be responsible for implementing our standard dimensions correctly in various programming languages and the infrastructure team can't trivially start applying new dimensions. What we really want is to be able to apply different dimensions based on the source IP of the client.

As it currently stands, we can hack this out in the metricproxy by setting up a listening port per-client IP and then using iptables to direct connections to different ports based on the IP, but that's rather complex. Additionally, that requires restarting the metricproxy because I was unable to find a way to dynamically reconfigure it at runtime.

It'd be awesome if I could provide a json file (or hit an http endpoint) mapping IPs (or some more generic rules) to dimensions and if there was some way to signal to the metricproxy that it should reload that. It'd also be awesome if this could be applied generically for all listeners somehow.

In the absence of this feature, I'm just going to make things use the collectd protocol since it puts the dimensions in URL parameters and use nginx instead of metricproxy to modify them, but that forces everything to use the collectd protocol so it'd be great if the metricproxy had a built-in feature for this.

What do you think?

Thanks,
Paul

Document cluster mode

Looking through the code base, I have noticed several mentions of clustering and etcd and I wanted to know how best to set this up locally to try it out.

I haven't seen any docs on this so far so I'd appreciate any advice on it.

readme typos

-- Correct first sentence to:
The SignalFx Metricproxy lets you aggregate metrics and send them to SignalFx.

--I believe this rewrite is accurate - the sentence as it stands is awkward/ambiguous:
We recommend placing the proxy either on the same server as another aggregator, such as statsd, or on a central server that is already receiving datapoints, such as graphite's carbon database.

-- In this sentence, either correct "not data" to "no data" or rewrite as shown:
correct: However, the service is limited in usefulness if there is not no data being sent to it.
or rewrite: However, the service is limited in usefulness if no data is being sent to it.

test

Setting AuthTokenEnvVar has no effect

I'm using a docker image that derivates from quay.io/signalfx/gateway:v1.0.12,
with the following environment (this under ECS):

"Env": [
                "AWS_EXECUTION_ENV=AWS_ECS_EC2",
                "SFX_ACCESS_TOKEN=yh44k..............",

and configuration:

{
  "MaxGracefulWaitTime": "30s",
    "GracefulCheckInterval": "1s",
    "SilentGracefulTime": "3s",
    "StatsDelay": "1s",
    "ForwardTo": [
      {
        "Type": "signalfx",
        "Name": "signalfxforwarder",
        "AuthTokenEnvVar": "SFX_ACCESS_TOKEN",
        "BufferSize": 1000000,
        "DrainingThreads": 50,
        "MaxDrainSize": 5000
      }
    ],
    "ListenFrom": [
      {
        "ListenAddr": "0.0.0.0:18080",
        "Type": "signalfx",
        "AdditionalDimentions": { }
      }
    ],
    "LogDir": "-"
}

however, the logs show:

time=2019-02-26T20:38:42Z caller=ratelimit.go:36 protocol=signalfx direction=forwarder name=signalfxforwarder struct=BufferedForwarder draining_index=14 err="invalid status code 401: " message="error sending datapoints"
time=2019-02-26T20:38:43Z caller=ratelimit.go:36 protocol=signalfx direction=forwarder name=signalfxforwarder err="invalid status code 401: " message="Unable to process datapoints"

I've verified the access token and it is active and working.

404 on collectd listener url /post-collectd

Despite the documentation here and the stanza at line 168 here, the url "/post-collectd" returns a 404. The "/healthz" path returns a 200 and "OK" response.

SignalFx Gateway Verison: 0.9.10+

gateway.conf:

  "MaxGracefulWaitTime": "1s",
  "GracefulCheckInterval": "1s",
  "SilentGracefulTime": "2s",
  "ForwardTo": [
        {
            "URL": "https://ingest.us1.signalfx.com/v2/datapoint",
            "DefaultAuthToken": "<redacted>",
            "Name": "<redacted>",
            "Type": "signalfx"
        }
    ],
    "ListenFrom": [
        {
            "ListenAddr": "0.0.0.0:18000",
            "Type": "collectd"
        }
    ],
    "LocalDebugServer": "0.0.0.0:6009",
    "LogDir": "-"
}```