go-graphite / carbonapi

Implementation of graphite API (graphite-web) in golang

carbonapi: replacement graphite API server

We are using to host our packages!

CarbonAPI supports a significant subset of graphite functions [see COMPATIBILITY]. In our testing it has shown to be 5x-10x faster than requesting data from graphite-web.

For requirements see Requirements section below.

Installation

At this moment we are building packages for CentOS 7, Rockylinux 8 (should be compatible with RHEL 8), Debian 10, Debian 11, Debian 12 (testing), Ubuntu 18.04, Ubuntu 20.04, Ubuntu 22.04. Installation guides are available on packagecloud (see the links below).

Stable versions: Stable repo

Autobuilds (master, might be unstable): Autobuild repo

Configuration guides: docs/configuration.md and example config.

There are multiple example configurations available for different backends: prometheus, graphite-clickhouse, go-carbon, victoriametrics, IRONdb

General information

Carbonapi can be configured by environment variables or by config file. For an example see carbonapi.example.yaml

$ ./carbonapi -config /etc/carbonapi.yaml

Request metrics will be dumped to graphite if corresponding config options are set, or if the GRAPHITEHOST/GRAPHITEPORT environment variables are found.

Request data will be stored in memory (default) or in memcache.

Configuration is described in docs

Configuration by environment variables

Every parameter in config file are mapped to environment variable. I.E.

concurency: 20
cache:
   # Type of caching. Valid: "mem", "memcache", "null"
   type: "mem"
upstreams:
    backends:
        - "http://10.0.0.1:8080"
        - "http://10.0.0.2:8080"

That config can be replaced by

CARBONAPI_CONCURENCY=20
CARBONAPI_CACHE_TYPE=mem
CARBONAPI_UPSTREAMS_BACKENDS="http://10.0.0.1:8080 http://10.0.0.2:8080"

You should be only aware of logging: because carbonapi support a list of logger, env variables will replace only first logger.

If you apply variable LOGGER_FILE=stdout to config:

logger:
    - logger: ""
      file: "stderr"
      level: "debug"
      encoding: "console"
      encodingTime: "iso8601"
      encodingDuration: "seconds"
    - logger: ""
      file: "carbonapi.log"
      level: "info"
      encoding: "json"

it will be equal to config:

logger:
    - logger: ""
      file: "stdout" # Changed only here
      level: "debug"
      encoding: "console"
      encodingTime: "iso8601"
      encodingDuration: "seconds"
    - logger: ""
      file: "carbonapi.log" # Not changed
      level: "info"
      encoding: "json"

Supported protocols

• auto - carbonapi will do it's best to determine backend's protocol. Currently it can identify only carbonapi_v2_pb or carbonapi_v3_pb
• carbonapi_v2_pb, pb, pb3, protobuf - carbonapi <0.11 style protocol. Supported by go-carbon and graphite-clickhouse older or equal version v0.11.7
• carbonapi_v3_pb - new carbonapi protocol, that supports passing metadata through. Supported by carbonzipper >=1.0.0.alpha.3, graphite-clickhouse newer then v0.12.0 and go-carbon newer then v0.13.0
• carbonapi_v3_grpc - grpc version of new carbonapi protocol. Currently no known implementation exists.
• msgpack - messagepack based protocol, used in graphite-web 1.1 and metrictank. It's still experimental and might contain bugs.
• prometheus - prometheus HTTP API
• victoriametrics - special version of prometheus backend to use with VictoriaMetrics.
• irondb - supports reading Graphite-compatible metrics from IRONdb from Circonus.

Requirements

Golang compatibility matrix:

Golang Version	Last supported carbonapi version
1.10	0.10.0.1
1.12	0.12.6
1.16 / 1.17	0.15.6
1.18	0.16.0-patch2
1.20	0.16.1

Overall rule of thumb is that carbonapi supports last 2 major go versions. E.x. at this moment Go 1.22 and 1.21 are supported.

You can verify current versions that are being tested in CI Configuration.

CarbonAPI uses protobuf-based protocol to talk with underlying storages. For current version the compatibility list is:

1. go-carbon >= 0.9.0 (Note: you need to enable carbonserver in go-carbon). Recommended to run latest version, that currently supports carbonapi_v3_pb
2. graphite-clickhouse any. That's alternative storage that doesn't use Whisper.
3. metrictank - supported via msgpack protocol. Support is not very well tested and might contain bugs. Use with cautions. Tags are not supported.
4. carbonapi >= 0.5. Note: starting from carbonapi 1274333ebd1fe50946cb4d51561e3e0f1060bc79 separate binary of carbonzipper is deprecated.
5. carbonserver@master (Note: you should probably switch to go-carbon in that case).
6. carbonzipper >= 0.50. Please note, carbonzipper functionality was merged to carbonapi and it's no longer needed to run separate zipper.

Supported architectures and OSs

Currently building is tested regularly on amd64 (automated) and arm64 (manual) only. However from time to time, riscv64 is also tested manually.

For OS support: Linux is the only OS that is well tested for production usage. Theoretically nothing prevents from running carbonapi on *BSD, however its not tested by developers (but bugs will be accepted and eventually fixed). Running on macos is supported for testing purposes but it is not tested for any production use case. Other platforms are not tested and not supported.

For any other OS or Architectures bugs won't be actively worked on, but PRs that fixes the OS and doesn't break any other supported platforms are more than welcome.

Some remarks on different backends

For backends that uses proper database (e.x. graphite-clickhouse) you should set maxBatchSize: 0 in your config file for this backend group.

For other backends (e.x. go-carbon) you should set it to some reasonable value. It increases response speed, but the cost is increased memory consumption.

Tag support was only tested with graphite-clickhouse, however it should work with any other database.

Internal Metrics

The internal metrics are configured inside the graphite subsection and sent to your destinated host on an specified interval. The metrics are:

Metric Name	Description
cache_items	if caching is enabled, this metric will contain many metrics are stored in cache
cache_size	configured query cache size in bytes
request_cache_hits	how many requests were served from cache. (this is for requests to /render endpoint)
request_cache_misses	how many requests were not in cache. (this is for requests to /render endpoint)
request_cache_overhead_ns	how much time in ns it took to talk to cache (that is useful to assess if cache actually helps you in terms of latency) (this is for
find_requests	requests server by endpoint /metrics/find
requests	requests served by endpoint /render
requests_in_XX_to_XX	request response times in percentiles
timeouts	number of timeouts while fetching from backend
backend_cache_hits	how many requests were not read from backend
backend_cache_misses	how many requests were not found in the backend

OSX Build Notes

Some additional steps may be needed to build carbonapi with cairo rendering on MacOSX.

Install cairo:

$ brew install Caskroom/cask/xquartz

$ brew install cairo --with-x11

Acknowledgement

This program was originally developed for Booking.com. With approval from Booking.com, the code was generalised and published as Open Source on github, for which the author would like to express his gratitude.

Booking.com's Fork

In summer 2018, Booking.com forked version 0.11 of carbonapi and continued development in their own repo: github.com/bookingcom/carbonapi.

License

This code is licensed under the BSD-2 license.

carbonapi's Issues

jsonp kills cache

Many graphing libraries use autogenerated jsonp callback names. This breaking caching heavily, since every request URI is now different.

carbonapi should report errors for unsupported functions

Right now we just return an empty metric list, instead a 500 or a json blob with a useful error message.

Add sumSeriesWithWildcards

http://graphite.readthedocs.org/en/latest/functions.html#graphite.render.functions.sumSeriesWithWildcards

Switch to https://github.com/gonum/plot

In addition to code.google.com shutting down, plotinum has moved to being part of the gonum project.

This move also comes with a substantial number of breaking API changes.

Support floating point constants with exponents

scale(foo.bar.baz, 1e-2)

Support rawData in render URLs

Apparently, in a render url, the rawData flag, sometimes written as rawData=1 isn't supported. We should map it onto format=raw.

Add aliasByMetric

carbonapi handles missing metrics differently to graphite

If you specify a missing metric as part of a larger formula, the query will fail in carbonapi but succeed in Graphite.

A simple example of this is:

data.metric1  # Exists
data.metric2  # Does not exists

diffSeries(data.metric1, data.metric2) # Fails on carbonapi
diffSeries(data.metric1, transformNull(data.metric2, 0)) # Also fails on carbonapi

I would expect the result to be data.metric1 since I'd assume that data.metric1 - [undefined] would equal data.metric1.

A valid use case in this situation is as followed:

# Metric layout
page.[page_name].requests
page.[page_name].errors

# The number of total successful pages served (number of requests - number of errors)
# If we automate this in any form we may hit a page that has never had an error recorded.
diffSeries(page.index_html.requests, page.index_html.errors) 

Add groupByNode()

Add hitcount()

http://graphite.readthedocs.org/en/latest/functions.html#graphite.render.functions.hitcount

Support jsonp=foo

add multiplySeries()

Or is there a way in there right now to multiply two series together? Can't seem to figure out a good way in grafana with the current api, I think multiplySeries() would do the job.

Handle the HTTP request method OPTIONS properly to allow all kinds of cross-domain requests

Sometimes the browser issues a HTTP request of the type OPTIONS when it's cross domain. I know a page that broke when using carbonapi because of this. It worked normally with the python version.

fix /info endpoint

Note that the breakage might actually be in carbonzipper or carbonserver.

Support for Pearson Product Moment Correlation Coefficient (PPMCC)

As mentioned in FB Gorilla paper add support for sorting/filtering metrics against a test metric using PPMCC

Improve expression parsing error messages

It turns out people have very poor graphite queries that fail to parse. Figuring out where the missing comma needs to go is tedious at the moment. Make this easier.

Don't cache failed requests

On carbonzipper restart, we will have a number of failed requests (~1 / cached idle connection). We don't want to add the results of these queries to our request or find cache.

License

What license is this and carbonzipper under?

movingMedian function

It would be nice to have it. Documentation here:
https://graphite.readthedocs.org/en/0.9.10/functions.html

Add stdev()

Log requests

time, ip, request, response size, ... ?

Fix diffSeries()'s multiple argument support

Support foo.{bar,baz}.qux in names

Name parsing code is too limited

/info?target= not working

If I send a request like this for a valid metric:
/info?target=metric_name
I always get the response:
empty target

If I sent the request like this:
/info/?target=metric_name
It's working.

add format=csv

Cache FetchResponses to speed up subsequent requests

The graphite front-end stores responses in memcache, meaning that subsequent requests for the same metric are much faster and don't hit the stores. This is important when lots of people are looking at the same graph or fetching the same metric.

An in-memory version is fine for starters, but we should move to externalize it so it can be shared across carbonapi instances.

Missing holtWinters* functions

There are people that would like to use with carbonapi the holtWinters functions present in the python version

Implement Kolmogorov–Smirnov 2 sample test

( Wikipedia description: Kolmogorov–Smirnov Test )

This would allow us to test, for example, a metric against itself in the past to see if the distribution has changed. Or if two metrics have similar distributions.

Naming might be among the hardest to implement.
My votes are:

• kolmogorovSmirnovTest2( seriesList, seriesList )
• ksTest2 (as the commonly used alias)

(I dislike the camelCasing standard here - but it maintains consistency)

"2" here because the 1-sample version, which essentially compares against a reference (usually normal) distribution, could also be implemented .. but there seem to be better tests for that.

Add asPercent()

Support /render/ and /render

Graphite handles both, but Go's default muxer doesn't.

missing, or incomplete support for stacked()

example request: https://graphite-api.[redacted]/render?target=aliasByNode(stacked(sys.*[redacted]*.[redacted].connections)%2C%201)&from=1439439654&until=1439482974&format=json&maxDataPoints=640

tukeyAbove(seriesList, interval, basis, n) : interval is unused

[This issue serves as a reminder / todo of interval's inteneded use.]

interval was conceived as a mechanism where the quantile threshold can be computed on a (leading IIRC) subset to enable this funciton to more efficiently process large seriesList over large intervals.

@dgryski and I discussed that we could move it to the end as an optional parameter.

Support from=-360h2m

Our internalString grammar is too simple.

Add support for the maxDataPoints parameter

It is documented here:

http://graphite.readthedocs.org/en/latest/render_api.html

and it seems to be used by grafana:

http://docs.grafana.org/installation/performance/

Per request rate limiting to carbonzipper

This will throttle bulk-style requests, but not adversely affect smaller API calls from proceding.

Requires a bit more thinking to figure out if this is a problem and if this is the best solution.

Time parsing code may not match exactly graphite's behaviour

Need to figure out "only minutely resolution" for queries, <= vs <, etc.

No CORS headers in response to OPTIONS (breaks preflighted AJAX requests)

The response to an OPTIONS request doesn't include the CORS headers that an preflight AJAX request expects[1] which makes the browser to refuse to perform the request.

[1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS#Preflighted_requests

Add minSeries()

http://graphite.readthedocs.org/en/latest/functions.html#graphite.render.functions.minSeries

" is a valid char in metric names

Figure out the full set. Figure out which subset we want to support. Hope we don't have to fully break expression parsing in the meantime.

Support PNG output

Work in progress on https://github.com/dgryski/carbonapi/tree/graph-support

Support /metrics/find/

via @atomicstack

function mostDeviant

The function mostDeviant is missing/not implemented. The implementation of this function in the original graphite seems to be very slow.

Add Tukey outlier detection

https://github.com/dgryski/go-onlinestats/blob/master/tukey.go

Provide default timezone for timespec's passed without one.

Question: preserve graphite bug compatibility or not?

Clean up evalExpr()

It's getting unwieldy.

Process query targets in parallel

At the moment, each metric for a given target is fetched in parallel, but the targets themselves are processed serially. For queries with a large number of targets, this can can unnecessary slow down.

"summarize" last element mismatch

When aggregating in a fixed time range with from/until, e.g., xxxxxx0 to xxxxxx84, aggregate by 5 seconds, there is sometimes an extra or a missing value, the array length is variable by 1.

Lots of request errors on carbonzipper restart

Because we maintain lots of cached http connections to carbonzipper, when that service is restarted we have a large number of (now invalid) sockets which cause failures during subsequent requests.

Work around is to restart carbonapi when carbonzipper is upgraded.

Cache more aggressively

Graphite munges timestamps to minutely resolution to increase the chance of a cache hit. It also stores the data returned by the stores so future queries on the same data can also be pulled from the cache.

Decide which of these we want to implement.

add remove(Above|Below)Percentile

I try to implement functions on demand.

These functions have been demanded.