pinterest / yuvi

Yuvi is an in-memory storage engine for recent time series metrics data.

Yuvi

Yuvi is an in-memory storage engine for recent time series metrics data. It has the following features:

• Implemented in Java.
• Supports OpenTSDB metric ingestion and OpenTSDB queries.
• Uses delta-of-delta encoding from Facebook Gorilla to store metrics data.
• Stores tag metadata in an inverted index for fast look ups during queries.
• Stores tag data and older metrics off heap to minimize GC pauses.
• Metrics data older than a few hours is rolled over.

Architecture

Yuvi consists a chunk manager that stores the metrics data the last few hours. A chunk manager manages several chunk, which consists of 2 hours of metrics data. Each chunk consists of a tag store to store metrics metdata and a metric store to store the metrics data. For efficiency, several chunks can also share a tag store. Once the data in a chunk is older than a configured amount of time, it is removed.

Sample code for using the library

import com.pinterest.yuvi.chunk.ChunkManager;

ChunkManager chunkManager = new ChunkManager("test", 1000);
chunkManager.addMetric("put metricName.cpu.util  1489647603 30 host=host1 cluster=c1");
List<TimeSeries> ts = chunkManager.query(Query.parse("metricName.cpu.util host=*"), 1489647600, 1489649600, QueryAggregation.NONE);

NOTE

This project is under active development. A dev version of OpenTSDB integration code can be found at: https://github.com/mansu/opentsdb/tree/yuvi-dev

yuvi's Issues

Add metrics to Yuvi

Add metrics to Yuvi using dropwizard metrics library.
http://metrics.dropwizard.io/3.2.3/getting-started.html

Add a YuviBuilder

Add a YuviBuilder that can build a Yuvi instance. This will simplify using Yuvi in the client code. Further, it will let us make internal code refactoring without affecting the clients.

Log errors from ScheduledThreadPool executors.

Currently, we schedule tasks using a ScheduledThreadPoolExector. When using this one has to be careful wrap all Runnable.run() methods in a try/catch block. If we fail to do that the repeating task fails to run after the first failure.

A smarter way is to use a loggingScheduledPoolExecutor like this one: https://github.com/apache/aurora/blob/c85bffdd6f68312261697eee868d57069adda434/src/main/java/org/apache/aurora/scheduler/base/AsyncUtil.java