We are making lots of changes right now.
- Project website: https://verdictdb.org
- Documentation: https://verdict.readthedocs.org
Verdict brings interactive-speed, resource-efficient data analytics, with the following key features:
- 200x faster by sacrificing only 1% accuracy Verdict can give you 99% accurate answers for your big data queries in a fraction of the time needed for calculating exact answers. If your data is too big to analyze in a couple of seconds, you will like Verdict.
- No change to your database Verdict is a middleware standing between your application and your backend engine. You can just issue the same queries as before and get precise estimates computed instantly.
- Runs on (almost) any database Verdict is designed to run on any database that supports standard SQL. Right now, we support Presto and will soon add other open source engines.
- Ease of use Verdict is a light-weight client-side library: no servers, no port configurations, no extra user authentication, etc., beyond what you already have.
Launch verdict in a single line (with Presto for its backend engine).
curl -s https://raw.githubusercontent.com/verdictproject/verdict/master/docker-compose-64gb.yaml \
| docker-compose -f - up
Once the docker containers run, start the Python shell as follows:
# bash
docker exec -it docker-verdict python
# Python shell
import verdict
v = verdict.presto(presto_host='presto') # connects to Presto via Verdict
v.sql("bypass show catalogs")
v.sql('bypass select count(*) from tpch.sf1.orders')
# Returning an answer in 8.863600015640259 sec(s).
# _col0
# 0 1500000
The bypass
keyword makes the query processed directly by the backend engine.
v.create_sample('tpch.sf1.orders')
The same count query:
v.sql('select count(*) from tpch.sf1.orders')
# Returning an answer in 0.17403197288513184 sec(s).
# c1
# 0 1503884
Another query:
v.sql('select orderpriority, count(*) from tpch.sf1.orders group by orderpriority')
# Returning an answer in 0.14169764518737793 sec(s).
# orderpriority c1
# 0 1-URGENT 300784
# 1 2-HIGH 301540
# 2 3-MEDIUM 298872
# 3 4-NOT SPECIFIED 302060
# 4 5-LOW 300628
You can issue more complex queries including joins or subqueries. See our documentation for more examples.
Note: The above latency comparisons are for quick demo of Verdict and are not meant to be scientific.
- Verdict rewrites your queries to use special types of samples (instead of original tables).
- The rewritten queries are processed by the backend engine in the regular way.
- Given the answers from the engine, Verdict composes statistically unbiased estimates (for your final answers), which are returned.
Even the samples are stored in your engines/stores (database, S3, and so on).
- User Story: https://verdictdb.org/documentation/success/
- Research: https://verdictdb.org/documentation/research/
- License: Apache License 2.0
- Project Maintainers: Alumni of the database group at University of Michigan, Ann Arbor
- How to Contact: Use our Gitter room or directly email us (Yongjoo and Barzan).