questdb / c-questdb-client

Rust, C and C++ client for QuestDB InfluxDB Line Protocol

 

English | 简体中文 | 繁體中文 | العربية | Italiano | Українська | Español | Português | 日本語 | Türkçe | हिंदी | Tiếng Việt

QuestDB

QuestDB is an open-source time-series database for high throughput ingestion and fast SQL queries with operational simplicity.

QuestDB is well-suited for financial market data, IoT sensor data and real-time dashboards. It shines for datasets with high cardinality and is a drop-in replacement for InfluxDB via support for the InfluxDB Line Protocol.

QuestDB implements ANSI SQL with native time-series SQL extensions. These SQL extensions make it simple to filter and downsample data, or correlate data from multiple sources using relational and time-series joins.

We achieve high performance by adopting a column-oriented storage model, parallelized vector execution, SIMD instructions, and low-latency techniques. The entire codebase is built from the ground up in Java, C++ and Rust with no dependencies and zero garbage collection.

QuestDB supports schema-agnostic streaming ingestion using the InfluxDB line protocol and a REST API for bulk imports and exports. The QuestDB SQL Web Console is an interactive SQL editor facilitating CSV import. Finally, QuestDB also includes the Postgres Wire Protocol for programmatic queries.

Popular tools that integrate with QuestDB include Apache Kafka, Grafana, Superset, Telegraf and Apache Flink.

Try QuestDB

Our live demo is provisioned with the latest QuestDB release and sample datasets:

• Trips: 10 years of NYC taxi trips with 1.6 billion rows
• Trades: live crypto market data with 30M+ rows per month
• Pos: geolocations of 250k unique ships over time

Query	Execution time
SELECT sum(double) FROM trips	0.15 secs
SELECT sum(double), avg(double) FROM trips	0.5 secs
SELECT avg(double) FROM trips WHERE time in '2019'	0.02 secs
SELECT time, avg(double) FROM trips WHERE time in '2019-01-01' SAMPLE BY 1h	0.01 secs
SELECT * FROM trades LATEST ON timestamp PARTITION BY symbol	0.00025 secs

Our demo is running on c5.metal instance and using 24 cores out of 96.

Get started

Install QuestDB

To run QuestDB, Docker can be used to get started quickly:

docker run -p 9000:9000 -p 9009:9009 -p 8812:8812 questdb/questdb

macOS users can use Homebrew:

brew install questdb
brew services start questdb

questdb start // To start questdb
questdb stop  // To stop questdb

The QuestDB downloads page provides direct downloads for binaries and has details for other installation and deployment methods.

QuestDB Enterprise

QuestDB Enterprise (self-hosted or BYOC) allows your organization to run the database at scale, securely. Additional features include multi-primary ingestion, read replica(s), cold storage integration, Role-based access control, monitoring, TLS, and more.

Connect to QuestDB

You can interact with QuestDB using the following interfaces:

• Web Console for an interactive SQL editor and CSV import on port 9000
• InfluxDB line protocol for streaming ingestion on port 9000
• PostgreSQL wire protocol for programmatic queries and transactional inserts on port 8812
• REST API for CSV import and cURL on port 9000

Insert data

Below are the official QuestDB clients for ingesting data via the InfluxDB Line Protocol:

• .NET
• C/C++
• Go
• Java
• NodeJS
• Python
• Rust

Configure QuestDB for production workloads

Find our capacity planning to fine-tune QuestDB for production workloads.

End-to-end quickstart

Want to walk through everything, from streaming ingestion to visualization with Grafana? Check out our multi-path quickstart repository.

How QuestDB compares to other open-source TSDBs

Checkout our benchmark blog post which compares QuestDB and InfluxDB across functionality, maturity and performance.

Resources

📚 Read the docs

• QuestDB documentation: understand how to run and configure QuestDB.
• Tutorials: learn what's possible with QuestDB step by step.
• Product roadmap: check out our plan for upcoming releases.

📈 Demo dashboards

• Live demo: explore a live instance of QuestDB with preloaded datasets.
• Real-time crypto dashboard: Monitor executed trades on Coinbase from more than 20 assets in real time.
• NYC taxi: A real-time replay of 146,393,317 taxi rides, carrying 238,016,495 passengers across New York City in 2016.

❓ Get support

• Community Forum: join technical discussions, ask questions, and meet other users!
• GitHub issues: report bugs or issues with QuestDB.
• Stack Overflow: look for common troubleshooting solutions.

🚢 Deploy QuestDB

• AWS AMI
• Google Cloud Platform
• Official Docker image
• DigitalOcean droplets
• Kubernetes Helm charts

Contribute

We welcome contributions to the project, whether source code, documentation, bug reports, feature requests or feedback. To get started with contributing:

• Have a look through GitHub issues labelled "Good first issue".
• For Hacktoberfest, see the relevant labelled issues
• Read the contribution guide.
• For details on building QuestDB, see the build instructions.
• Create a fork of QuestDB and submit a pull request with your proposed changes.

✨ As a sign of our gratitude, we also send QuestDB swag to our contributors. Claim your swag.

A big thanks goes to the following wonderful people who have contributed to QuestDB: (emoji key):

clickingbuttons 💻 🤔 📓	ideoma 💻 📓 ⚠️	tonytamwk 💻 📓	sirinath 🤔	igor-suhorukov 💻 🤔	mick2004 💻 📦	rawkode 💻 🚇
solidnerd 💻 🚇	solanav 💻 📖	shantanoo-desai 📝 💡	alexprut 💻 🚧	lbowman 💻 ⚠️	chankeypathak 📝	upsidedownsmile 💻
Nagriar 💻	piotrrzysko 💻 ⚠️	mpsq 💻	siddheshlatkar 💻	Yitaek ✅ 💡	gabor-boros ✅ 💡	kovid-r ✅ 💡
TimBo93 🐛 📓	zikani03 💻	jaugsburger 💻 🚧	TheTanc 📆 🖋 🤔	davidgs 🐛 🖋	kaishin 💻 💡	bluestreak01 💻 🚧 ⚠️
patrickSpaceSurfer 💻 🚧 ⚠️	chenrui333 🚇	bsmth 📖 🖋	Ugbot 💬 📓 📢	lepolac 💻 🔧	tiagostutz 📓 🐛 📆	Lyncee59 🤔 💻
rrjanbiah 🐛	sarunas-stasaitis 🐛	RiccardoGiro 🐛	duggar 🐛	postol 🐛	petrjahoda 🐛	t00 🐛
snenkov 📓 🐛 🤔	marregui 💻 🤔 🎨	bratseth 💻 🤔 📓	welly87 🤔	fuzzthink 🤔 📓	nexthack 💻	g-metan 🐛
tim2skew 🐛 📓	ospqsp 🐛	SuperFluffy 🐛	nu11ptr 🐛	comunidadio 🐛	mugendi 🤔 🐛 📖	paulwoods222 🐛
mingodad 🤔 🐛 📖	houarizegai 📖	jjsaunier 🐛	zanek 🤔 📆	Geekaylee 📓 🤔	lg31415 🐛 📆	null-dev 🐛 📆
ultd 🤔 📆	ericsun2 🤔 🐛 📆	giovannibonetti 📓 🐛 📆	wavded 📓 🐛	puzpuzpuz 📖 💻 📓	rstreics 💻 🚇 📖	mariusgheorghies 💻 🚇 📖
pswu11 🖋 🤔 🎨	insmac 💻 🤔 🎨	eugenels 💻 🤔 🚧	bziobrowski 💻 📆	Zapfmeister 💻 📓	mkaruza 💻	DylanDKnight 📓 🐛
enolal826 💻	glasstiger 💻	argshook 💻 🤔 🎨 🐛	amunra 💻 📖 🐛	GothamsJoker 💻	kocko 💻	jerrinot 💻 🤔 🐛
rberrelleza 💻	Cobalt-27 💻	eschultz 💻	XinyiQiao 💻	terasum 📖	PlamenHristov 💻	tris0laris 📝 🤔
HeZean 💻 🐛	iridess 💻 📖	selmanfarukyilmaz 🐛	donet5 🤔 🐛	Zahlii 🐛	salsasepp 🐛	EmmettM 🐛 ⚠️
robd003 🤔	AllenEdison 🐛	CSharpDummy 🐛	shimondoodkin 🐛 🤔	huuhait 🐛 🤔	alexey-milovidov 🐛	suconghou 🐛
allegraharris 💻	oliver-daniel 💻	kerimsenturk5734 📖

This project adheres to the all-contributors specification. Contributions of any kind are welcome!

c-questdb-client's Issues

Error in Getting Started example

The Getting Started at https://crates.io/crates/questdb-rs has the following example:

This Rust crate supports a number of optional features.
For example, if you want to work with ILP/HTTP and work with Chrono timestamps, use:
cargo add questdb-rs --features ilp-over-http chrono

but when I execute cargo add questdb-rs --features ilp-over-http chrono I get this error:

error: feature ilp-over-http must be qualified by the dependency it's being activated for, like questdb-rs/ilp-over-http, chrono/ilp-over-http

I tried to add chrono as a dependency first and only then run cargo add questdb-rs --features ilp-over-http chrono but I'm still getting the same error.

Implement new HTTP retry logic to match Java HTTP client logic

Rework as so (pseudo-code):

ok, err = send_request()
if ok:
    return ok
if config.retry_timeout > 0:
    retry_begin = now()
    retry_interval = 10ms
    loop:
        jitter = millis(rand_between(0, 10)) - 5ms  # centre jitter
        sleep(retry_interval + jitter)
        ok, err = send_request()
        if ok:
            return ok
        elapsed = now() - retry_begin
        if elapsed > config.retry_timeout:
            return err
        retry_interval = min(retry_interval * 2, 1000ms)
else:
    return err

Note this drops the retry_interval and retry_count builder settings and introduces a new retry_timeout in its place.

Missing cmake install section

Hello! Is it reasonable to add install instructions to CMakeLists.txt?

install(TARGETS 
c_questdb_client
c_questdb_client_static 
line_sender_c_example
line_sender_cpp_example
test_line_sender
DESTINATION ${CMAKE_INSTALL_PREFIX})

if(UNIX)
install(TARGETS 
c_questdb_client_static_pic
DESTINATION ${CMAKE_INSTALL_PREFIX})
endif()

Is it possible to use this client library via find_package?

Reduce the number of necessary parameters when using ILP auth

For connecting via ILP auth, only the key_id and the private_key/d parameter should be needed. However, this client also requests the pub_key_x and pub_key_y, which is annoying as those are not really needed, and the developer need to treat them as secrets.

Some official clients like Go, JAVA, or C# don't need the extra parameters.

Once changed, the documentation should be updated to reflect the simplified connection params.

No error or write when attempting to write without auth

Using questdb-rs 3.0.0 when attempting to connect and write to a QuestDB service that requires authentication no error is returned and no data is written.

Code:

use questdb::{
    ingress::{Buffer, SenderBuilder},
    Result,
};

fn main() -> Result<()> {
    let mut sender = SenderBuilder::new("127.0.0.1", 9009).connect()?;
    let mut buffer = Buffer::new();
    buffer
        .table("test_table")?
        .column_str("id", "hello world")?
        .column_f64("temperature", 20.0)?
        .column_i64("humidity", 50)?
        .at_now()?;
    sender.flush(&mut buffer)?;
    Ok(())
}

QuestDB 7.3.2 service running locally exposed on 9009 but requires authentication (setup using https://questdb.io/docs/reference/api/ilp/authenticate/).

I would expect an error rather than silently failing when attempting to connect to service that requires authentication.

More information:
The table exists with those columns and connecting with authentication or removing authentication requirement from service successfully writes the data to the table. Also if I remove authentication requirement from service and try to connect with authentication I get an error as expected (Error { code: AuthError, msg: "Did not receive auth challenge. Is the database configured to require authentication?" }).

Question/Bug: Support more than 6 significant digits

Hello,

I am having problems (and maybe found a bug) with exporting large double values to my local QuestDB instance.
I have made some test cases to demonstrate them.
Any help is appreciated!

General Question:

Is there a way to increase the significant digits for double values that are sent via ILP,
because per default only 6 significant digits are supported.
I could not find any configuration for this.

--> See test cases 1 and 2

Problem

When the scientific notation is used (7 digits or more before the decimal point) in the line protocol and it seems like QuestDB can not parse this.
InfluxDB supports the scientific notation.

--> See test case 3

Setup

CREATE TABLE 'test' (
 ts TIMESTAMP,
 sensor_id SYMBOL,
 value double
)
timestamp(ts) PARTITION BY Day;

 sender
   .table(table_name)
   .symbol(id_name, "sensor_1"_utf8)
   .column(value_name, value)
   .at_now();  

Test Cases

Case 1

  double value = 1.1234567;

Result in WebUI:

Rounded to only 6 significant digits.

Case 2

 double value = 654321.123;

Result in WebUI:

Only 6 significant digits. Digits after decimal point are ignored.

Case 3

 double value = 7654321.123;

Console:

Could not flush buffered messages: An established connection was aborted by the software in your host machine.

QuestDB Log:

E i.q.c.l.t.LineTcpConnectionContext [2756] could not parse measurement, INVALID_FIELD_VALUE at 38, line (may be mangled due to partial parsing): 'test,sensor=sensor_1 value=7.65432e+06'

As said, the scientific notation, that is used for 7 digits or more, can not be parsed by QuestDB.

Incomplete ingres documentation

This page was partially updated to accomodate for ILP@HTTP changes, but it looks incomplete:

1. It says:

   The ingress module implements QuestDB’s variant of the InfluxDB Line Protocol (ILP) over TCP.

   This is misleading at best. The client supports both TCP and HTTP transport. The example below this sentence uses the HTTP Transport

2. The paragraph Connection Security Options has a code sample which links to a URL which returns 404:
   

3. The document is silent on configuration, error-handling, retrying, etc. Chances are it's described elsewhere, but as a Rust developer, I would expect this document to give me an overview of all important concepts.

Error running the basic example

Running the basic example in https://crates.io/crates/questdb-rs and getting

   Compiling questdb-rs v2.1.3
error: failed to run custom build command for `questdb-rs v2.1.3`

Caused by:
  process didn't exit successfully: `/Users/samlaf/devel/sui/sui-mev-bot/rust/target/debug/build/questdb-rs-b4cbfe11b6777abc/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-changed=build.rs
  cargo:rerun-if-changed=Cargo.lock
  cargo:rerun-if-changed=src/test/interop/ilp-client-interop-test.json

  --- stderr
  thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error("invalid type: map, expected f64", line: 27, column: 13)', /Users/samlaf/.cargo/registry/src/github.com-1ecc6299db9ec823/questdb-rs-2.1.3/build.rs:83:39
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...

:(

Enable KeepAlive when creating the socket connection

Enable keepalive right after this line:

c-questdb-client/questdb-rs/src/ingress/mod.rs

Line 1630 in ad3776e

sock.set_nodelay(true)

This is to keep the connection open in case of long period of inactivity.

Support for explicit timestamp column.

See:

• questdb/go-questdb-client#2
• questdb/questdb#2257

Rust implementation of the client does not disconnect on failed authentication

See: questdb/py-questdb-client#49

[docs] Link pointing to generic page

At the README for the c client we mention

See the main client libraries documentation for the full config string params, including authentication, tls, etc.

But that link does not really explain what promises. We should probably either document parameters, remove the link, or point to the page with the relevant information

Implement optional authentication

The current implementation lacks authentication and it should be implemented.

error LNK2019: unresolved external symbol __imp_NtCreateFile referenced in function ...

Reqs: Build the c-questdb-client lib and link it to a cpp project
Setup: Windows Server 2019 Standard, MS Visual Studio 2022 v.17.7.3
Steps to reproduce: Just build c-questdb-client library in Visual Studio.
Output: the same errors for all *.c and *.cpp examples

FAILED: line_sender_cpp_example.exe 
cmd.exe /C "cd . && "C:\Program Files\CMake\bin\cmake.exe" -E vs_link_exe --intdir=CMakeFiles\line_sender_cpp_example.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100220~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100220~1.0\x64\mt.exe --manifests  -- C:\PROGRA~1\MICROS~1\2022\COMMUN~1\VC\Tools\MSVC\1437~1.328\bin\Hostx64\x64\link.exe /nologo CMakeFiles\line_sender_cpp_example.dir\examples\line_sender_cpp_example.cpp.obj  /out:line_sender_cpp_example.exe /implib:line_sender_cpp_example.lib /pdb:line_sender_cpp_example.pdb /version:0.0 /machine:x64 /debug /INCREMENTAL /subsystem:console  questdb_client.lib  advapi32.lib  userenv.lib  ws2_32.lib  bcrypt.lib  kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib && cd ."
LINK Pass 1: command "C:\PROGRA~1\MICROS~1\2022\COMMUN~1\VC\Tools\MSVC\1437~1.328\bin\Hostx64\x64\link.exe /nologo CMakeFiles\line_sender_cpp_example.dir\examples\line_sender_cpp_example.cpp.obj /out:line_sender_cpp_example.exe /implib:line_sender_cpp_example.lib /pdb:line_sender_cpp_example.pdb /version:0.0 /machine:x64 /debug /INCREMENTAL /subsystem:console questdb_client.lib advapi32.lib userenv.lib ws2_32.lib bcrypt.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST /MANIFESTFILE:CMakeFiles\line_sender_cpp_example.dir/intermediate.manifest CMakeFiles\line_sender_cpp_example.dir/manifest.res" failed (exit code 1120) with the following output:
LINK : warning LNK4098: defaultlib 'MSVCRT' conflicts with use of other libs; use /NODEFAULTLIB:library
questdb_client.lib(std-0f357f0f8e8eb48f.std.895242974ff7d01b-cgu.0.rcgu.o) : error LNK2019: unresolved external symbol __imp_NtCreateFile referenced in function _ZN3std3sys7windows2fs20open_link_no_reparse17h262f7b3e7853935cE
questdb_client.lib(std-0f357f0f8e8eb48f.std.895242974ff7d01b-cgu.0.rcgu.o) : error LNK2019: unresolved external symbol __imp_RtlNtStatusToDosError referenced in function _ZN3std3sys7windows2fs20open_link_no_reparse17h262f7b3e7853935cE
questdb_client.lib(std-0f357f0f8e8eb48f.std.895242974ff7d01b-cgu.0.rcgu.o) : error LNK2019: unresolved external symbol __imp_NtReadFile referenced in function _ZN3std3sys7windows6handle6Handle16synchronous_read17h00067f19ba39bde9E
questdb_client.lib(std-0f357f0f8e8eb48f.std.895242974ff7d01b-cgu.0.rcgu.o) : error LNK2019: unresolved external symbol __imp_NtWriteFile referenced in function _ZN3std3sys7windows6handle6Handle17synchronous_write17h977c567d7f264f32E
line_sender_cpp_example.exe : fatal error LNK1120: 4 unresolved externals

Also if to try to link the library to a test project (created in accordance with DEPENDENCY.md) the errors are same.

Flushing buffer to closed connection causes crash

When the connection with the database server is interrupted, flushing buffers to the sender cause crash.
Example:

bool sendRandomBuffer(questdb::ilp::line_sender * sender)
{
  try  
  {
      questdb::ilp::line_sender_buffer buffer;

      auto bufferSize = randomInt();
      for(int i = 0; i < bufferSize; i++)
      {
          // CREATE TABLE myTable (myInt INT, myString STRING, time TIMESTAMP) TIMESTAMP(time);
          buffer 
              .table("myTable")
              .column("myInt", randomInt())
              .column("myString", randomString())
              .at_now();
      }
      sender->flush(buffer);
      return true;
  }
  catch(std::runtime_error & e)
  {
      std::cerr << "error: " << e.what() << std::endl;
      return false;
  }
}

int main()
{
  std::unique_ptr<questdb::ilp::line_sender> sender;
  try
  {
    sender = std::make_unique<questdb::ilp::line_sender>("localhost", 9009);
  }
  catch(std::runtime_error & e)
  {
      std::cerr << "error: " << e.what() << std::endl;
      return 1;
  }

  std::cout << sendRandomBuffer(sender.get()) << std::endl; // everything is awesome

  //wait to stop database (./questdb.sh stop)
  std::cin.get();

  std::cout << sendRandomBuffer(sender.get()) << std::endl; // success? HOW? Not the biggest problem anyway
  std::cout << sendRandomBuffer(sender.get()) << std::endl; // crash, the biggest problem

  return 0;
}

Flushing the buffer to the disconnected sender should throw an exception every time. Or there should be something like bool line_sender::is_connected() to check if I can flush the buffer to the sender.

Not query new data when write in quick speed

When I flush data in quick speed about 10ms, then query in WebUI, but get not newest data;

Crates.io Questdb name

Hi! I'm the current owner of crates.io/crates/questdb. If you'd like me to transfer it to you, I'll gladly do it.

Missing flush_and_keep_with_flags() in C++

The C++ class line_sender is missing flush_and_keep_with_flags(). Without this method, the user can't make a flush that is checked to be transactional.

If it contains Chinese, it will report an error

If it contains Chinese, it will report an error

Support for LONG, BINARY types with rust client

Is there an accepted approach in Rust to write LONG / LONG256 / BINARY types using the Buffer's provided?

Investigate HTTP timeout issues

Details are here: questdb/py-questdb-client#75