Coder Social home page Coder Social logo

clickhouse-cpp's Introduction

ClickHouse C++ client Linux macOS Windows MSVC Windows mingw

C++ client for ClickHouse.

Supported data types

  • Array(T)
  • Date
  • DateTime, DateTime64
  • DateTime([timezone]), DateTime64(N, [timezone])
  • Decimal32, Decimal64, Decimal128
  • Enum8, Enum16
  • FixedString(N)
  • Float32, Float64
  • IPv4, IPv6
  • Nullable(T)
  • String
  • LowCardinality(String) or LowCardinality(FixedString(N))
  • Tuple
  • UInt8, UInt16, UInt32, UInt64, Int8, Int16, Int32, Int64
  • Int128
  • UUID
  • Map
  • Point, Ring, Polygon, MultiPolygon

Dependencies

In the most basic case one needs only:

  • a C++-17-complaint compiler,
  • cmake (3.12 or newer), and
  • ninja

Optional dependencies:

  • openssl
  • liblz4
  • libabsl
  • libzstd

Building

$ mkdir build .
$ cd build
$ cmake .. [-DBUILD_TESTS=ON]
$ make

Plese refer to the workflows for the reference on dependencies/build options

Example application build with clickhouse-cpp

There are various ways to integrate clickhouse-cpp with the build system of an application. Below example uses the simple approach based on submodules presented in https://www.youtube.com/watch?v=ED-WUk440qc .

  • mkdir clickhouse-app && cd clickhouse-app && git init
  • git submodule add https://github.com/ClickHouse/clickhouse-cpp.git contribs/clickhouse-cpp
  • touch app.cpp, then copy the following C++ code into that file
#include <iostream>
#include <clickhouse/client.h>

using namespace clickhouse;

int main()
{
    /// Initialize client connection.
    Client client(ClientOptions().SetHost("localhost"));

    /// Create a table.
    client.Execute("CREATE TABLE IF NOT EXISTS default.numbers (id UInt64, name String) ENGINE = Memory");

    /// Insert some values.
    {
        Block block;

        auto id = std::make_shared<ColumnUInt64>();
        id->Append(1);
        id->Append(7);

        auto name = std::make_shared<ColumnString>();
        name->Append("one");
        name->Append("seven");

        block.AppendColumn("id"  , id);
        block.AppendColumn("name", name);

        client.Insert("default.numbers", block);
    }

    /// Select values inserted in the previous step.
    client.Select("SELECT id, name FROM default.numbers", [] (const Block& block)
        {
            for (size_t i = 0; i < block.GetRowCount(); ++i) {
                std::cout << block[0]->As<ColumnUInt64>()->At(i) << " "
                          << block[1]->As<ColumnString>()->At(i) << "\n";
            }
        }
    );

    /// Delete table.
    client.Execute("DROP TABLE default.numbers");

    return 0;
}
  • touch CMakeLists.txt, then copy the following CMake code into that file
cmake_minimum_required(VERSION 3.12)
project(application-example)

set(CMAKE_CXX_STANDARD 17)

add_subdirectory(contribs/clickhouse-cpp)

add_executable(${PROJECT_NAME} "app.cpp")

target_include_directories(${PROJECT_NAME} PRIVATE contribs/clickhouse-cpp/ contribs/clickhouse-cpp/contrib/absl)

target_link_libraries(${PROJECT_NAME} PRIVATE clickhouse-cpp-lib)
  • run rm -rf build && cmake -B build -S . && cmake --build build -j32 to remove remainders of the previous builds, run CMake and build the application. The generated binary is located in location build/application-example.

Thread-safety

⚠ Please note that Client instance is NOT thread-safe. I.e. you must create a separate Client for each thread or utilize some synchronization techniques. ⚠

Retries

If you wish to implement some retry logic atop of clickhouse::Client there are few simple rules to make you life easier:

  • If previous attempt threw an exception, then make sure to call clickhouse::Client::ResetConnection() before the next try.
  • For clickhouse::Client::Insert() you can reuse a block from previous try, no need to rebuild it from scratch.

See #184 for details.

clickhouse-cpp's People

Contributors

1261385937 avatar abyss7 avatar alenstarx avatar alesapin avatar alexey-milovidov avatar arthurpassos avatar artpaul avatar ays7 avatar bduminiuc avatar byian avatar ciberst avatar clrusby avatar den818 avatar enmk avatar filimonov avatar german3w1 avatar huyphams avatar itrofimow avatar jihadist avatar mikhailburdukov avatar nul1 avatar oraoto avatar pakapor avatar pavel-odintsov avatar rschu1ze avatar rtcote avatar timoll avatar traceon avatar tshev avatar v5r avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

clickhouse-cpp's Issues

zSTD Support

Hi, is there a reason zstd support isn't in this client? Would you be willing to accept a PR for it or is there a reason it's purposely left out?

Thread safety of Client

It seems Client class is not thread safe - i.e. sending concurrent queries from different threads might break. Can you please either make it thread-safe or at least document the current state of things.

Bug in EnumType::GetName

An EnumType with empty map cores, or loops forever, depending on the implementation of std::map iterator.
It starts of with it = map.begin() and then increments it before checking for map.== end()

secure and verify option for connection

Dear clickhouse-cpp team,

this python driver package for ch supports secure and verify options, which are connection params for
--> secure: establish secure connection.
--> verify: specifies whether a certificate is required and whether it will be validated after connection.

I wanted to ask if there are any plans in the future to adopt these features for a more secure connection.

Best regards,
tridelt

undefined reference to `clickhouse::Block::GetRowCount() const' occurs every time

I'm trying to run example and every time occurs next errors:
/usr/bin/ld: CMakeFiles/myTest.dir/main.cpp.o: in function main::{lambda(clickhouse::Block const&)#1}::operator()(clickhouse::Block const&) const': /root/CLionProjects/myTest/main.cpp:35: undefined reference to clickhouse::Block::GetRowCount() const'
/usr/bin/ld: /root/CLionProjects/myTest/main.cpp:36: undefined reference to clickhouse::Block::operator[](unsigned long) const' /usr/bin/ld: /root/CLionProjects/myTest/main.cpp:36: undefined reference to clickhouse::ColumnVector::At(unsigned long) const'
/usr/bin/ld: /root/CLionProjects/myTest/main.cpp:37: undefined reference to clickhouse::Block::operator[](unsigned long) const' /usr/bin/ld: /root/CLionProjects/myTest/main.cpp:37: undefined reference to clickhouse::ColumnString::At[abi:cxx11](unsigned long) const'
/usr/bin/ld: CMakeFiles/myTest.dir/main.cpp.o: in function main': /root/CLionProjects/myTest/main.cpp:9: undefined reference to clickhouse::Client::Client(clickhouse::ClientOptions const&)'
/usr/bin/ld: /root/CLionProjects/myTest/main.cpp:12: undefined reference to clickhouse::Query::Query(char const*)' /usr/bin/ld: /root/CLionProjects/myTest/main.cpp:12: undefined reference to clickhouse::Client::Execute(clickhouse::Query const&)'
/usr/bin/ld: /root/CLionProjects/myTest/main.cpp:12: undefined reference to clickhouse::Query::~Query()' /usr/bin/ld: /root/CLionProjects/myTest/main.cpp:16: undefined reference to clickhouse::Block::Block()'
/usr/bin/ld: /root/CLionProjects/myTest/main.cpp:19: undefined reference to clickhouse::ColumnVector<unsigned long>::Append(unsigned long const&)' /usr/bin/ld: /root/CLionProjects/myTest/main.cpp:20: undefined reference to clickhouse::ColumnVector::Append(unsigned long const&)'
/usr/bin/ld: /root/CLionProjects/myTest/main.cpp:23: undefined reference to clickhouse::ColumnString::Append(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /usr/bin/ld: /root/CLionProjects/myTest/main.cpp:24: undefined reference to clickhouse::ColumnString::Append(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
/usr/bin/ld: /root/CLionProjects/myTest/main.cpp:26: undefined reference to clickhouse::Block::AppendColumn(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<clickhouse::Column> const&)' /usr/bin/ld: /root/CLionProjects/myTest/main.cpp:27: undefined reference to clickhouse::Block::AppendColumn(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::shared_ptrclickhouse::Column const&)'
/usr/bin/ld: /root/CLionProjects/myTest/main.cpp:29: undefined reference to clickhouse::Client::Insert(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, clickhouse::Block const&)' /usr/bin/ld: /root/CLionProjects/myTest/main.cpp:16: undefined reference to clickhouse::Block::~Block()'
/usr/bin/ld: /root/CLionProjects/myTest/main.cpp:33: undefined reference to clickhouse::Client::Select(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void (clickhouse::Block const&)>)' /usr/bin/ld: /root/CLionProjects/myTest/main.cpp:43: undefined reference to clickhouse::Query::Query(char const*)'
/usr/bin/ld: /root/CLionProjects/myTest/main.cpp:43: undefined reference to clickhouse::Client::Execute(clickhouse::Query const&)' /usr/bin/ld: /root/CLionProjects/myTest/main.cpp:43: undefined reference to clickhouse::Query::~Query()'
/usr/bin/ld: /root/CLionProjects/myTest/main.cpp:9: undefined reference to clickhouse::Client::~Client()' /usr/bin/ld: /root/CLionProjects/myTest/main.cpp:12: undefined reference to clickhouse::Query::~Query()'
/usr/bin/ld: /root/CLionProjects/myTest/main.cpp:16: undefined reference to clickhouse::Block::~Block()' /usr/bin/ld: /root/CLionProjects/myTest/main.cpp:43: undefined reference to clickhouse::Query::~Query()'
/usr/bin/ld: /root/CLionProjects/myTest/main.cpp:9: undefined reference to clickhouse::Client::~Client()' /usr/bin/ld: CMakeFiles/myTest.dir/main.cpp.o: in function std::shared_ptrclickhouse::ColumnString std::dynamic_pointer_cast<clickhouse::ColumnString, clickhouse::Column>(std::shared_ptrclickhouse::Column const&)':
/usr/include/c++/8/bits/shared_ptr.h:514: undefined reference to typeinfo for clickhouse::ColumnString' /usr/bin/ld: CMakeFiles/myTest.dir/main.cpp.o: in function void __gnu_cxx::new_allocator<clickhouse::ColumnVector >::construct<clickhouse::ColumnVector>(clickhouse::ColumnVector)':
/usr/include/c++/8/ext/new_allocator.h:136: undefined reference to clickhouse::ColumnVector<unsigned long>::ColumnVector()' /usr/bin/ld: CMakeFiles/myTest.dir/main.cpp.o: in function void __gnu_cxx::new_allocatorclickhouse::ColumnString::constructclickhouse::ColumnString(clickhouse::ColumnString
)':
/usr/include/c++/8/ext/new_allocator.h:136: undefined reference to `clickhouse::ColumnString::ColumnString()'
collect2: error: ld returned 1 exit status
make[3]: *** [CMakeFiles/myTest.dir/build.make:84: myTest] Error 1
make[2]: *** [CMakeFiles/Makefile2:76: CMakeFiles/myTest.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/myTest.dir/rule] Error 2
make: *** [Makefile:118: myTest] Error 2

OS - clean installed Debian 10 amd64
gcc version 8.3.0
cmake version 3.13.4
CMAKE_CXX_STANDARD 14

I've built this library using official steps:
$ mkdir build .
$ cd build
$ cmake ..
$ make

After I'm trying main example and every time occurs problem with undefined reference.
Please help.

Performance improvements

  • add bool WireFormat::SkipString(CodedInputStream*) (used below);
  • Client::Impl::ReceiveData() uses SkipString to skip over an unused string in the input, instead of creating a std::string, reading into it, then discarding it;
  • Client::Impl::ReadBlock(...) reuses the std::string variables name and type for all columns, instead of allocating these two std::strings for each column;
  • Client::Impl::Insert(...) builds the fields_section more efficiently, directly from the block, without creating and copying from the block into a std::vector<std::string>, and then writing to std::stringstream from it.

Cannot insert data into the table with ' ' (space) in one of its column name.

From one of the example cases, if I put the column name test id and run the insert function, the program throws the exception.

Example Code:

#include <clickhouse/client.h>
using namespace clickhouse;

/// Initialize client connection.
Client client(ClientOptions().SetHost("localhost"));

/// Create a table.
client.Execute("CREATE TABLE IF NOT EXISTS test.numbers (\`test id\` UInt64, name String) ENGINE = Memory");

/// Insert some values.
try {
	Block block;

	auto id = std::make_shared<ColumnUInt64>();
	id->Append(1);
	id->Append(7);

	auto name = std::make_shared<ColumnString>();
	name->Append("one");
	name->Append("seven");

	block.AppendColumn("test id"  , id);
	block.AppendColumn("name", name);

	client.Insert("test.numbers", block);
} catch (std::exception &e) {
	printf("Error: %s\n", e.what());
}

Error: DB::Exception: Syntax error: failed at position 33: id,name ) VALUES. Expected one of: token, ClosingRoundBracket, Comma, Dot

I then tried putting ' " and ` around the column name for the data block, it throws on another line instead.

block.AppendColumn("`test id`" , id);

Error: DB::Exception: Not found column test id in block. There are only columns: test id, name

ipv4 column append problem

Dear clickhouse-cpp team,

The following function in clickhouse-cpp/clickhouse/columns/ip4.cpp is always thrown an exception, when ipv4 address is 255.255.255.255:

void ColumnIPv4::Append(const std::string& str) {
    in_addr_t addr = inet_addr(str.c_str());
    if (addr == INADDR_NONE) {
        throw std::runtime_error("invalid IPv4 format, ip: " + str);
    }
    data_->Append(htonl(addr));
}

Because inet_addr function is obsolete, and this is a known issue. https://www.gnu.org/software/libc/manual/html_node/Host-Address-Functions.html

The possible solution is the following:

void ColumnIPv4::Append(const std::string& str) {
    struct in_addr addr;
    if (inet_aton(str.c_str(),&addr) == 0) {
        throw std::runtime_error("invalid IPv4 format, ip: " + str);
    }
    data_->Append(htonl(addr.s_addr));
}

Please solve this problem.

Thank you and best regards,
BB

[error C2668] 'std::uncaught_exception'

Hi Clickhouse Team:
When i update clickhouse-cpp to version 1.5.0. I build failed due to:

error C2668: 'signbit': ambiguous call to overloaded function

return std::signbit(v);

I have a temporary workround, I want to know if it is feasible, or do you have a better solution?

diff --git a/clickhouse/columns/decimal.cpp b/clickhouse/columns/decimal.cpp
index 7334bb1..41d43b6 100644
--- a/clickhouse/columns/decimal.cpp
+++ b/clickhouse/columns/decimal.cpp
@@ -27,7 +27,7 @@ inline bool mulOverflow(const Int128 & l, const T & r, Int128 * result)
 
 #else
 template <typename T>
-inline bool getSignBit(const T & v)
+inline bool getSignBit(const T & (long double v))
 {
     return std::signbit(v);
 }

Related pr #18330

i need urgent help

I couldn't uninstall the project.
$ mkdir build .
$ cd build
$ cmake ..
$ make
i made this. Next step
myexampleCode
examp.cpp
#include <clickhouse/client.h>
using namespace clickhouse;
int main (){
/// Initialize client connection.
Client client(ClientOptions().SetHost("localhost"));

/// Create a table.
client.Execute("CREATE TABLE IF NOT EXISTS test.ips (id UInt64, v4 IPv4, v6 IPv6) ENGINE = Memory");

/// Insert some values.
try {
Block block;

auto id = std::make_shared<ColumnUInt64>();
id->Append(1);
id->Append(7);

auto name = std::make_shared<ColumnString>();
name->Append("one");
name->Append("seven");

block.AppendColumn("test id"  , id);
block.AppendColumn("name", name);

client.Insert("test.numbers", block);

} catch (std::exception &e) {
printf("Error: %s\n", e.what());
}

return 0 ;
}
i am run this -> g++ -std=c++17 examp.cpp

the error i got :
/usr/bin/ld: /tmp/ccgnim45.o: in function main': examp.cpp:(.text+0x1bf): undefined reference to clickhouse::Client::Client(clickhouse::ClientOptions const&)'
/usr/bin/ld: examp.cpp:(.text+0x202): undefined reference to clickhouse::Query::Query(char const*)' /usr/bin/ld: examp.cpp:(.text+0x21b): undefined reference to clickhouse::Client::Execute(clickhouse::Query const&)'
/usr/bin/ld: examp.cpp:(.text+0x22a): undefined reference to clickhouse::Query::~Query()' /usr/bin/ld: examp.cpp:(.text+0x239): undefined reference to clickhouse::Block::Block()'
/usr/bin/ld: examp.cpp:(.text+0x277): undefined reference to clickhouse::ColumnVector<unsigned long>::Append(unsigned long const&)' /usr/bin/ld: examp.cpp:(.text+0x2a6): undefined reference to clickhouse::ColumnVector::Append(unsigned long const&)'
/usr/bin/ld: examp.cpp:(.text+0x2f9): undefined reference to clickhouse::ColumnString::Append(std::basic_string_view<char, std::char_traits<char> >)' /usr/bin/ld: examp.cpp:(.text+0x33d): undefined reference to clickhouse::ColumnString::Append(std::basic_string_view<char, std::char_traits >)'
/usr/bin/ld: examp.cpp:(.text+0x3a2): undefined reference to clickhouse::Block::AppendColumn(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<clickhouse::Column> const&)' /usr/bin/ld: examp.cpp:(.text+0x434): undefined reference to clickhouse::Block::AppendColumn(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::shared_ptrclickhouse::Column const&)'
/usr/bin/ld: examp.cpp:(.text+0x4ad): undefined reference to clickhouse::Client::Insert(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, clickhouse::Block const&)' /usr/bin/ld: examp.cpp:(.text+0x4f8): undefined reference to clickhouse::Block::~Block()'
/usr/bin/ld: examp.cpp:(.text+0x50c): undefined reference to clickhouse::Client::~Client()' /usr/bin/ld: examp.cpp:(.text+0x584): undefined reference to clickhouse::Query::~Query()'
/usr/bin/ld: examp.cpp:(.text+0x69b): undefined reference to clickhouse::Block::~Block()' /usr/bin/ld: examp.cpp:(.text+0x720): undefined reference to clickhouse::Client::~Client()'
/usr/bin/ld: /tmp/ccgnim45.o: in function __static_initialization_and_destruction_0(int, int)': examp.cpp:(.text+0x769): undefined reference to clickhouse::NetrworkInitializer::NetrworkInitializer()'
/usr/bin/ld: /tmp/ccgnim45.o: in function void __gnu_cxx::new_allocator<clickhouse::ColumnVector<unsigned long> >::construct<clickhouse::ColumnVector<unsigned long>>(clickhouse::ColumnVector<unsigned long>*)': examp.cpp:(.text._ZN9__gnu_cxx13new_allocatorIN10clickhouse12ColumnVectorImEEE9constructIS3_JEEEvPT_DpOT0_[_ZN9__gnu_cxx13new_allocatorIN10clickhouse12ColumnVectorImEEE9constructIS3_JEEEvPT_DpOT0_]+0x31): undefined reference to clickhouse::ColumnVector::ColumnVector()'
/usr/bin/ld: /tmp/ccgnim45.o: in function void __gnu_cxx::new_allocator<clickhouse::ColumnString>::construct<clickhouse::ColumnString>(clickhouse::ColumnString*)': examp.cpp:(.text._ZN9__gnu_cxx13new_allocatorIN10clickhouse12ColumnStringEE9constructIS2_JEEEvPT_DpOT0_[_ZN9__gnu_cxx13new_allocatorIN10clickhouse12ColumnStringEE9constructIS2_JEEEvPT_DpOT0_]+0x31): undefined reference to clickhouse::ColumnString::ColumnString()'
collect2: error: ld returned 1 exit status

help pls

Compilation on Windows?

Hi, my application is cross platform, however the ck driver is unable to compile on windows platform, I got this error:
clickhouse\types\types.h(259): error C4235: nonstandard extension used: '__int128' keyword not supported on this architecture

Execute throws exception, but command line client accepts same query.

I am executing a query, client.Execute(q_create); where,

        clickhouse::Query q_create("create table if not exists crd_test.id_bbgid_map ("
                                   "id_key String, id_val String"
                                   ") engine = MergeTree order by (\"id_key\", \"id_val\");");

Or, without the escaped quotes:

create table if not exists crd_test.id_bbgid_map (id_key String, id_val String) engine = MergeTree order by ("id_key", "id_val");

This query works in the clickhouse-client but fails when executed through the clickhouse-cpp client. The error is:

terminate called after throwing an instance of 'clickhouse::ServerException'
  what():  DB::Exception: Syntax error: failed at position 68 (end of query): ;. Expected one of: identifier with optional parameters, DEFAULT, MATERIALIZED, ALIAS, COMMENT, CODEC, function, token, identifier, Dot, nested table, identifier with parameters

Is there something wrong or unsupported with that query?

Fixes for compile and work in Windows (with mingw)

  1. need add definition _WIN32_WINNT for your window version (at leat 0x0600, or will not have WSAPoll)
  2. might need define WIN32_LEAN_AND_MEAN (not affected here, but might affect other places)
  3. get errno in the Windows socket world should use WSAGetLastError
  4. connect in non blocking mode will always return -1 and set errno WSAEWOULDBLOCK. I commented the SetNonBlock before the connects.
  5. a bug in gcc8 (at least fixed in gcc10), in the declaration of WSAPoll, a const void * signature missed const, neet add const_cast for fix

[Q] Is there support for MSVC?

clickhouse-cpp located in 3rd folder.

There's my cmake file:

cmake_minimum_required(VERSION 3.16)
project(clickhouse_bench)

set(CMAKE_CXX_STANDARD 17)

add_subdirectory(3rd/clickhouse-cpp)

include_directories(3rd/clickhouse-cpp)


add_executable(clickhouse_bench main.cpp)
target_link_libraries(clickhouse_bench clickhouse-cpp-lib)

clickhouse-cpp-lib is compiled successfully, but clickhouse_bech isn't compiled because nmake can't find clickhouse-cpp-lib library. What's the problem?

UPD: set(CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS ON) solved the problem, but now while starting the program Windows can't find clickhouse-cpp-lib.dll

UPD2: I put clickhouse-cpp-lib.dll into the folder where executable file is located. At least it starts, but now it fails on creating client

UPD3: I removed SetNonBlock before connect (line). Thanks, @victor-zou (#80, p. 4). Now selecting and inserting data works fine!

Unable to link CK again

Because the link socket is not closed, the system FD is used up, and the program can no longer link to CK。

Solve after adding close (FD):

`SOCKET SocketConnect(const NetworkAddress& addr) {
int last_err = 0;
for (auto res = addr.Info(); res != nullptr; res = res->ai_next) {
SOCKET s(socket(res->ai_family, res->ai_socktype, res->ai_protocol));

    if (s == -1) {
        continue;
    }

    SetNonBlock(s, true);

    if (connect(s, res->ai_addr, (int)res->ai_addrlen) != 0) {
        int err = errno;
        if (err == EINPROGRESS || err == EAGAIN || err == EWOULDBLOCK) {
            pollfd fd;
            fd.fd = s;
            fd.events = POLLOUT;
            fd.revents = 0;
            ssize_t rval = Poll(&fd, 1, 5000);

            if (rval == -1) {
				close(s);
                throw std::system_error(errno, std::system_category(), "fail to connect");
            }
            if (rval > 0) {
                socklen_t len = sizeof(err);
                getsockopt(s, SOL_SOCKET, SO_ERROR, (char*)&err, &len);

                if (!err) {
                    SetNonBlock(s, false);
                    return s;
                }
               last_err = err;
			   close(s);
            }
        }
    } else {
        SetNonBlock(s, false);
        return s;
    }
}
if (last_err > 0) {
    throw std::system_error(last_err, std::system_category(), "fail to connect");
}
throw std::system_error(
    errno, std::system_category(), "fail to connect"
);

}`

relative import error

I did

$ git clone https://github.com/abseil/abseil-cpp.git (latest commit 1415b5936a2ac2f084850b09057e05fb5798b2f1)
$ cmake ..
$ make
$ sudo make install

Then I tried to build this example

                 from /usr/local/include/clickhouse/block.h:3,
                 from /usr/local/include/clickhouse/query.h:3,
                 from /usr/local/include/clickhouse/client.h:3,
                 from /home/guyos/Documents/analytics/main.cpp:1:
/usr/local/include/clickhouse/types/types.h:3:10: fatal error: absl/numeric/int128.h: No such file or directory
    3 | #include "absl/numeric/int128.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~

Seems like absl should not be explicitly included to types.h but be linked with some function instead. Relative import error

Nested tables support

If I have simple table like this:

CREATE TABLE test.nested
(
account UInt64,
Orders Nested(
id UInt64,
order_id String)
)
ENGINE = MergeTree
ORDER BY account

account │ UInt64
Orders.id │ Array(UInt64)
Orders.order_id │ Array(String)

How I can insert data into this table using clickhouse-cpp API? I’m working with Orders.id and Orders.order_id columns as arrays but when method block.AppendColumn called, I’m getting exception:
all columns in block must have same count of rows.

Is there any existing workaround or plans to extend API for nested tables support?

Simple COUNT example

I'm trying to determine if the table is empty. SInce all select methods return void, the only way I can think of is this:

//first attempt
int items = 0;
client.Select("SELECT id, name FROM test.numbers", [&items] (const Block& block)
              {
                  items++;
              }
);
std::cout << items << std::endl;

//second attempt
client.Select("SELECT id, name FROM test.numbers", [] (const Block& block)
                  {
                      std::cout << "block count: " << block.GetRowCount() << std::endl;
                  }
    );

But with 16 items in the table (select count(*) from test.numbers returns 16), the output is:

items: 10
block count: 0
block count: 2
block count: 2
block count: 2
block count: 2
block count: 2
block count: 2
block count: 2
block count: 2
block count: 0

What seems to work is:

int items = 0;
client.Select("SELECT id, name FROM test.numbers", [&items] (const Block& block)
              {
                  for (size_t i = 0; i < block.GetRowCount(); ++i) {
                        ++items;
                    }
              }
);

but surely there is a simpler solution. How can it be determined if a table is empty and what would a COUNT example look like?

Exception on inserting LowCardinality(Nullable(String))

Try to insert data to field LowCardinality(Nullable(String)),
exception throw:
LowCardinality(Nullable) not supported

Questions:

  1. Does the clickhouse-cpp support LowCardinality(Nullable(String))
  2. Any unit test or example on LowCardinality() type

Thanks a lot!

How to correct clean Block class variable?

Based on main page example :

{
    Block block;
    auto id = std::make_shared<ColumnUInt64>();
    id->Append(1);
    id->Append(7);
    auto name = std::make_shared<ColumnString>();
    name->Append("one");
    name->Append("seven");
    block.AppendColumn("id"  , id);
    block.AppendColumn("name", name);
    client.Insert("test.numbers", block);
}

I'm trying to clean variable 'block' and cannot execute it. Due to documentation delete (memset) command is not supporting and occurring next error :

free(): invalid size

When I'm using it in cycle (instead 1, 7, "one", "seven" I'm using variables) I'm getting duplicate values in my db. Could you please advise how to correct clean it?

Using of clickhouse-cpp with C++11 codebase

Hi Clickhouse Team.

I'v been using some two-year-old snapshot of the clickhouse-cpp library for the last couple of years and it worked just fine with my C++11 codebase. However, now I need to use DateTime64 columns, that's why I'm trying to upgrade to the latest version. Ooops... Now it is strictly C++17, so I cannot use it from my C++11 codebase. Migrating the huge codebase to C++17 would be an immense task (although inevitable in the future). Moreover, C++11 is the only 'native' standard for my RHEL 7.5 (yes, it is awful).

What I'v tried:

  1. Compile the clickhouse-cpp with RHEL's devtoolset-7 and then link the so to my code. No success (obviously) as the headers rely in C++17 features (such as string_view) - so my version of gcc cannot handle it.
  2. Prepare a tiny wrapper of my code encapsulating all the communications with clickhouse-cpp, and build it as a static library with RHEL's devtoolset-7, then link this library + clickhouse-cpp so to my main executable. RHEL/gcc promise the ABI compatibility. Successfully linked, but crashes due to memory corruption during execution, somewhere in the CH connection establishment procedure.

There are other crazy options like reading with Python or native client in a dedicated process and then passing the results to C++ part, which I would like to avoid as much as possible.

Any advice would be appreciated with respect to possible ways to use clickhouse-cpp from C++11.

BR, Andrey.

Selecting elements of nested array

Given table:

CREATE TABLE IF NOT EXISTS maps.multiarray
(
    `arr` Array(Array(UInt64))
)
ENGINE = Memory;

with single row:

INSERT INTO maps.multiarray VALUES ([[0,1,2,3,4,5], [100, 200], [10,20, 50, 70]]);

I am trying to run select, using code similar to what is written at test:

string SELECT_QUERY = "SELECT arr FROM maps.multiarray";
client.Select(SELECT_QUERY, [](const Block& block)
{
    if (block.GetRowCount() == 0) {
         return;
    }

    for (size_t c = 0; c < block.GetRowCount(); ++c) {
        auto col = block[0]->As<ColumnArray>()->GetAsColumn(c);
	cout << col->Size() << endl;
	cout << "[";
	for (size_t i = 0; i < col->Size(); ++i) {
	      auto col2 = col->As<ColumnArray>()->GetAsColumn(i);
              for (size_t j = 0; j < col2->Size(); ++j) {
		     cout << (int)(*col2->As<ColumnUInt64>())[j];
                     if (j + 1 != col2->Size()) {
			cout << " ";
                     }
               }
       }
       std::cout << "]" << std::endl;
    }
});

And as a result got only this kind of output:

1
[0]

While I was expecting to have something like that:

5
[0 1 2 3 4 5]
2
[100 200]
4
[10 20 50 70]

I've tried to access exact same table using client in other languages and it seems to work as expected:

from clickhouse_driver import connect
conn = connect('clickhouse://localhost')
cursor = conn.cursor()
cursor.execute('SELECT * FROM maps.multiarray')
cursor.fetchall()
[([[0, 1, 2, 3, 4, 5], [100, 200], [10, 20, 50, 70]],)]

I've also saw similar issue within alternative(?) implementation - here.

I can sort of work-around it by issuing first SELECT length(arr) FROM maps.multiarray with subsequent series of calls select arrayElement(arr, idx) from maps.multiarray;

But it is not clear is it limitation of existing client or I just miss something important in my implementation?

exception : fail to connect: Operation completed successfully

Hi @Enmk:
I'm happy to use clickhouse-cpp, and I successfully compiled the library under Qt Creator5.9.8 software, but when I run the test program I have a problem, and I need your help. It always reports an error: "exception : fail to connect: Operation completed successfully", Looking forward to your reply, Thank you very much!

Cannot access array of tuples generated by groupArray()

It appears that there's an issue in accessing data returned by a query containing groupArray().

I have a table which contains a nested array named "eventz". The following is part of the create statement as returned by the "show create table" command:

eventz.EventCode` Array(UInt16) CODEC(LZ4HC(9)), `eventz.EventCnt` Array(UInt32) CODEC(LZ4HC(9)))

Trying the following query from the command-line clickhouse-client works just fine:

SELECT groupArray((`eventz.EventCode`, `eventz.EventCnt`)) from db1.tupletest array join eventz group by cl limit 10;

It returns:

┌─groupArray(tuple(eventz.EventCode, eventz.EventCnt))───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ [(123,10),(456,100),(789,1000),(123,10),(456,100),(789,1000),(123,10),(456,100),(789,1000),(123,10),(456,100),(789,1000)]                                                                                                                                  │
│ [(123,10),(456,100),(789,1000),(123,10),(456,100),(789,1000)]         

...

However, from a C++ application using the Clickhouse C++ client library, I get a NULL pointer when I try calling GetAsColumn(). Consider the following code:

client.Select("SELECT groupArray((`eventz.EventCode`, `eventz.EventCnt`)) from db1.tupletest array join eventz group by cl limit 10", [] (const Block& block) mutable
    {
        size_t rows = block.GetRowCount();
        for (int i=0; i < rows; i++){
            std::cout << block[0]->As<ColumnArray>() << "\n";
            std::cout << block[0]->As<ColumnArray>()->Size() << "\n";
            std::cout << block[0]->As<ColumnArray>()->GetAsColumn(i);

...

Returns the following:

0x105d6c0
10
0

...

As you can see, GetAsColumn returned 0 (NULL) instead of a correct pointer.

Crash when running queries with nested arrays

Hello,

Having this table:

CREATE TABLE integrity_test_SUITE.nested_array(timestamp DateTime, arr Array(Array(Array(String))), arr_null Array(Array(Array(Nullable(String)))) ) ENGINE = MergeTree PARTITION BY toYYYYMM(timestamp) ORDER BY (timestamp)

Insert one row:

INSERT INTO integrity_test_SUITE.nested_array(timestamp) VALUES (1590418220)

Then try to select data using clickhouse-cpp:

 /// Initialize client connection.
    Client client(ClientOptions().SetHost("localhost"));

    /// Select values inserted in the previous step.
    client.Select("SELECT arr FROM integrity_test_SUITE.nested_array", [] (const Block& block) {

    });

The driver is crashing.

AddAsColumn inefficiency

Hi,

This check, on every addition of an array to a column, is very expensive for enums:

//---- columns/array.cpp:
void ColumnArray::AppendAsColumn(ColumnRef array) {
    if (!data_->Type()->IsEqual(array->Type())) {
        throw std::runtime_error(...

//---- types/types.h:
bool IsEqual(const TypeRef& other) const { return this->GetName() == other->GetName(); }
                                                   -----^

EnumType::GetName() is walking a std::map, on every call, to generate the full enum description string: "A"=1, "B"=2, ...
I believe a 64 bit hash of whatever type info would make this comparison properly fast.

Thanks,
Yakov

Appending an Array of Tuples to a Column fails.

This could be related to #32.

When I try to fill a Column that has type Array(Tuple(UInt64, String)) with AppendAsColumn then no actual data is added and the insert fails.

#include <clickhouse/client.h>

using namespace std;
using namespace clickhouse;

int main()
{

    Block block;

    Client client(ClientOptions().SetHost("localhost"));
    client.Execute("CREATE TABLE mynumbers (tupleArray Array(Tuple(UInt64, String))) ENGINE = Memory");

    auto tupleArray = make_shared<ColumnArray>(
                make_shared<ColumnTuple>(
                    vector<ColumnRef>({
                                          make_shared<ColumnUInt64>(),
                                          make_shared<ColumnString>()
                                      })
                    )
                );
    auto tupleArrayEntry1 = make_shared<ColumnTuple>(
                vector<ColumnRef>({
                                      make_shared<ColumnUInt64>(),
                                      make_shared<ColumnString>()
                                  })
                );
    auto tupleArrayEntry2 = make_shared<ColumnTuple>(
                vector<ColumnRef>({
                                      make_shared<ColumnUInt64>(),
                                      make_shared<ColumnString>()
                                  })
                );

    (*tupleArrayEntry1)[0]->As<ColumnUInt64>()->Append(2);
    (*tupleArrayEntry1)[1]->As<ColumnString>()->Append("2");
    (*tupleArrayEntry2)[0]->As<ColumnUInt64>()->Append(10);
    (*tupleArrayEntry2)[1]->As<ColumnString>()->Append("10");
    (*tupleArrayEntry2)[0]->As<ColumnUInt64>()->Append(11);
    (*tupleArrayEntry2)[1]->As<ColumnString>()->Append("11");

    tupleArray->AppendAsColumn(tupleArrayEntry1);
    tupleArray->AppendAsColumn(tupleArrayEntry2);

    block.AppendColumn("tupleArray", tupleArray);

    client.Insert("mynumbers", block);

    return 0;
}

Why is the clickhouse-cpp much slower than clickhouse-client in insertion

I use clickhouse-client to batch insert 1 million rows, the inserting speed can reach to almost 140 thousands rows per sencond(although it is slow also), but when I use clickhouse-cpp in my program, I tested the inserting speed, It is only 10 thousands rows per second, I have modified clickhouse.cpp file to support continuous insertion without declaring 'INSERT INTO ... ' statement, the speed has growed up to 10 thousands rows per second from the original 600 rows per second.

Fix for compilation on GCC 10.2.1 / FC32

Hi,

This is a trivial fix for a compilation failure that happens in newer GCC, probably due to internal header include cleanup.
First the error:

20/57] Building CXX object clickhouse/CMakeFiles/clickhouse-cpp-lib-static.dir/types/types.cpp.o
FAILED: clickhouse/CMakeFiles/clickhouse-cpp-lib-static.dir/types/types.cpp.o 
/usr/bin/c++   -I../. -I../contrib -O2 -pthread -Wall -Wextra -Werror -O2 -DNDEBUG   -std=gnu++17 -MD -MT clickhouse/CMakeFiles/clickhouse-cpp-lib-static.dir/types/types.cpp.o -MF clickhouse/CMakeFiles/clickhouse-cpp-lib-static.dir/types/types.cpp.o.d -o clickhouse/CMakeFiles/clickhouse-cpp-lib-static.dir/types/types.cpp.o -c ../clickhouse/types/types.cpp
../clickhouse/types/types.cpp: In constructor ‘clickhouse::DateTime64Type::DateTime64Type(size_t)’:
../clickhouse/types/types.cpp:234:20: error: ‘runtime_error’ is not a member of ‘std’
  234 |         throw std::runtime_error("DateTime64 precision is > 18");
      |                    ^~~~~~~~~~~~~

This is fixed by adding the missing header, of course:

diff --git a/clickhouse/types/types.cpp b/clickhouse/types/types.cpp
index 0eaf939..b273364 100644
--- a/clickhouse/types/types.cpp
+++ b/clickhouse/types/types.cpp
@@ -1,6 +1,7 @@
 #include "types.h"
 
-#include <assert.h>
+#include <stdexcept>
+#include <cassert> 

 namespace clickhouse {

Retrieving columns information

Please add a function to receive Protocol::Server::TableColumns packet from server by table name.
It needed to handle INSERT column types without describing them at scripting languages without strict types (PHP)

For example, PHP sends zend_long for all integer types, but ClickHouse needs to specify exact [U]Int[8,16,32,64,128] type.

If it will be released i can make type auto-conversion from zend_long (and etc) to ClickHouse type on extension side

Q how can I run the example with C++11

hi !
please tell me how can i run the example with C++11? when I run ,I meet the problem:

In file included from clickhouse/columns/column.h:4,
                 from clickhouse/block.h:3,
                 from clickhouse/query.h:3,
                 from clickhouse/client.h:3,
                 from main.cpp:1:
clickhouse/columns/../columns/itemview.h:19:27: 错误:‘string_view’不是命名空间‘std’中的一个类型名
     using DataType = std::string_view;
                           ^~~~~~~~~~~
clickhouse/columns/../columns/itemview.h:19:22: 附注:‘std::string_view’ is only available from C++17 onwards
     using DataType = std::string_view;
                      ^~~
clickhouse/columns/../columns/itemview.h:22:11: 错误:‘DataType’ does not name a type; did you mean ‘EnumType’?
     const DataType data;

Current setup with absl is dangerous

The way this library currently uses absl can mess up a lot of things. In my opinion it should be reconsidered or at least (as a temporary workaround measure) a switch must be added to CMakeLists.txt to switch off int128 support (and any absl dependency coming with it).

Scenario 1:

  1. User decides to use link with both clickhouse-cpp and absl from master branch and in their project
  2. int128.cc is compiled twice - once as part of clickhouse build, once as part of absl build.
  3. Build likely fails because the same functions will be exported twice from different translation units

Scenario 2:

  1. User decides to use link with both clickhouse-cpp and absl from lts_* branch and in their project. Let's assume neither of those are installed in the system, but are rather plugged into the project via FetchContent/ExternalProject in CMake.
  2. lts_ branch puts all code into something like absl::lts_20210324:: inline namespace. So from linker point of view all absl symbols will be there, even though from compiler point of view they will be visible in absl::.
  3. columns/numeric.cpp will then use absl from contrib.
  4. Any user code that directly or indirectly includes columns/numeric.h will probably use the other absl.
  5. This will lead to mysterious linker failures because now declaration from columns/numeric.h and definition in columns/numeric.cpp are ABI-incompatible.

My understanding is that scenario 2 will only occur if the user attempts to actually use Int128 columns. Scenario 1 may occur if the the project uses int128 somewhere else, even if it doesn't need Int128 columns. There is a way to convert scenario 1 to scenario 2 by enabling inline namespace (and giving it unique name) inside absl/base/options.h. The disadvantage is that scenario 2 will likely produce more incomprehensible diagnostic messages from the linker.

Build is broken (gcc 7.4, x64, linux)

In file included from /build/clickhouse/src/clickhouse/columns/column.h:6:0,
                 from /build/clickhouse/src/clickhouse/columns/numeric.h:3,
                 from /build/clickhouse/src/clickhouse/columns/numeric.cpp:1:
/build/clickhouse/src/clickhouse/columns/../columns/itemview.h: In instantiation of ‘clickhouse::ItemView::ItemView(clickhouse::Type::Code, const T&) [with T = __int128]’:
/build/clickhouse/src/clickhouse/columns/numeric.cpp:84:51:   required from ‘clickhouse::ItemView clickhouse::ColumnVector<T>::GetItem(size_t) const [with T = __int128; size_t = long unsigned int]’
/build/clickhouse/src/clickhouse/columns/numeric.cpp:96:16:   required from here
/build/clickhouse/src/clickhouse/columns/../columns/itemview.h:50:54: error: invalid use of void expression
         : ItemView(type, ConvertToStorageValue(value))

Reproduction steps from docker run -it --rm ubuntu:latest (using gcc 7.3 though, on 7.5 it doesn't reproduce):

apt install \
  git \
  cmake \
  g++-7=7.3.0-16ubuntu3 \
  gcc-7-base=7.3.0-16ubuntu3 \
  gcc-7=7.3.0-16ubuntu3 \
  libstdc++-7-dev=7.3.0-16ubuntu3 \
  cpp-7=7.3.0-16ubuntu3 \
  libgcc-7-dev=7.3.0-16ubuntu3 \
  libasan4=7.3.0-16ubuntu3 \
  libubsan0=7.3.0-16ubuntu3 \
  libcilkrts5=7.3.0-16ubuntu3
git -c http.sslVerify=false clone https://github.com/ClickHouse/clickhouse-cpp.git
mkdir build &&  cd build/
cmake \
  -DCMAKE_C_COMPILER=gcc-7 \
  -DCMAKE_CXX_COMPILER=g++-7 \
  -DCMAKE_CXX_STANDARD=17 \
  -DCMAKE_CXX_STANDARD_REQUIRED=ON \
  -DCMAKE_CXX_EXTENSIONS=OFF \
  ../clickhouse-cpp/
make -j4

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.