shlima / click_house Goto Github PK

If change the precision value from 10 to > 17, the test will fail, this is not correct because the precision can go up to 76 (https://clickhouse.com/docs/en/sql-reference/data-types/decimal/#parameters)

This is due to the fact that in BigDecimal when initializing Float, the maximum precision will be 17:
If initial is a Float, and the digits is larger than Float::DIG + 1, this exception is raised.
https://ruby-doc.org/stdlib-2.5.1/libdoc/bigdecimal/rdoc/BigDecimal.html#method-c-new-label-Exceptions

Can try converting the value to a string, but it is not clear whether precision works if pass a string:

Currently, the library can raise a Faraday::TimeoutError.

In the RaiseError middleware we catch Faraday::ConnectionFailed and remap it to ClickHouse::NetworkException:

However, TimeoutError isn't a descendant of ConnectionFailed:

https://github.com/lostisland/faraday/blob/ea30bd0b543882f1cf26e75ac4e46e0705fa7e68/lib/faraday/error.rb#L132

I suggest we also catch Faraday::TimeoutError (and perhaps a few more Faraday errors?) and remap it to ClickHouse errors

Hi. If I understand correctly, click_house is not supported processing migrations files likerake db:migrate with store in migration table processed migrations. Isn't it?
I have not-rails application with ActiveRecord. And I want to organize creating and processing migrations.

This isn't an issue exactly but is there a compatibility matrix for the gem with clickhouse versions ?
I plan to upgrade clickhouse from 21.1 to 21.7 and later to v22 and i couldn't find anything in the gem docs that would indicate compatibility

How I can insert nested data below?

{"id" => 1, "name" => 'one', "tests" => [{"test_id"=>1099803, "test_number"=>"35545585544", "test_number_2"=>"123456", "test_source"=>nil}, "test_id"=>1099804, "test_number"=>"1313113", "test_number_2"=>"654321", "test_source"=>nil}, {...}]}

and describe pls table structure for this. Thanks

I'd like to use Lowcardinality, but as far as I can tell that isn't currently implemented in this library.

I can contribute a patch to help implement this feature, but I wanted look for your guidance first. Do you have any recommendations for implementing this feature? It looks like the existing Array type might be a good starting place.

You only need X-ClickHouse-Summary. And that is returned with send_progress_in_http_headers = 0. I.e. as far as I can see send_progress_in_http_headers can be safely changed to 0. With send_progress_in_http_headers = 1, a relatively slow query (2 sec), and clickhouse being behind nginx, the latter says "upstream sent too big header" because clickhouse produces a lot of "progress" headers:

$ curl -sSv 'http://xx.yyy.xx.yyy:8123/?send_progress_in_http_headers=1' -d 'select ..., count(*) from ... group by ... order by ... limit 1' -X GET -u '...:...'
* Expire in 0 ms for 6 (transfer ...)
*   Trying xx.yyy.xx.yyy...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer ...)
* Connected to xx.yyy.xx.yyy (xx.yyy.xx.yyy) port 8123 (#0)
* Server auth using Basic with user '...'
> GET /?send_progress_in_http_headers=1 HTTP/1.1
> Host: xx.yyy.xx.yyy:8123
> Authorization: Basic ...
> User-Agent: curl/7.64.0
> Accept: */*
> Content-Length: 94
> Content-Type: application/x-www-form-urlencoded
> 
} [94 bytes data]
* upload completely sent off: 94 out of 94 bytes
< HTTP/1.1 200 OK
< Date: Thu, 08 Feb 2024 13:22:37 GMT
< Connection: Keep-Alive
< Content-Type: text/tab-separated-values; charset=UTF-8
< X-ClickHouse-Server-Display-Name: ...
< Transfer-Encoding: chunked
< X-ClickHouse-Query-Id: ...
< X-ClickHouse-Format: TabSeparated
< X-ClickHouse-Timezone: UTC
< Keep-Alive: timeout=3
< X-ClickHouse-Progress: {"read_rows":"1111953","read_bytes":"8895624","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"9434898","read_bytes":"75479184","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"18412695","read_bytes":"147301560","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"27260817","read_bytes":"218086536","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"35690512","read_bytes":"285524096","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"44667166","read_bytes":"357337328","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"53121310","read_bytes":"424970480","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"61969813","read_bytes":"495758504","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"70226587","read_bytes":"561812696","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"78697369","read_bytes":"629578952","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"86824087","read_bytes":"694592696","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"94819987","read_bytes":"758559896","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Summary: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< 
{ [11 bytes data]
* Connection #0 to host xx.yyy.xx.yyy left intact
165	1

Always i get this error on my requests, does anyone know what it could be?

Hi.
Noticed that now clickhouse supports Ipv4/Ipv6 https://clickhouse.tech/docs/en/sql-reference/data-types/domains/ipv4/.

I think would be nice to add that here.

Btw I can implement it in free time

Problem: Creating a column with a codec type throws an error.

Migration Repro:

    ClickHouse.connection.create_table('visits', engine: 'MergeTree ORDER BY date') do |t|
      t.Date     :date, compression_codec: 'CODEC(DoubleDelta, ZSTD(1))'
    end

Error:

Caused by:
NoMethodError: undefined method `compression_codec=' for #<ClickHouse::Definition::Column:0x0000000105b5abe8 @type="Date", @name=:date, @extensions=[{:compression_codec=>"CODEC(DoubleDelta, ZSTD(1))"}]> (NoMethodError)

        params.each { |k, v| public_send("#{k}=", v) }
                             ^^^^^^^^^^^

In addition to compression_codec, trying to use: compression:, codec:, and codec_expression: all failed.

I was able to work around the issue by using raw SQL.

    ClickHouse.connection.create_table('visits', engine: 'MergeTree ORDER BY date') do |t|
      t << "date Date CODEC(DoubleDelta, ZSTD(1))"
    end

It'd be great feature to have this as supported for DSL statements.

By default, if we check the available types we can see that Nullable(Float32) and Nullable(Float64) are casted to nullable integers instead of nullable floats:

> ClickHouse.types
...
"Nullable(Float32)"=>#<ClickHouse::Type::NullableType:0x00007fa730bb1aa0 @subtype=#<ClickHouse::Type::IntegerType:0x00007fa730bb1ac8>>,
...
"Nullable(Float64)"=>#<ClickHouse::Type::NullableType:0x00007fa730bb19d8 @subtype=#<ClickHouse::Type::IntegerType:0x00007fa730bb1a00>>,
...

There is a workaround to this issue. We can execute (if we are in Rails we can do it in an initializer):

ClickHouse.add_type('Nullable(Float32)', ClickHouse::Type::NullableType.new(ClickHouse::Type::FloatType.new))
ClickHouse.add_type('Nullable(Float64)', ClickHouse::Type::NullableType.new(ClickHouse::Type::FloatType.new))

Hey @shlima!

During local testing, I'm noticing inconsistent data returned by the .select_all and .select_one methods. Specifically, when there is X amount of data inserted into the table (in my local dev I only have ~2.5k rows).

Here's an example with .select_one:

As you can see, the first example returns an opening bracket cast as a string. The second example returns what I'd usually expect.

Here's an example with .select_all:

Here, the first example is returning raw JSON, this is not expected.

I have an inclination that Faraday is struggling to deserialize the JSON response (because of the response size) which causes the above issues. I don't have a proposed fix but I'm wondering if you've encountered this before and have suggestions on how to move forward?

I am using Ruby on Rails 7 with multiple databases. I got this error.

1. I tried with:adapter: net_http but it isn't recognized.
2. Removing this line it doesn't raise exception but it doesn't connect to the secondary database.
   How do I say to connect to the secondary database (ams) instead of primary ?

app/models/ams_record.rb
connects_to database: { writing: :ams, reading: :ams }

/Users/name/.rvm/gems/ruby-3.1.2/gems/activerecord-7.0.8/lib/active_record/connection_adapters/abstract/connection_handler.rb:261:in `resolve_pool_config':

database configuration does not specify adapter (ActiveRecord::AdapterNotSpecified)

database.yml

default_ams: &default_ams
  timeout: 60
  open_timeout: 3

development:
  primary:
    <<: *default
    database: xxx_development
    password: password
    host: localhost
    port: 5432
  ams:
    <<: *default_ams
    database: ams_development
    username: default
    password: XXXX
    url: https://xxx.us-east-1.aws.clickhouse.cloud:8443

app/models/click_house_record.rb

class ClickHouseRecord < ActiveRecord::Base
  self.abstract_class = true

  class << self
    def agent
      ClickHouse.connection
    end

    def insert(*argv, &block)
      agent.insert(table_name, *argv, &block)
    end

    def select_one
      agent.select_one(current_scope.to_sql)
    end

    def select_value
      agent.select_value(current_scope.to_sql)
    end

    def select_all
      agent.select_all(current_scope.to_sql)
    end

    def explain
      agent.explain(current_scope.to_sql)
    end
  end
end

app/models/ams_record.rb

class AmsRecord < ClickHouseRecord
  self.abstract_class = true

  connects_to database: { writing: :ams, reading: :ams }

end

app/models/ams_traffic.rb

class AmsTraffic < AmsRecord
   # This is my table on Click House
end

Does this gem support Mutation-based updates using ALTER…UPDATE in case of irregular and non-frequent changes ?

see https://clickhouse.com/docs/en/sql-reference/statements/alter/update/

I can see that insert operations uses to_json which do not support non unicode data. But every other part of the system support binary data. Can we do something so that we do not need to replace invalid utf-8 chars using scrub?

DB support binary data
https://clickhouse.com/docs/en/sql-reference/data-types/string/#encodings

FORMAT supports binary data
http://www.devdoc.net/database/ClickhouseDocs_19.4.1.3-docs/interfaces/formats/#jsoneachrow

Gem uses to_json to convert hash(representing values of a row) to generate the query.

But to_json does not work with binary data. It expects data to have valid unicode characters only.

There are many other formats also but I am not sure what format will solve this issue: https://clickhouse.com/docs/en/interfaces/formats/#jsoneachrow

[65] pry(main)> str = "\xc2"
=> "\xC2"
[66] pry(main)> Model.insert { |buffer| buffer << {:col1 => 1, :col2 => str} }
JSON::GeneratorError: partial character in source, but hit end
from /home/user/.rvm/gems/ruby-2.7.4/gems/activesupport-5.2.2.1/lib/active_support/core_ext/object/json.rb:38:in `to_json'

[69] pry(main)> str.encoding
=> #<Encoding:UTF-8>
[70] pry(main)> str.valid_encoding?
=> false

I need to check how other db drivers support binary data.

I will be interested in contributing if needed and required.

Do you have ideas on how can we achieve this?

create table dt_type_test 
( 
  a DateTime
, b Nullable(DateTime)
, c DateTime64(3)
, d Nullable(DateTime64(3))
, e DateTime64(3, 'UTC')
, f Nullable(DateTime64(3, 'UTC'))
, g Decimal(18,2) 
) 
engine=Log ;

insert into dt_type_test values (now(), now(), now(), now(), now(), now(), 5.99);

[14] pry(main)> rec = ClickHouse.connection.select_one("select * from dt_type_test limit 1")
SQL (Total: 10MS, CH: 4MS) select * from dt_type_test limit 1;
                                                              Read: 1 rows, 51.0B. Written: 0 rows, 0B
=> {"a"=>Fri, 01 Jul 2022 17:48:53 +0000,
 "b"=>Fri, 01 Jul 2022 17:48:53 +0000,
 "c"=>"2022-07-01 17:48:53.000",
 "d"=>"2022-07-01 17:48:53.000",
 "e"=>"2022-07-01 23:48:53.000",
 "f"=>"2022-07-01 23:48:53.000",
 "g"=>5.99}
[15] pry(main)> rec["g"].class
=> Float

My expectation is that all of a, b, c, d, e, and f would be of type DateTime, and g would be of type BigDecimal. Instead we see c-f as strings, and g as a Float.

Poking around, I found at least two problems.

One is that DateTime64(%d) is not a registered type. That's an easy fix.

The bigger problem is that parameterized types with %d placeholders don't seem to work at all. For example, it appears that ClickHouse::Response::ResultSet#extract_type_info parses DateTime64(3, 'UTC') as DateTime64(%s, %s) rather than DateTtime64(%d, %s). And since DateTime64(%s, %s) isn't a registered type, it falls back to treating it as a string. Similarly, Decimal(%d, %d) is a registered type, but gets parsed as Decimal(%s, %s) which doesn't match.

In fact, extract_type_info makes no attempt to insert %d placeholders, always using %s, which is inconsistent with how the type registry works. So if I'm understanding this right, no parameterized type with a %d placeholder will ever work.

Currently gem doesn't support params in SELECT queries AFAIK. I mean params like SELECT * FROM t WHERE id = {id:Int64}

Do you consider adding this? I can provide PR for that

I already use fork with this feature added. But I did not update it for a year, so I can't provide PR right away 😅 You can check here, what it looks like

I'm wanting to use Lowcardinality type , from what I can tell, this isn't supported by this library today, is that correct?

I'd be willing to create a patch, as I require this functionality myself. Do you have any recommendations before I get started on that?
'

Hello, firstly thank you for the gem.

I have a parsing problem with inserting BigDecimal values to a table. The problem can be reproduced both for inserting BigDecimal type value and for manual casting:

irb(main):003:0> c = ClickHouse.connection
irb(main):004:0> c.execute "CREATE TABLE test(id Decimal64(2)) ENGINE TinyLog"
irb(main):004:0> c.insert("test", columns: [:id], values: [[1.0]])
=> true

irb(main):006:0> c.insert("test", columns: [:id], values: [[BigDecimal("1.0")]])
ClickHouse::DbException ([400] Code: 27, e.displayText() = DB::ParsingException: Cannot parse input: expected ',' before: '"0.1e1"}'{} (version 21.1.2.15 (official build)))
irb(main):007:0> c.insert("test", columns: [:id], values: [[ClickHouse::Type::DecimalType.new.serialize(1.0)]])
ClickHouse::DbException ([400] Code: 27, e.displayText() = DB::ParsingException: Cannot parse input: expected ',' before: '"0.1e1"}'{} (version 21.1.2.15 (official build)))

As I understand this is due to 2 reasons:

1. Parsed BigDecimal value in Ruby represents as JSON String
2. Clichouse requires not to wrap in quotes numeric values while inserting to a table (using JSONEachRow format)

Do you have any ideas to solve this?

Hello I have this error when I run 200 workers in one time,
"ActiveRecord::StatementInvalid: Errno::EMFILE: Failed to open TCP connection to 192.168.0.1:8123 (Too many open files - socket(2) for"

How I understand the problem is because I didn't close my old connection, can you tell me please if your gem has any method for closing connection with the socket?

Thank you.

Using "tuples" insert notation I noticed that if I attempt to write nil to it like this

ClickHouse.connection.insert('devices', columns: %i(device_id kind model vendor os_name os_version referer updated_at created_at)) do |buf|
  buf << ['608253467_null', 'mobile', 'ALCATEL ONE TOUCH 993D', 'samsung', 'android', '19', nil, DateTime.parse('Sat, 13 Feb 2016 04:32:48').strftime('%Y-%m-%d %H:%M:%S'), DateTime.parse('Sat, 13 Feb 2016 03:39:07').strftime('%Y-%m-%d %H:%M:%S')]
end

(notice the nil value) raises exception ClickHouse::DbException ([400] Code: 26, e.displayText() = DB::Exception: Cannot parse JSON string: expected opening quote: (while read the value of key referer): (at row 1))

I assume that it would be better to silently convert value to its default value by column type on the fly ('' for String, 0 for Int and so on).

It is also surprising that time have to be specially formatted for clickhouse by hand as well.

What do you think? Is it made to give more controll over data to user to avoid server versions compatibility issues?

Is there a way to expose the exceptions and allow callers to capture and handle appropriately? Right now running a select_all on a rogue query quietly fails saying it got 0 results, as opposed to fail because of a logic error:

Simple steps to reproduce from the rails console:

>> # This should cause an exception as the function ccccccc() doesn't exist
>> ClickHouse.connection.select_all("SELECT ccccccc() from cccccccxxxxxx")

SQL (Total: 104MS, CH: 0MS)� SELECT ccccccc() from cccccccxxxxxx;�
Read: 0 rows, 0B. Written: 0 rows, 0B
[]

>> ClickHouse.connection.select_all("SELECT now()")

SQL (Total: 171MS, CH: 1MS)� SELECT now();�
Read: 1 rows, 1.0B. Written: 0 rows, 0B
[{"now()"=>2024-04-04 17:25:04 -0400}]

The exception is there in the faraday.body, but when converting to a ResultSet in the ClickHouse::Response::Factory.response() method that exception is never checked.

As I understand from ClickHouse logs, your driver inserts with JSONEachRow format, but the previous unmaintainable one are inserting with CSVWithNames format. Is there a way to make this configurable or no?

I want to checkout to use your client, but need to modify bunch of things for that

Hi @shlima
There is one question.
When a query is made via (select_all, select_value or select_one) method, get method is called:

def select_all(sql)
  response = get(body: Util::Statement.format(sql, 'JSON'))
  Response::Factory[response]
end

When logging such requests at the INFO level, we get this result:

[2022-06-06T12:06:25.936896] #044c48a9: SQL (Total: 32779MS, CH: 3MS) ;
[2022-06-06T12:06:25.937438] #044c48a9: Read: 4 rows, 4.2KiB. Written: 0 rows, 0B
[2022-06-06T12:08:39.955952] #2610758b: SQL (Total: 117118MS, CH: 6MS) ;
[2022-06-06T12:08:39.956922] #2610758b: Read: 4 rows, 4.2KiB. Written: 0 rows, 0B

If you look at the logger code, you can see that we are waiting for the query output to the console (look query method):

def on_complete(env)
  summary = extract_summary(env.response_headers)
  logger.info("\e[1m�[35mSQL (#{duration_stats_log(env.body)})�\e[0m #{query(env)};�")
  logger.debug(body) if body && !query_in_body?(env)
  logger.info("\e[1m�[36mRead: #{summary.fetch(:read_rows)} rows, #{summary.fetch(:read_bytes)}. Written: #{summary.fetch(:written_rows)} rows, #{summary.fetch(:written_bytes)}\e[0m")
end

According to the code, we must take the value from @Body, but it will only be if we have a DEBUG logging level:

def call(environment)
  @starting = timestamp
  @body = environment.body if log_body?
  @app.call(environment).on_complete(&method(:on_complete))
end

private

def log_body?
  logger.level == Logger::DEBUG
end

In this case, if you make a request through execute method, a POST request will be made and in this case, the query will get into the logger (INFO).

[2022-06-06T10:03:50.059963] #88d8d7f8: SQL (Total: 25MS) SELECT * FROM test;
[2022-06-06T10:03:50.060545] #88d8d7f8: Read: 3 rows, 3.1KiB. Written: 0 rows, 0B

Should it work like this?

I know my setting are correct but I'm unable to ping. Do I need to set send_progress_in_http_headers to a certain value to get this to work?

2.3.8 :044 > require 'click_house'
=> false
2.3.8 :045 > ClickHouse.config do |config|
2.3.8 :046 > config.logger = Logger.new(STDOUT)
2.3.8 :047?> config.adapter = :net_http
2.3.8 :048?> config.database = 'default'
2.3.8 :049?> config.timeout = 60
2.3.8 :050?> config.open_timeout = 3
2.3.8 :051?> config.ssl_verify = false
2.3.8 :052?> config.headers = {}
2.3.8 :053?>
2.3.8 :054 > # or provide connection options separately
2.3.8 :055 > config.scheme = 'http'
2.3.8 :056?> config.host = 'localhost'
2.3.8 :057?> config.port = '8123'
2.3.8 :058?>
2.3.8 :062?> end
=> #<ClickHouse::Config:0x00007fbeb02571c0 @adapter=:net_http, @url=nil, @scheme="http", @host="localhost", @PORT="8123", @logger=#<Logger:0x00007fbeb0256798 @progname=nil, @Level=0, @default_formatter=#<Logger::Formatter:0x00007fbeb0256770 @datetime_format=nil>, @Formatter=nil, @logdev=#<Logger::LogDevice:0x00007fbeb0256720 @shift_size=nil, @shift_age=nil, @filename=nil, @dev=#<IO:>, @mon_owner=nil, @mon_count=0, @mon_mutex=#Thread::Mutex:0x00007fbeb0256568>>, @database="", @username="", @password="", @timeout=60, @open_timeout=3, @ssl_verify=false, @headers={}>
2.3.8 :063 > ClickHouse.connection.ping
NoMethodError: undefined method compact' for {:send_progress_in_http_headers=>nil, :database=>nil}:Hash from /Users/slee/.rvm/gems/ruby-2.3.8/gems/click_house-1.3.6/lib/click_house/connection.rb:49:in compose'
from /Users/slee/.rvm/gems/ruby-2.3.8/gems/click_house-1.3.6/lib/click_house/connection.rb:25:in get' from /Users/slee/.rvm/gems/ruby-2.3.8/gems/click_house-1.3.6/lib/click_house/extend/connection_healthy.rb:8:in ping'
from (irb):63
from /Users/slee/.rvm/rubies/ruby-2.3.8/bin/irb:11:in `

ClickHouse.rb, is %d

ResultSet.rb , the output is %s

Therefore the BigDecimal type will be treated as undefined_type

We sometimes fetch a lot of data from Clickhouse and the default Faraday JSON parser is quite slow. Patching the JSON library with the Oj gem would help, but I can't do that in my project.

I can think of two different solutions to that problem:

1. Elasticsearch gem uses MultiJson which automatically picks the most performant JSON library: https://github.com/elastic/elastic-transport-ruby/blob/50dc0216789aadbe3dc6e2c7283128a7615ed627/lib/elastic/transport/transport/serializer/multi_json.rb#L40 (https://github.com/intridea/multi_json). Provide a custom middleware that uses MultiJson
2. Allow a user to customize building the Faraday client in ClickHouse::Connection#transport. Perhaps that could be a part of the gem configuration? For example:

ClickHouse.config do |config|
  # You can override config.faraday as a proc. If you don't there would be a default 
  # proc that builds Faraday in the config.
  config.faraday do |conn|
        conn.options.timeout = config.timeout
        conn.options.open_timeout = config.open_timeout
        conn.headers = config.headers
        conn.ssl.verify = config.ssl_verify
        conn.request(:basic_auth, config.username, config.password) if config.auth?
        conn.response Middleware::RaiseError
        conn.response Middleware::Logging, logger: config.logger!
        conn.response :json, content_type: %r{application/json}
        conn.response Middleware::ParseCsv, content_type: %r{text/csv}
        conn.adapter config.adapter
      end
end

class ClickHouse
  class Connection
    def transport
      @transport ||= config.faraday.call(config)
    end
  end
end

I'm happy to start a PR for either solution, but wanted to get your opinion first