shlima / click_house Goto Github PK
View Code? Open in Web Editor NEWModern Ruby database driver for ClickHouse
Home Page: https://clickhouse.yandex/docs/en/
License: MIT License
Modern Ruby database driver for ClickHouse
Home Page: https://clickhouse.yandex/docs/en/
License: MIT License
If change the precision value from 10 to > 17, the test will fail, this is not correct because the precision can go up to 76 (https://clickhouse.com/docs/en/sql-reference/data-types/decimal/#parameters)
click_house/spec/click_house/type_spec.rb
Line 216 in 1282038
This is due to the fact that in BigDecimal when initializing Float, the maximum precision will be 17:
If initial is a Float, and the digits is larger than Float::DIG + 1, this exception is raised.
https://ruby-doc.org/stdlib-2.5.1/libdoc/bigdecimal/rdoc/BigDecimal.html#method-c-new-label-Exceptions
Can try converting the value to a string, but it is not clear whether precision works if pass a string:
Currently, the library can raise a Faraday::TimeoutError
.
In the RaiseError
middleware we catch Faraday::ConnectionFailed
and remap it to ClickHouse::NetworkException
:
However, TimeoutError
isn't a descendant of ConnectionFailed
:
I suggest we also catch Faraday::TimeoutError
(and perhaps a few more Faraday errors?) and remap it to ClickHouse
errors
Hi. If I understand correctly, click_house is not supported processing migrations files likerake db:migrate
with store in migration table processed migrations. Isn't it?
I have not-rails application with ActiveRecord. And I want to organize creating and processing migrations.
This isn't an issue exactly but is there a compatibility matrix for the gem with clickhouse versions ?
I plan to upgrade clickhouse from 21.1 to 21.7 and later to v22 and i couldn't find anything in the gem docs that would indicate compatibility
How I can insert nested data below?
{"id" => 1, "name" => 'one', "tests" => [{"test_id"=>1099803, "test_number"=>"35545585544", "test_number_2"=>"123456", "test_source"=>nil}, "test_id"=>1099804, "test_number"=>"1313113", "test_number_2"=>"654321", "test_source"=>nil}, {...}]}
and describe pls table structure for this. Thanks
I'd like to use Lowcardinality
, but as far as I can tell that isn't currently implemented in this library.
I can contribute a patch to help implement this feature, but I wanted look for your guidance first. Do you have any recommendations for implementing this feature? It looks like the existing Array type might be a good starting place.
You only need X-ClickHouse-Summary
. And that is returned with send_progress_in_http_headers = 0
. I.e. as far as I can see send_progress_in_http_headers
can be safely changed to 0
. With send_progress_in_http_headers = 1
, a relatively slow query (2 sec), and clickhouse
being behind nginx
, the latter says "upstream sent too big header" because clickhouse
produces a lot of "progress" headers:
$ curl -sSv 'http://xx.yyy.xx.yyy:8123/?send_progress_in_http_headers=1' -d 'select ..., count(*) from ... group by ... order by ... limit 1' -X GET -u '...:...'
* Expire in 0 ms for 6 (transfer ...)
* Trying xx.yyy.xx.yyy...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer ...)
* Connected to xx.yyy.xx.yyy (xx.yyy.xx.yyy) port 8123 (#0)
* Server auth using Basic with user '...'
> GET /?send_progress_in_http_headers=1 HTTP/1.1
> Host: xx.yyy.xx.yyy:8123
> Authorization: Basic ...
> User-Agent: curl/7.64.0
> Accept: */*
> Content-Length: 94
> Content-Type: application/x-www-form-urlencoded
>
} [94 bytes data]
* upload completely sent off: 94 out of 94 bytes
< HTTP/1.1 200 OK
< Date: Thu, 08 Feb 2024 13:22:37 GMT
< Connection: Keep-Alive
< Content-Type: text/tab-separated-values; charset=UTF-8
< X-ClickHouse-Server-Display-Name: ...
< Transfer-Encoding: chunked
< X-ClickHouse-Query-Id: ...
< X-ClickHouse-Format: TabSeparated
< X-ClickHouse-Timezone: UTC
< Keep-Alive: timeout=3
< X-ClickHouse-Progress: {"read_rows":"1111953","read_bytes":"8895624","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"9434898","read_bytes":"75479184","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"18412695","read_bytes":"147301560","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"27260817","read_bytes":"218086536","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"35690512","read_bytes":"285524096","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"44667166","read_bytes":"357337328","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"53121310","read_bytes":"424970480","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"61969813","read_bytes":"495758504","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"70226587","read_bytes":"561812696","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"78697369","read_bytes":"629578952","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"86824087","read_bytes":"694592696","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"94819987","read_bytes":"758559896","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Summary: {"read_rows":"101288000","read_bytes":"810304000","written_rows":"0","written_bytes":"0","total_rows_to_read":"101288000","result_rows":"0","result_bytes":"0"}
<
{ [11 bytes data]
* Connection #0 to host xx.yyy.xx.yyy left intact
165 1
Hi.
Noticed that now clickhouse supports Ipv4/Ipv6 https://clickhouse.tech/docs/en/sql-reference/data-types/domains/ipv4/.
I think would be nice to add that here.
Btw I can implement it in free time
Problem: Creating a column with a codec type throws an error.
Migration Repro:
ClickHouse.connection.create_table('visits', engine: 'MergeTree ORDER BY date') do |t|
t.Date :date, compression_codec: 'CODEC(DoubleDelta, ZSTD(1))'
end
Error:
Caused by:
NoMethodError: undefined method `compression_codec=' for #<ClickHouse::Definition::Column:0x0000000105b5abe8 @type="Date", @name=:date, @extensions=[{:compression_codec=>"CODEC(DoubleDelta, ZSTD(1))"}]> (NoMethodError)
params.each { |k, v| public_send("#{k}=", v) }
^^^^^^^^^^^
In addition to compression_codec
, trying to use: compression:
, codec:
, and codec_expression:
all failed.
I was able to work around the issue by using raw SQL.
ClickHouse.connection.create_table('visits', engine: 'MergeTree ORDER BY date') do |t|
t << "date Date CODEC(DoubleDelta, ZSTD(1))"
end
It'd be great feature to have this as supported for DSL statements.
By default, if we check the available types we can see that Nullable(Float32)
and Nullable(Float64)
are casted to nullable integers instead of nullable floats:
> ClickHouse.types
...
"Nullable(Float32)"=>#<ClickHouse::Type::NullableType:0x00007fa730bb1aa0 @subtype=#<ClickHouse::Type::IntegerType:0x00007fa730bb1ac8>>,
...
"Nullable(Float64)"=>#<ClickHouse::Type::NullableType:0x00007fa730bb19d8 @subtype=#<ClickHouse::Type::IntegerType:0x00007fa730bb1a00>>,
...
There is a workaround to this issue. We can execute (if we are in Rails we can do it in an initializer):
ClickHouse.add_type('Nullable(Float32)', ClickHouse::Type::NullableType.new(ClickHouse::Type::FloatType.new))
ClickHouse.add_type('Nullable(Float64)', ClickHouse::Type::NullableType.new(ClickHouse::Type::FloatType.new))
Hey @shlima!
During local testing, I'm noticing inconsistent data returned by the .select_all
and .select_one
methods. Specifically, when there is X amount of data inserted into the table (in my local dev I only have ~2.5k rows).
Here's an example with .select_one
:
As you can see, the first example returns an opening bracket cast as a string. The second example returns what I'd usually expect.
Here's an example with .select_all
:
Here, the first example is returning raw JSON, this is not expected.
I have an inclination that Faraday is struggling to deserialize the JSON response (because of the response size) which causes the above issues. I don't have a proposed fix but I'm wondering if you've encountered this before and have suggestions on how to move forward?
I am using Ruby on Rails 7 with multiple databases. I got this error.
app/models/ams_record.rb
connects_to database: { writing: :ams, reading: :ams }
/Users/name/.rvm/gems/ruby-3.1.2/gems/activerecord-7.0.8/lib/active_record/connection_adapters/abstract/connection_handler.rb:261:in `resolve_pool_config':
database configuration does not specify adapter (ActiveRecord::AdapterNotSpecified)
database.yml
default_ams: &default_ams
timeout: 60
open_timeout: 3
development:
primary:
<<: *default
database: xxx_development
password: password
host: localhost
port: 5432
ams:
<<: *default_ams
database: ams_development
username: default
password: XXXX
url: https://xxx.us-east-1.aws.clickhouse.cloud:8443
app/models/click_house_record.rb
class ClickHouseRecord < ActiveRecord::Base
self.abstract_class = true
class << self
def agent
ClickHouse.connection
end
def insert(*argv, &block)
agent.insert(table_name, *argv, &block)
end
def select_one
agent.select_one(current_scope.to_sql)
end
def select_value
agent.select_value(current_scope.to_sql)
end
def select_all
agent.select_all(current_scope.to_sql)
end
def explain
agent.explain(current_scope.to_sql)
end
end
end
app/models/ams_record.rb
class AmsRecord < ClickHouseRecord
self.abstract_class = true
connects_to database: { writing: :ams, reading: :ams }
end
app/models/ams_traffic.rb
class AmsTraffic < AmsRecord
# This is my table on Click House
end
Does this gem support Mutation-based updates using ALTER…UPDATE in case of irregular and non-frequent changes ?
see https://clickhouse.com/docs/en/sql-reference/statements/alter/update/
I can see that insert operations uses to_json which do not support non unicode data. But every other part of the system support binary data. Can we do something so that we do not need to replace invalid utf-8 chars using scrub?
DB support binary data
https://clickhouse.com/docs/en/sql-reference/data-types/string/#encodings
FORMAT supports binary data
http://www.devdoc.net/database/ClickhouseDocs_19.4.1.3-docs/interfaces/formats/#jsoneachrow
Gem uses to_json to convert hash(representing values of a row) to generate the query.
But to_json does not work with binary data. It expects data to have valid unicode characters only.
There are many other formats also but I am not sure what format will solve this issue: https://clickhouse.com/docs/en/interfaces/formats/#jsoneachrow
[65] pry(main)> str = "\xc2"
=> "\xC2"
[66] pry(main)> Model.insert { |buffer| buffer << {:col1 => 1, :col2 => str} }
JSON::GeneratorError: partial character in source, but hit end
from /home/user/.rvm/gems/ruby-2.7.4/gems/activesupport-5.2.2.1/lib/active_support/core_ext/object/json.rb:38:in `to_json'
[69] pry(main)> str.encoding
=> #<Encoding:UTF-8>
[70] pry(main)> str.valid_encoding?
=> false
I need to check how other db drivers support binary data.
I will be interested in contributing if needed and required.
Do you have ideas on how can we achieve this?
create table dt_type_test
(
a DateTime
, b Nullable(DateTime)
, c DateTime64(3)
, d Nullable(DateTime64(3))
, e DateTime64(3, 'UTC')
, f Nullable(DateTime64(3, 'UTC'))
, g Decimal(18,2)
)
engine=Log ;
insert into dt_type_test values (now(), now(), now(), now(), now(), now(), 5.99);
[14] pry(main)> rec = ClickHouse.connection.select_one("select * from dt_type_test limit 1")
SQL (Total: 10MS, CH: 4MS) select * from dt_type_test limit 1;
Read: 1 rows, 51.0B. Written: 0 rows, 0B
=> {"a"=>Fri, 01 Jul 2022 17:48:53 +0000,
"b"=>Fri, 01 Jul 2022 17:48:53 +0000,
"c"=>"2022-07-01 17:48:53.000",
"d"=>"2022-07-01 17:48:53.000",
"e"=>"2022-07-01 23:48:53.000",
"f"=>"2022-07-01 23:48:53.000",
"g"=>5.99}
[15] pry(main)> rec["g"].class
=> Float
My expectation is that all of a, b, c, d, e, and f would be of type DateTime
, and g would be of type BigDecimal
. Instead we see c-f as strings, and g as a Float.
Poking around, I found at least two problems.
One is that DateTime64(%d)
is not a registered type. That's an easy fix.
The bigger problem is that parameterized types with %d
placeholders don't seem to work at all. For example, it appears that ClickHouse::Response::ResultSet#extract_type_info
parses DateTime64(3, 'UTC')
as DateTime64(%s, %s)
rather than DateTtime64(%d, %s)
. And since DateTime64(%s, %s)
isn't a registered type, it falls back to treating it as a string. Similarly, Decimal(%d, %d)
is a registered type, but gets parsed as Decimal(%s, %s)
which doesn't match.
In fact, extract_type_info
makes no attempt to insert %d
placeholders, always using %s
, which is inconsistent with how the type registry works. So if I'm understanding this right, no parameterized type with a %d
placeholder will ever work.
Currently gem doesn't support params in SELECT
queries AFAIK. I mean params like SELECT * FROM t WHERE id = {id:Int64}
Do you consider adding this? I can provide PR for that
I already use fork with this feature added. But I did not update it for a year, so I can't provide PR right away 😅 You can check here, what it looks like
I'm wanting to use Lowcardinality
type , from what I can tell, this isn't supported by this library today, is that correct?
I'd be willing to create a patch, as I require this functionality myself. Do you have any recommendations before I get started on that?
'
Hello, firstly thank you for the gem.
I have a parsing problem with inserting BigDecimal values to a table. The problem can be reproduced both for inserting BigDecimal type value and for manual casting:
irb(main):003:0> c = ClickHouse.connection
irb(main):004:0> c.execute "CREATE TABLE test(id Decimal64(2)) ENGINE TinyLog"
irb(main):004:0> c.insert("test", columns: [:id], values: [[1.0]])
=> true
irb(main):006:0> c.insert("test", columns: [:id], values: [[BigDecimal("1.0")]])
ClickHouse::DbException ([400] Code: 27, e.displayText() = DB::ParsingException: Cannot parse input: expected ',' before: '"0.1e1"}'{} (version 21.1.2.15 (official build)))
irb(main):007:0> c.insert("test", columns: [:id], values: [[ClickHouse::Type::DecimalType.new.serialize(1.0)]])
ClickHouse::DbException ([400] Code: 27, e.displayText() = DB::ParsingException: Cannot parse input: expected ',' before: '"0.1e1"}'{} (version 21.1.2.15 (official build)))
As I understand this is due to 2 reasons:
Do you have any ideas to solve this?
Hello I have this error when I run 200 workers in one time,
"ActiveRecord::StatementInvalid: Errno::EMFILE: Failed to open TCP connection to 192.168.0.1:8123 (Too many open files - socket(2) for"
How I understand the problem is because I didn't close my old connection, can you tell me please if your gem has any method for closing connection with the socket?
Thank you.
Using "tuples" insert notation I noticed that if I attempt to write nil to it like this
ClickHouse.connection.insert('devices', columns: %i(device_id kind model vendor os_name os_version referer updated_at created_at)) do |buf|
buf << ['608253467_null', 'mobile', 'ALCATEL ONE TOUCH 993D', 'samsung', 'android', '19', nil, DateTime.parse('Sat, 13 Feb 2016 04:32:48').strftime('%Y-%m-%d %H:%M:%S'), DateTime.parse('Sat, 13 Feb 2016 03:39:07').strftime('%Y-%m-%d %H:%M:%S')]
end
(notice the nil value) raises exception ClickHouse::DbException ([400] Code: 26, e.displayText() = DB::Exception: Cannot parse JSON string: expected opening quote: (while read the value of key referer): (at row 1))
I assume that it would be better to silently convert value to its default value by column type on the fly ('' for String, 0 for Int and so on).
It is also surprising that time have to be specially formatted for clickhouse by hand as well.
What do you think? Is it made to give more controll over data to user to avoid server versions compatibility issues?
Is there a way to expose the exceptions and allow callers to capture and handle appropriately? Right now running a select_all on a rogue query quietly fails saying it got 0 results, as opposed to fail because of a logic error:
Simple steps to reproduce from the rails console:
>> # This should cause an exception as the function ccccccc() doesn't exist
>> ClickHouse.connection.select_all("SELECT ccccccc() from cccccccxxxxxx")
SQL (Total: 104MS, CH: 0MS)� SELECT ccccccc() from cccccccxxxxxx;�
Read: 0 rows, 0B. Written: 0 rows, 0B
[]
>> ClickHouse.connection.select_all("SELECT now()")
SQL (Total: 171MS, CH: 1MS)� SELECT now();�
Read: 1 rows, 1.0B. Written: 0 rows, 0B
[{"now()"=>2024-04-04 17:25:04 -0400}]
The exception is there in the faraday.body, but when converting to a ResultSet in the ClickHouse::Response::Factory.response()
method that exception is never checked.
As I understand from ClickHouse logs, your driver inserts with JSONEachRow format, but the previous unmaintainable one are inserting with CSVWithNames format. Is there a way to make this configurable or no?
I want to checkout to use your client, but need to modify bunch of things for that
Hi @shlima
There is one question.
When a query is made via (select_all, select_value or select_one) method, get method is called:
def select_all(sql)
response = get(body: Util::Statement.format(sql, 'JSON'))
Response::Factory[response]
end
When logging such requests at the INFO level, we get this result:
[2022-06-06T12:06:25.936896] #044c48a9: SQL (Total: 32779MS, CH: 3MS) ;
[2022-06-06T12:06:25.937438] #044c48a9: Read: 4 rows, 4.2KiB. Written: 0 rows, 0B
[2022-06-06T12:08:39.955952] #2610758b: SQL (Total: 117118MS, CH: 6MS) ;
[2022-06-06T12:08:39.956922] #2610758b: Read: 4 rows, 4.2KiB. Written: 0 rows, 0B
If you look at the logger code, you can see that we are waiting for the query output to the console (look query method):
def on_complete(env)
summary = extract_summary(env.response_headers)
logger.info("\e[1m�[35mSQL (#{duration_stats_log(env.body)})�\e[0m #{query(env)};�")
logger.debug(body) if body && !query_in_body?(env)
logger.info("\e[1m�[36mRead: #{summary.fetch(:read_rows)} rows, #{summary.fetch(:read_bytes)}. Written: #{summary.fetch(:written_rows)} rows, #{summary.fetch(:written_bytes)}\e[0m")
end
According to the code, we must take the value from @Body, but it will only be if we have a DEBUG logging level:
def call(environment)
@starting = timestamp
@body = environment.body if log_body?
@app.call(environment).on_complete(&method(:on_complete))
end
private
def log_body?
logger.level == Logger::DEBUG
end
In this case, if you make a request through execute
method, a POST request will be made and in this case, the query will get into the logger (INFO).
[2022-06-06T10:03:50.059963] #88d8d7f8: SQL (Total: 25MS) SELECT * FROM test;
[2022-06-06T10:03:50.060545] #88d8d7f8: Read: 3 rows, 3.1KiB. Written: 0 rows, 0B
Should it work like this?
I know my setting are correct but I'm unable to ping. Do I need to set send_progress_in_http_headers to a certain value to get this to work?
2.3.8 :044 > require 'click_house'
=> false
2.3.8 :045 > ClickHouse.config do |config|
2.3.8 :046 > config.logger = Logger.new(STDOUT)
2.3.8 :047?> config.adapter = :net_http
2.3.8 :048?> config.database = 'default'
2.3.8 :049?> config.timeout = 60
2.3.8 :050?> config.open_timeout = 3
2.3.8 :051?> config.ssl_verify = false
2.3.8 :052?> config.headers = {}
2.3.8 :053?>
2.3.8 :054 > # or provide connection options separately
2.3.8 :055 > config.scheme = 'http'
2.3.8 :056?> config.host = 'localhost'
2.3.8 :057?> config.port = '8123'
2.3.8 :058?>
2.3.8 :062?> end
=> #<ClickHouse::Config:0x00007fbeb02571c0 @adapter=:net_http, @url=nil, @scheme="http", @host="localhost", @PORT="8123", @logger=#<Logger:0x00007fbeb0256798 @progname=nil, @Level=0, @default_formatter=#<Logger::Formatter:0x00007fbeb0256770 @datetime_format=nil>, @Formatter=nil, @logdev=#<Logger::LogDevice:0x00007fbeb0256720 @shift_size=nil, @shift_age=nil, @filename=nil, @dev=#<IO:>, @mon_owner=nil, @mon_count=0, @mon_mutex=#Thread::Mutex:0x00007fbeb0256568>>, @database="", @username="", @password="", @timeout=60, @open_timeout=3, @ssl_verify=false, @headers={}>
2.3.8 :063 > ClickHouse.connection.ping
NoMethodError: undefined method compact' for {:send_progress_in_http_headers=>nil, :database=>nil}:Hash from /Users/slee/.rvm/gems/ruby-2.3.8/gems/click_house-1.3.6/lib/click_house/connection.rb:49:in
compose'
from /Users/slee/.rvm/gems/ruby-2.3.8/gems/click_house-1.3.6/lib/click_house/connection.rb:25:in get' from /Users/slee/.rvm/gems/ruby-2.3.8/gems/click_house-1.3.6/lib/click_house/extend/connection_healthy.rb:8:in
ping'
from (irb):63
from /Users/slee/.rvm/rubies/ruby-2.3.8/bin/irb:11:in `
We sometimes fetch a lot of data from Clickhouse and the default Faraday JSON parser is quite slow. Patching the JSON library with the Oj gem would help, but I can't do that in my project.
I can think of two different solutions to that problem:
MultiJson
which automatically picks the most performant JSON library: https://github.com/elastic/elastic-transport-ruby/blob/50dc0216789aadbe3dc6e2c7283128a7615ed627/lib/elastic/transport/transport/serializer/multi_json.rb#L40 (https://github.com/intridea/multi_json). Provide a custom middleware that uses MultiJsonClickHouse::Connection#transport
. Perhaps that could be a part of the gem configuration? For example:ClickHouse.config do |config|
# You can override config.faraday as a proc. If you don't there would be a default
# proc that builds Faraday in the config.
config.faraday do |conn|
conn.options.timeout = config.timeout
conn.options.open_timeout = config.open_timeout
conn.headers = config.headers
conn.ssl.verify = config.ssl_verify
conn.request(:basic_auth, config.username, config.password) if config.auth?
conn.response Middleware::RaiseError
conn.response Middleware::Logging, logger: config.logger!
conn.response :json, content_type: %r{application/json}
conn.response Middleware::ParseCsv, content_type: %r{text/csv}
conn.adapter config.adapter
end
end
class ClickHouse
class Connection
def transport
@transport ||= config.faraday.call(config)
end
end
end
I'm happy to start a PR for either solution, but wanted to get your opinion first
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.