bug to update date_time_input_format from basic to best_effort about clickhouse

[Algunenano]

This is expected. The client is doing the parsing and has different settings.

Related #56976

[pushpenderjindal]

what the solutions in that case. how can we fixed that?

[pushpenderjindal]

its working if i remove Z from time stamp timestamp":"2024-05-14T06:24:40.682561117Z

[Algunenano]

https://clickhouse.com/docs/en/interfaces/cli#interfaces_cli_configuration

[pushpenderjindal]

we are using vector-agent to push kubermets logs to CH. so still confused how can change the date_time_input_format to best_effort for the time inserting query by vector-agent?
refering : https://clickhouse.com/docs/en/operations/settings/formats#date_time_input_format
what excatly i need to do to insert

`clickhouse :) SELECT name, value, changed FROM system.settings WHERE name LIKE 'date_time%'

SELECT
name,
value,
changed
FROM system.settings
WHERE name LIKE 'date_time%'

Query id: 641252ca-e14a-4706-b248-da7207b62381

┌─name────────────────────────┬─value───────┬─changed─┐

1. │ date_time_input_format │ best_effort │ 1 │
2. │ date_time_output_format │ iso │ 1 │
3. │ date_time_overflow_behavior │ ignore │ 0 │
   └─────────────────────────────┴─────────────┴─────────┘

3 rows in set. Elapsed: 0.002 sec. Processed 1.01 thousand rows, 222.85 KB (602.90 thousand rows/s., 133.03 MB/s.)
Peak memory usage: 0.00 B.

clickhouse :) INSERT INTO newtable3 FORMAT JSONEachRow {"file":"/example/path.log","timestamp":"2024-05-14T06:24:40.682561117Z","kubernetes_container_id":"example_id","stream":"stdout"}

INSERT INTO newtable3 FORMAT JSONEachRow

Query id: a0cd7e37-d808-4a45-b82d-73c8719b5c76

Ok.
Exception on client:
Code: 27. DB::Exception: Cannot parse input: expected '"' before: 'Z","kubernetes_container_id":"example_id","stream":"stdout"}': (while reading the value of key timestamp): (at row 1)
: While executing ParallelParsingBlockInputFormat: data for INSERT was parsed from query. (CANNOT_PARSE_INPUT_ASSERTION_FAILED)`

pls ref. DB table schema above

[den-crane]

Actually it should work for vector.dev
Because vector works over HTTP and the profile date_time_input_format setting will be applied.

(the problem with the clickhouse-client that it parses the data locally without using profile date_time_input_format, because it has own date_time_input_format, and its not like this in case of HTTP)

Anyway there are various methods to set this setting in vector.dev config.
You can simply use endpoint = "http://localhost:8123?date_time_input_format=best_effort"
OR encoding.timestamp_format = "unix" https://vector.dev/docs/reference/configuration/sinks/websocket/#encoding.timestamp_format