Comments (3)
To test, prepare some line protocol like this:
log,host=domain.0,service=service-0 message="{\"time\":1701388800000000000,\"level\":\"info\",\"message\":\"Successfully queried data source\"}" 1701388800000000000
log,host=domain.1,service=service-1 message="{\"time\":1701388800000000000,\"level\":\"info\",\"message\":\"Backend rendering via external http server\"}" 1701388800000000000
log,host=domain.0,service=service-0 message="{\"time\":1701388860000000000,\"level\":\"warn\",\"message\":\"Failed to evaluate permission\"}" 1701388860000000000
log,host=domain.1,service=service-1 message="{\"time\":1701388860000000000,\"level\":\"info\",\"message\":\"Got event, updating modules\"}" 1701388860000000000
For example, generated by simulator.
$ command_to_generate_some_data --use-case log --scale 10 --start '2023-12-01T00:00:00Z' --end '2024-01-01T00:00:00Z' --interval '1s' > ./log_1.txt
$ head -n 1000 ./log_1.txt > ./log_0.txt
$ ls -lh
-rw-r--r--@ 1 a b 156K 12 11 15:37 log_0.txt
-rw-r--r--@ 1 a b 4.0G 12 11 15:36 log_1.txt
$ cnosdb-cli
public ❯ \w log_0.txt
Query took 0.030 seconds.
public ❯ \w log_1.txt
Query took 71.274 seconds.
from cnosdb.
cluster_schema ❯ \w log_1.txt
Query took 3.603 seconds.
cluster_schema ❯ \d
+------------+
| table_name |
+------------+
| air |
| log |
+------------+
Query took 0.005 seconds.
cluster_schema ❯ select count(*) from log;
+-----------------+
| COUNT(UInt8(1)) |
+-----------------+
| 864000 |
+-----------------+
Query took 0.027 seconds.
version:cnosdb 2.3.4, revision 07e229d
from cnosdb.
data generate as above method, there is no panic.
data sample:
log,host=domain.0,service=server-0 message="{\"time\":1703980800000000000,\"level\":\"warn\",\"message\":\"Failed to query data source\"}" 1703980800000000000
log,host=domain.1,service=server-1 message="{\"time\":1703980800000000000,\"level\":\"info\",\"message\":\"Sending alerts to local notifier\"}" 1703980800000000000
log,host=domain.2,service=service-2 message="{\"time\":1703980800000000000,\"level\":\"info\",\"message\":\"Checking table checksum\"}" 1703980800000000000
log,host=domain.3,service=service-3 message="{\"time\":1703980800000000000,\"level\":\"info\",\"message\":\"Checking table checksum\"}" 1703980800000000000
log,host=domain.4,service=service-4 message="{\"time\":1703980800000000000,\"level\":\"info\",\"message\":\"Sending alerts to local notifier\"}" 1703980800000000000
log,host=domain.5,service=server-5 message="{\"time\":1703980800000000000,\"level\":\"error\",\"message\":\"Failed to get IP from client address\"}" 1703980800000000000
log,host=domain.6,service=service-6 message="{\"time\":1703980800000000000,\"level\":\"error\",\"message\":\"Failed to validate URL\"}" 1703980800000000000
log,host=domain.7,service=service-7 message="{\"time\":1703980800000000000,\"level\":\"warn\",\"message\":\"Remote rendering request failed\"}" 1703980800000000000
log,host=domain.8,service=service-8 message="{\"time\":1703980800000000000,\"level\":\"warn\",\"message\":\"Failed to close response body\"}" 1703980800000000000
log,host=domain.9,service=server-9 message="{\"time\":1703980800000000000,\"level\":\"info\",\"message\":\"Applied new configuration\"}" 1703980800000000000
test step as follow:
public ❯ \w log_1.txt
Query took 2.667 seconds.
public ❯ \d
+------------+
| table_name |
+------------+
| log |
+------------+
Query took 0.045 seconds.
public ❯ select count(*) from log;
+-----------------+
| COUNT(UInt8(1)) |
+-----------------+
| 864000 |
+-----------------+
Query took 0.045 seconds.
version:cnosdb 2.4.0, revision db434d4
from cnosdb.
Related Issues (20)
- [BUG][resource status]The member role cannot find the deletion information of db
- [optimization]It is expected that the check subcommand of cnosdb can be optimized
- [BUG] cnosdb-cli cannot recognize the -p parameter HOT 1
- [optimization]When OOM occurs, it is recommended to print the log of cnosdb being killed.
- [Bug] restart_test failed HOT 1
- [Bug] test_reselect_new_leader failed
- Creating an external table in the format of `ndjson` is invalid
- [BUG][COPY INTO] "Too many open files" occasionally appears when exporting HOT 5
- [DISCUSS]Is the current way of handling "division by zero calculation" reasonable?
- [BUG]Computing and storage separate deployment "show datanodes" don't show query node. HOT 1
- Error creating `placement_policy` using `singleton` mode
- [BUG][function][count]Calculate the number of tags, cnosdb process panic HOT 1
- [BUG]The data node "Unreachable" cannot be restored to "healthy" after
- [Bug][Doc] The information in the Infrastructure Storage Separation section of the documentation is outdated.
- query and insert very slow HOT 2
- [BUG]Container version 2.4 is upgraded to community-2024-02-19 data is incompatible HOT 2
- [bug]have wrong when using the `make build` command HOT 5
- [BUG]TSM file xxx size less than index offset xxx HOT 1
- [Doc][Bug]The official configuration document states that "cache" refers to the channel capacity size without units.
- [Doc][Bug] inconsistencies between the cluster in the configuration file and the actual community edition configuration
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cnosdb.