Coder Social home page Coder Social logo

Comments (3)

zipper-meng avatar zipper-meng commented on June 15, 2024

To test, prepare some line protocol like this:

log,host=domain.0,service=service-0 message="{\"time\":1701388800000000000,\"level\":\"info\",\"message\":\"Successfully queried data source\"}" 1701388800000000000
log,host=domain.1,service=service-1 message="{\"time\":1701388800000000000,\"level\":\"info\",\"message\":\"Backend rendering via external http server\"}" 1701388800000000000
log,host=domain.0,service=service-0 message="{\"time\":1701388860000000000,\"level\":\"warn\",\"message\":\"Failed to evaluate permission\"}" 1701388860000000000
log,host=domain.1,service=service-1 message="{\"time\":1701388860000000000,\"level\":\"info\",\"message\":\"Got event, updating modules\"}" 1701388860000000000

For example, generated by simulator.

$ command_to_generate_some_data --use-case log --scale 10 --start '2023-12-01T00:00:00Z' --end '2024-01-01T00:00:00Z' --interval '1s' > ./log_1.txt
$ head -n 1000 ./log_1.txt > ./log_0.txt
$ ls -lh
-rw-r--r--@ 1 a  b   156K 12 11 15:37 log_0.txt
-rw-r--r--@ 1 a  b   4.0G 12 11 15:36 log_1.txt
$ cnosdb-cli
public ❯ \w log_0.txt
Query took 0.030 seconds.
public ❯ \w log_1.txt
Query took 71.274 seconds.

from cnosdb.

Benxiaohai001 avatar Benxiaohai001 commented on June 15, 2024
cluster_schema ❯ \w log_1.txt
Query took 3.603 seconds.
cluster_schema ❯ \d
+------------+
| table_name |
+------------+
| air        |
| log        |
+------------+
Query took 0.005 seconds.
cluster_schema ❯ select count(*) from log;
+-----------------+
| COUNT(UInt8(1)) |
+-----------------+
| 864000          |
+-----------------+
Query took 0.027 seconds.

version:cnosdb 2.3.4, revision 07e229d

from cnosdb.

Benxiaohai001 avatar Benxiaohai001 commented on June 15, 2024

data generate as above method, there is no panic.
data sample:

log,host=domain.0,service=server-0 message="{\"time\":1703980800000000000,\"level\":\"warn\",\"message\":\"Failed to query data source\"}" 1703980800000000000
log,host=domain.1,service=server-1 message="{\"time\":1703980800000000000,\"level\":\"info\",\"message\":\"Sending alerts to local notifier\"}" 1703980800000000000
log,host=domain.2,service=service-2 message="{\"time\":1703980800000000000,\"level\":\"info\",\"message\":\"Checking table checksum\"}" 1703980800000000000
log,host=domain.3,service=service-3 message="{\"time\":1703980800000000000,\"level\":\"info\",\"message\":\"Checking table checksum\"}" 1703980800000000000
log,host=domain.4,service=service-4 message="{\"time\":1703980800000000000,\"level\":\"info\",\"message\":\"Sending alerts to local notifier\"}" 1703980800000000000
log,host=domain.5,service=server-5 message="{\"time\":1703980800000000000,\"level\":\"error\",\"message\":\"Failed to get IP from client address\"}" 1703980800000000000
log,host=domain.6,service=service-6 message="{\"time\":1703980800000000000,\"level\":\"error\",\"message\":\"Failed to validate URL\"}" 1703980800000000000
log,host=domain.7,service=service-7 message="{\"time\":1703980800000000000,\"level\":\"warn\",\"message\":\"Remote rendering request failed\"}" 1703980800000000000
log,host=domain.8,service=service-8 message="{\"time\":1703980800000000000,\"level\":\"warn\",\"message\":\"Failed to close response body\"}" 1703980800000000000
log,host=domain.9,service=server-9 message="{\"time\":1703980800000000000,\"level\":\"info\",\"message\":\"Applied new configuration\"}" 1703980800000000000

test step as follow:

public ❯ \w log_1.txt
Query took 2.667 seconds.
public ❯ \d
+------------+
| table_name |
+------------+
| log        |
+------------+
Query took 0.045 seconds.
public ❯ select count(*) from log;
+-----------------+
| COUNT(UInt8(1)) |
+-----------------+
| 864000          |
+-----------------+
Query took 0.045 seconds.

version:cnosdb 2.4.0, revision db434d4

from cnosdb.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.