Comments (4)
yu285
commented on June 12, 2024
May I have your "show create database xxxx\G".
And the "describe tablename".
I need to get your database configuration and table structure.
from tdengine.
yu285
commented on June 12, 2024
in the first time the speed went down , could you provide the capture of "top -Hp pidof taosd" ? we can check the thread workload during the problem.
from tdengine.
moaazassali
commented on June 12, 2024
show create database test\G (this is using the default parameters except for DURATION=1d and VGROUPS=8):
Database: test
Create Database: CREATE DATABASE `test` BUFFER 256 CACHESIZE 1 CACHEMODEL 'none' COMP 2 DURATION 1440m WAL_FSYNC_PERIOD 3000 MAXROWS 4096 MINROWS 100 STT_TRIGGER 1 KEEP 5256000m,5256000m,5256000m PAGES 256 PAGESIZE 4 PRECISION 'ms' REPLICA 1 WAL_LEVEL 1 VGROUPS 8 SINGLE_STABLE 0 TABLE_PREFIX 0 TABLE_SUFFIX 0 TSDB_PAGESIZE 4 WAL_RETENTION_PERIOD 3600 WAL_RETENTION_SIZE 0 KEEP_TIME_OFFSET 0 ENCRYPT_ALGORITHM 'none' S3_CHUNKSIZE 262144 S3_KEEPLOCAL 5256000m S3_COMPACT 0
describe test.devices:
field | type | length | note | encode | compress | level |
================================================================================================================================================
ts | TIMESTAMP | 8 | | delta-i | lz4 | medium |
temperature | INT | 4 | | simple8b | lz4 | medium |
pressure | DOUBLE | 8 | | delta-d | lz4 | medium |
humidity | BIGINT | 8 | | simple8b | lz4 | medium |
device_id | VARCHAR | 10 | TAG | disabled | disabled | disabled |
top -Hp 341534 of first run with write speed of 11.6M/s (not full list of threads but sorted by highest CPU%):
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
341760 root 20 0 8548344 408336 18568 S 3.3 5.8 0:00.10 vnode-write
341844 root 20 0 8548344 408336 18568 S 3.3 5.8 0:00.10 vnode-write
341845 root 20 0 8548344 408336 18568 S 3.3 5.8 0:00.10 vnode-write
341758 root 20 0 8548344 408336 18568 S 3.0 5.8 0:00.09 vnode-write
341840 root 20 0 8548344 408336 18568 S 3.0 5.8 0:00.09 vnode-write
341756 root 20 0 8548344 408336 18568 S 2.7 5.8 0:00.08 vnode-write
341761 root 20 0 8548344 408336 18568 S 2.7 5.8 0:00.08 vnode-write
341839 root 20 0 8548344 408336 18568 S 2.7 5.8 0:00.08 vnode-write
341536 root 20 0 8548344 408336 18568 S 2.0 5.8 0:00.08 trans-svr-work
341537 root 20 0 8548344 408336 18568 S 1.7 5.8 0:00.12 trans-svr-work
341539 root 20 0 8548344 408336 18568 S 1.7 5.8 0:00.08 trans-svr-work
341538 root 20 0 8548344 408336 18568 S 1.3 5.8 0:00.07 trans-svr-work
341535 root 20 0 8548344 408336 18568 S 0.3 5.8 0:00.32 log
341681 root 20 0 8548344 408336 18568 S 0.3 5.8 0:00.01 vnode-fetch
341703 root 20 0 8548344 408336 18568 S 0.3 5.8 0:00.01 vnode-fetch
341721 root 20 0 8548344 408336 18568 S 0.3 5.8 0:00.01 vnode-fetch
341534 root 20 0 8548344 408336 18568 S 0.0 5.8 0:00.09 taosd
341540 root 20 0 8548344 408336 18568 S 0.0 5.8 0:00.00 trans-accept
341541 root 20 0 8548344 408336 18568 S 0.0 5.8 0:00.00 dnode-cli
341542 root 20 0 8548344 408336 18568 S 0.0 5.8 0:00.00 dnode-cli
341543 root 20 0 8548344 408336 18568 S 0.0 5.8 0:00.37 tmr-taskQ
341544 root 20 0 8548344 408336 18568 S 0.0 5.8 0:01.63 tmr
341545 root 20 0 8548344 408336 18568 S 0.0 5.8 0:00.04 dnode-sta-cli
top -Hp 341534 of second run with write speed of 12.0M/s:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
341758 root 20 0 8548236 750200 18568 S 3.7 10.7 0:00.20 vnode-write
341844 root 20 0 8548236 750200 18568 S 3.7 10.7 0:00.21 vnode-write
341845 root 20 0 8548236 750200 18568 S 3.7 10.7 0:00.21 vnode-write
341756 root 20 0 8548236 750200 18568 S 3.3 10.7 0:00.18 vnode-write
341760 root 20 0 8548236 750200 18568 S 3.3 10.7 0:00.20 vnode-write
341839 root 20 0 8548236 750200 18568 S 3.3 10.7 0:00.18 vnode-write
341761 root 20 0 8548236 750200 18568 S 3.0 10.7 0:00.17 vnode-write
341840 root 20 0 8548236 750200 18568 S 3.0 10.7 0:00.18 vnode-write
341536 root 20 0 8548236 750200 18568 S 1.7 10.7 0:00.17 trans-svr-work
341537 root 20 0 8548236 750200 18568 S 1.3 10.7 0:00.24 trans-svr-work
341539 root 20 0 8548236 750200 18568 S 1.3 10.7 0:00.15 trans-svr-work
341538 root 20 0 8548236 750200 18568 S 1.0 10.7 0:00.15 trans-svr-work
341544 root 20 0 8548236 750200 18568 S 0.3 10.7 0:03.04 tmr
341534 root 20 0 8548236 750200 18568 S 0.0 10.7 0:00.17 taosd
341535 root 20 0 8548236 750200 18568 S 0.0 10.7 0:00.59 log
341540 root 20 0 8548236 750200 18568 S 0.0 10.7 0:00.00 trans-accept
341541 root 20 0 8548236 750200 18568 S 0.0 10.7 0:00.01 dnode-cli
341542 root 20 0 8548236 750200 18568 S 0.0 10.7 0:00.01 dnode-cli
341543 root 20 0 8548236 750200 18568 S 0.0 10.7 0:00.70 tmr-taskQ
341545 root 20 0 8548236 750200 18568 S 0.0 10.7 0:00.09 dnode-sta-cli
341546 root 20 0 8548236 750200 18568 S 0.0 10.7 0:00.00 dnode-sync-cli
341547 root 20 0 8548236 750200 18568 S 0.0 10.7 0:00.00 dnode-sync-cli
341548 root 20 0 8548236 750200 18568 S 0.0 10.7 0:00.00 dnode-mgmt
top -Hp 341534 of third run with write speed of 1.9M/s:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
342432 root 20 0 8564628 998.3m 18692 D 12.0 14.6 0:00.36 vnode-commit
342433 root 20 0 8564628 998.3m 18692 D 12.0 14.6 0:00.36 vnode-commit
342434 root 20 0 8564628 998.3m 18692 D 10.0 14.6 0:00.30 vnode-commit
342435 root 20 0 8564628 998.3m 18692 D 8.3 14.6 0:00.25 vnode-commit
341840 root 20 0 8564628 998.3m 18692 S 4.7 14.6 0:00.32 vnode-write
341844 root 20 0 8564628 998.3m 18692 S 4.3 14.6 0:00.34 vnode-write
341758 root 20 0 8564628 998.3m 18692 S 4.0 14.6 0:00.32 vnode-write
341845 root 20 0 8564628 998.3m 18692 S 4.0 14.6 0:00.33 vnode-write
341756 root 20 0 8564628 998.3m 18692 S 3.7 14.6 0:00.29 vnode-write
341760 root 20 0 8564628 998.3m 18692 S 3.7 14.6 0:00.31 vnode-write
341839 root 20 0 8564628 998.3m 18692 S 3.7 14.6 0:00.29 vnode-write
341761 root 20 0 8564628 998.3m 18692 S 3.0 14.6 0:00.26 vnode-write
341537 root 20 0 8564628 998.3m 18692 S 1.7 14.6 0:00.32 trans-svr-work
341536 root 20 0 8564628 998.3m 18692 S 1.3 14.6 0:00.24 trans-svr-work
341539 root 20 0 8564628 998.3m 18692 S 1.3 14.6 0:00.23 trans-svr-work
341538 root 20 0 8564628 998.3m 18692 S 1.0 14.6 0:00.21 trans-svr-work
341721 root 20 0 8564628 998.3m 18692 S 0.3 14.6 0:00.04 vnode-fetch
341534 root 20 0 8564628 998.3m 18692 S 0.0 14.6 0:00.18 taosd
341535 root 20 0 8564628 998.3m 18692 S 0.0 14.6 0:00.65 log
341540 root 20 0 8564628 998.3m 18692 S 0.0 14.6 0:00.00 trans-accept
341541 root 20 0 8564628 998.3m 18692 S 0.0 14.6 0:00.01 dnode-cli
341542 root 20 0 8564628 998.3m 18692 S 0.0 14.6 0:00.01 dnode-cli
341543 root 20 0 8564628 998.3m 18692 S 0.0 14.6 0:00.77 tmr-taskQ
It seems that the difference is the vnode-commit threads in the third run.
Fourth run had a write speed of 7.3M/s with only one vnode-commit thread with CPU usage of 11.7%.
Fifth run had a write speed of 3.4M/s with two vnode-commit threads with 11.6% and 11.0% CPU usage.
Note that the default frequency of top is 3 seconds, so the numbers aren't very accurate especially since the first two runs are done in like 0.25s. Nonetheless, they show the relevant differences.
from tdengine.
moaazassali
commented on June 12, 2024
@yu285 any updates on this performance issue?
from tdengine.
Related Issues (20)
- can I set the timezone parameter in the Go connector?
- Taos 进程重启后，taos / vnode-commit 对 CPU 产生高占用 HOT 1
- client failed to install in Alpine Linux
- After data was migrated with taosdump, the tags are not consistent with pre-migration HOT 1
- taosdump还原数据的时候，能否恢复到新的数据库，不是备份的数据库的名字，修改dbs.sql无效 HOT 3
- jdbc 3.2.10 & TDengine 3.2.1 - concurrent queries throw FETCH TIME OUT HOT 2
- failed to init dnode since unsupported platform HOT 2
- high availability in the 3.2.3.0 client HOT 5
- 单行函数嵌入到流式计算中报错 HOT 1
- delete data failed HOT 3
- When stream contains aggregate UDF, partition by is must-have
- range too large when database using precision=us(or ns) and query with interval(1n)
- Managed to build but failed to run TDengine 3.3.0.0 on unsupported Linux distribution
- win系统刚安装好之后重启电脑，运行C:\TDengine\taos.exe 提示failed to connect to server
- Ambiguity in official documentation
- java用restfull调用td数据插入，当字段数据类型FLOAT，且是负数时，报异常，直接执行sql无问题
- taosd crashes with Segmentation fault when trying to query table with PRIMARY KEY column defined HOT 4
- 关于INTERVAL(1w)的周期问题 HOT 4
- In Linux system deployment, inserting TDengine database to save, but running the same code on Windows is successful
- 2.6.0.34版本内存占用过高，无法释放 HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tdengine.