Comments (7)
Hi,
How do you fetch the data - do you use streaming (take a look at the 2 links below)
https://github.com/SAP/node-hdb#streaming-results
https://github.com/SAP/node-hdb#streaming-large-objects
Best regards
from node-hdb.
Yeah, I'm using streaming. I'm doing something like:
var FETCH_SIZE = Math.pow(2,14) // 16384
, READ_SIZE = Math.pow(2,10); // 1024
...
client.execute(sql, function(err, resultSet) {
resultSet.setFetchSize(FETCH_SIZE);
var stream = resultSet.createArrayStream(READ_SIZE);
stream.pipe(...);
});
Once I hit FETCH_SIZE
rows, the stream seems to pause for a bit (up to a minute), and then it resumes. I'd like to reduce this pause if possible.
from node-hdb.
I am logging the stream in my application. I've bolded the rows that show the pause.
timestamp | time delta | message | total rows |
---|---|---|---|
2017-03-06 15:11:03-08:00 | 1ms | client.execute called | 0 |
2017-03-06 15:11:09-08:00 | 5174ms | execute callback | 0 |
2017-03-06 15:11:33-08:00 | 23924ms | Chunk: 1024 | 1024 |
2017-03-06 15:11:33-08:00 | 75ms | Chunk: 1024 | 2048 |
2017-03-06 15:11:33-08:00 | 79ms | Chunk: 1024 | 3072 |
2017-03-06 15:11:33-08:00 | 79ms | Chunk: 1024 | 4096 |
2017-03-06 15:11:33-08:00 | 91ms | Chunk: 1024 | 5120 |
2017-03-06 15:11:33-08:00 | 85ms | Chunk: 1024 | 6144 |
2017-03-06 15:11:33-08:00 | 91ms | Chunk: 1024 | 7168 |
2017-03-06 15:11:33-08:00 | 92ms | Chunk: 1024 | 8192 |
2017-03-06 15:11:33-08:00 | 92ms | Chunk: 1024 | 9216 |
2017-03-06 15:11:33-08:00 | 82ms | Chunk: 1024 | 10240 |
2017-03-06 15:11:33-08:00 | 70ms | Chunk: 1024 | 11264 |
2017-03-06 15:11:33-08:00 | 61ms | Chunk: 1024 | 12288 |
2017-03-06 15:11:33-08:00 | 71ms | Chunk: 1024 | 13312 |
2017-03-06 15:11:34-08:00 | 62ms | Chunk: 1024 | 14336 |
2017-03-06 15:11:34-08:00 | 69ms | Chunk: 1024 | 15360 |
2017-03-06 15:11:34-08:00 | 73ms | Chunk: 1024 | 16384 |
2017-03-06 15:11:57-08:00 | 23576ms | Chunk: 1024 | 17408 |
2017-03-06 15:11:57-08:00 | 83ms | Chunk: 1024 | 18432 |
2017-03-06 15:11:57-08:00 | 85ms | Chunk: 1024 | 19456 |
2017-03-06 15:11:57-08:00 | 76ms | Chunk: 1024 | 20480 |
2017-03-06 15:11:58-08:00 | 84ms | Chunk: 1024 | 21504 |
2017-03-06 15:11:58-08:00 | 93ms | Chunk: 1024 | 22528 |
2017-03-06 15:11:58-08:00 | 95ms | Chunk: 1024 | 23552 |
2017-03-06 15:11:58-08:00 | 97ms | Chunk: 1024 | 24576 |
2017-03-06 15:11:58-08:00 | 105ms | Chunk: 1024 | 25600 |
2017-03-06 15:11:58-08:00 | 106ms | Chunk: 1024 | 26624 |
2017-03-06 15:11:58-08:00 | 63ms | Chunk: 1024 | 27648 |
2017-03-06 15:11:58-08:00 | 67ms | Chunk: 1024 | 28672 |
2017-03-06 15:11:58-08:00 | 62ms | Chunk: 1024 | 29696 |
2017-03-06 15:11:58-08:00 | 69ms | Chunk: 1024 | 30720 |
2017-03-06 15:11:58-08:00 | 63ms | Chunk: 1024 | 31744 |
2017-03-06 15:11:58-08:00 | 63ms | Chunk: 1024 | 32768 |
2017-03-06 15:12:15-08:00 | 16079ms | Chunk: 1024 | 33792 |
2017-03-06 15:12:15-08:00 | 90ms | Chunk: 1024 | 34816 |
2017-03-06 15:12:15-08:00 | 90ms | Chunk: 1024 | 35840 |
2017-03-06 15:12:15-08:00 | 90ms | Chunk: 1024 | 36864 |
2017-03-06 15:12:15-08:00 | 92ms | Chunk: 1024 | 37888 |
2017-03-06 15:12:15-08:00 | 85ms | Chunk: 1024 | 38912 |
2017-03-06 15:12:15-08:00 | 103ms | Chunk: 1024 | 39936 |
2017-03-06 15:12:15-08:00 | 90ms | Chunk: 1024 | 40960 |
2017-03-06 15:12:15-08:00 | 100ms | Chunk: 1024 | 41984 |
2017-03-06 15:12:15-08:00 | 96ms | Chunk: 1024 | 43008 |
2017-03-06 15:12:16-08:00 | 128ms | Chunk: 1024 | 44032 |
2017-03-06 15:12:16-08:00 | 125ms | Chunk: 1024 | 45056 |
2017-03-06 15:12:16-08:00 | 61ms | Chunk: 1024 | 46080 |
2017-03-06 15:12:16-08:00 | 64ms | Chunk: 1024 | 47104 |
2017-03-06 15:12:16-08:00 | 70ms | Chunk: 1024 | 48128 |
2017-03-06 15:12:16-08:00 | 63ms | Chunk: 1024 | 49152 |
2017-03-06 15:12:35-08:00 | 19411ms | Chunk: 1024 | 50176 |
2017-03-06 15:12:35-08:00 | 79ms | Chunk: 1024 | 51200 |
2017-03-06 15:12:35-08:00 | 86ms | Chunk: 1024 | 52224 |
..... |
from node-hdb.
Could you enable GC tracing, from the looks of it - it might be that the GC is kicking-in.
from node-hdb.
Here is the same log but with gc tracing enabled (--trace_gc
).
2017-03-07 11:00:02-08:00|1ms|Executing query...
[34651:0x102004600] 16576 ms: Scavenge 124.2 (161.3) -> 112.3 (161.3) MB, 6.6 / 0 ms [allocation failure].
2017-03-07 11:00:08-08:00|5275ms|SQL execution started...
[34651:0x102004600] 56743 ms: Scavenge 126.2 (161.3) -> 115.7 (161.3) MB, 29.8 / 0 ms [allocation failure].
2017-03-07 11:00:43-08:00|35799ms|First data chunk received after 35.80s
[34651:0x102004600] 56762 ms: Scavenge 118.8 (161.3) -> 117.3 (161.3) MB, 4.4 / 0 ms [allocation failure].
[34651:0x102004600] 56776 ms: Scavenge 119.3 (161.3) -> 118.6 (161.3) MB, 2.6 / 0 ms [allocation failure].
[34651:0x102004600] 56787 ms: Scavenge 120.3 (161.3) -> 119.7 (162.3) MB, 2.5 / 0 ms [allocation failure].
[34651:0x102004600] 56803 ms: Scavenge 121.6 (162.3) -> 120.9 (163.3) MB, 2.8 / 0 ms (+ 2.4 ms in 29 steps since last GC) [allocation failure].
2017-03-07 11:00:43-08:00|66ms|Chunk: 1024 (1024 total)
[34651:0x102004600] 56817 ms: Scavenge 122.6 (163.3) -> 122.0 (164.3) MB, 2.3 / 0 ms (+ 1.7 ms in 27 steps since last GC) [allocation failure].
[34651:0x102004600] 56832 ms: Scavenge 123.9 (164.3) -> 123.1 (165.3) MB, 2.8 / 0 ms (+ 2.3 ms in 30 steps since last GC) [allocation failure].
[34651:0x102004600] 56843 ms: Scavenge 125.0 (165.3) -> 124.2 (166.3) MB, 2.7 / 0 ms (+ 2.0 ms in 29 steps since last GC) [allocation failure].
[34651:0x102004600] 56880 ms: Scavenge 129.0 (166.3) -> 127.3 (167.3) MB, 3.0 / 0 ms (+ 6.2 ms in 75 steps since last GC) [allocation failure].
[34651:0x102004600] 56902 ms: Scavenge 130.1 (167.3) -> 129.1 (171.3) MB, 3.8 / 0 ms (+ 2.0 ms in 44 steps since last GC) [allocation failure].
2017-03-07 11:00:43-08:00|103ms|Chunk: 1024 (2048 total)
[34651:0x102004600] 56962 ms: Scavenge 139.1 (171.3) -> 134.9 (173.3) MB, 5.3 / 0 ms (+ 7.8 ms in 157 steps since last GC) [allocation failure].
2017-03-07 11:00:44-08:00|87ms|Chunk: 1024 (3072 total)
[34651:0x102004600] 57010 ms: Scavenge 140.9 (173.3) -> 138.1 (178.3) MB, 6.5 / 0 ms (+ 6.5 ms in 93 steps since last GC) [allocation failure].
[34651:0x102004600] 57085 ms: Scavenge 149.9 (178.3) -> 144.6 (181.3) MB, 6.8 / 0 ms (+ 13.2 ms in 186 steps since last GC) [allocation failure].
2017-03-07 11:00:44-08:00|92ms|Chunk: 1024 (4096 total)
[34651:0x102004600] 57145 ms: Scavenge 153.7 (181.3) -> 149.7 (188.3) MB, 8.7 / 0 ms (+ 11.6 ms in 143 steps since last GC) [allocation failure].
2017-03-07 11:00:44-08:00|91ms|Chunk: 1024 (5120 total)
[34651:0x102004600] 57210 ms: Scavenge 160.2 (188.3) -> 155.5 (193.3) MB, 7.9 / 0 ms (+ 11.7
ms in 165 steps since last GC) [allocation failure].
[34651:0x102004600] 57274 ms: Scavenge 164.7 (193.3) -> 160.3 (198.3) MB, 7.0 / 0 ms (+ 8.9 ms in 142 steps since last GC) [allocation failure].
2017-03-07 11:00:44-08:00|91ms|Chunk: 1024 (6144 total)
[34651:0x102004600] 57341 ms: Scavenge 170.2 (198.3) -> 165.8 (204.3) MB, 8.6 / 0 ms (+ 13.4 ms in 154 steps since last GC) [allocation failure].
2017-03-07 11:00:44-08:00|96ms|Chunk: 1024 (7168 total)
[34651:0x102004600] 57385 ms: Mark-sweep 170.1 (204.3) -> 141.3 (200.4) MB, 10.7 / 0 ms (+ 97.7 ms in 1341 steps since start of marking, biggest step 0.6 ms) [GC interrupt] [GC in old space requested].
[34651:0x102004600] 57445 ms: Scavenge 154.3 (200.4) -> 148.5 (201.4) MB, 5.3 / 0 ms [allocation failure].
2017-03-07 11:00:44-08:00|76ms|Chunk: 1024 (8192 total)
[34651:0x102004600] 57488 ms: Scavenge 157.0 (201.4) -> 153.3 (205.4) MB, 8.3 / 0 ms [allocation failure].
2017-03-07 11:00:44-08:00|65ms|Chunk: 1024 (9216 total)
[34651:0x102004600] 57538 ms: Scavenge 164.2 (205.4) -> 159.3 (208.4) MB, 6.9 / 0 ms [allocation failure].
2017-03-07 11:00:44-08:00|65ms|Chunk: 1024 (10240 total)
[34651:0x102004600] 57585 ms: Scavenge 168.3 (208.4) -> 164.0 (213.4) MB, 7.5 / 0 ms [allocation failure].
[34651:0x102004600] 57636 ms: Scavenge 173.9 (213.4) -> 169.5 (219.4) MB, 9.0 / 0 ms [allocation failure].
2017-03-07 11:00:44-08:00|73ms|Chunk: 1024 (11264 total)
[34651:0x102004600] 57684 ms: Scavenge 179.6 (219.4) -> 175.1 (223.4) MB, 9.7 / 0 ms [allocation failure].
2017-03-07 11:00:44-08:00|66ms|Chunk: 1024 (12288 total)
[34651:0x102004600] 57730 ms: Scavenge 184.9 (223.4) -> 180.4 (227.4) MB, 6.8 / 0 ms [allocation failure].
[34651:0x102004600] 57778 ms: Scavenge 190.0 (227.4) -> 185.6 (232.4) MB, 7.4 / 0 ms [allocation failure].
2017-03-07 11:00:44-08:00|68ms|Chunk: 1024 (13312 total)
[34651:0x102004600] 57826 ms: Scavenge 195.4 (232.4) -> 191.1 (238.4) MB, 7.9 / 0 ms [allocation failure].
2017-03-07 11:00:44-08:00|65ms|Chunk: 1024 (14336 total)
[34651:0x102004600] 57873 ms: Scavenge 201.2 (238.4) -> 196.6 (242.4) MB, 8.9 / 0 ms [allocation failure].
2017-03-07 11:00:44-08:00|65ms|Chunk: 1024 (15360 total)
[34651:0x102004600] 57922 ms: Scavenge 206.2 (242.4) -> 201.6 (246.4) MB, 6.6 / 0 ms [allocation failure].
[34651:0x102004600] 57968 ms: Scavenge 211.3 (246.4) -> 207.0 (252.4) MB, 8.3 / 0 ms [allocation failure].
2017-03-07 11:00:45-08:00|70ms|Chunk: 1024 (16384 total)
--------- PAUSE ----------------------------
[34651:0x102004600] 71561 ms: Scavenge 217.4 (253.4) -> 210.6 (257.4) MB, 10.3 / 0 ms [allocation failure].
[34651:0x102004600] 72902 ms: Mark-sweep 222.5 (258.4) -> 212.8 (260.4) MB, 122.7 / 0 ms [allocation failure] [promotion limit reached].
2017-03-07 11:00:59-08:00|14927ms|Chunk: 1024 (17408 total)
[34651:0x102004600] 72949 ms: Scavenge 222.2 (260.4) -> 218.1 (264.4) MB, 11.6 / 0 ms [allocation failure].
2017-03-07 11:01:00-08:00|68ms|Chunk: 1024 (18432 total)
[34651:0x102004600] 72996 ms: Scavenge 228.3 (264.4) -> 223.7 (268.4) MB, 6.9 / 0 ms [allocation failure].
[34651:0x102004600] 73044 ms: Scavenge 232.9 (268.4) -> 228.6 (273.4) MB, 7.8 / 0 ms [allocation failure].
2017-03-07 11:01:00-08:00|68ms|Chunk: 1024 (19456 total)
[34651:0x102004600] 73091 ms: Scavenge 238.5 (273.4) -> 234.2 (279.4) MB, 7.9 / 0 ms [allocation failure].
2017-03-07 11:01:00-08:00|73ms|Chunk: 1024 (20480 total)
[34651:0x102004600] 73155 ms: Scavenge 244.3 (279.4) -> 239.7 (284.4) MB, 8.0 / 0 ms (+ 11.6 ms in 156 steps since last GC) [allocation failure].
2017-03-07 11:01:00-08:00|84ms|Chunk: 1024 (21504 total)
[34651:0x102004600] 73217 ms: Scavenge 249.3 (284.4) -> 244.7 (289.4) MB, 7.4 / 0 ms (+ 6.2 ms in 147 steps since last GC) [allocation failure].
[34651:0x102004600] 73276 ms: Scavenge 254.4 (289.4) -> 250.0 (295.4) MB, 7.5 / 0 ms (+ 8.6 ms in 150 steps since last GC) [allocation failure].
2017-03-07 11:01:00-08:00|90ms|Chunk: 1024 (22528 total)
[34651:0x102004600] 73339 ms: Scavenge 260.2 (295.4) -> 255.8 (300.4) MB, 8.8 / 0 ms (+ 10.6 ms in 158 steps since last GC) [allocation failure].
2017-03-07 11:01:00-08:00|89ms|Chunk: 1024 (23552 total)
[34651:0x102004600] 73402 ms: Scavenge 265.4 (300.4) -> 261.0 (306.4) MB, 7.5 / 0 ms (+ 11.8 ms in 148 steps since last GC) [allocation failure].
[34651:0x102004600] 73466 ms: Scavenge 270.6 (306.4) -> 266.2 (311.4) MB, 7.6 / 0 ms (+ 9.6 ms in 150 steps since last GC) [allocation failure].
2017-03-07 11:01:00-08:00|90ms|Chunk: 1024 (24576 total)
[34651:0x102004600] 73523 ms: Scavenge 276.0 (311.4) -> 271.7 (317.4) MB, 9.3 / 0 ms (+ 6.3 ms in 152 steps since last GC) [allocation failure].
2017-03-07 11:01:00-08:00|85ms|Chunk: 1024 (25600 total)
[34651:0x102004600] 73591 ms: Scavenge 281.8 (317.4) -> 277.2 (322.4) MB, 8.6 / 0 ms (+ 14.1 ms in 154 steps since last GC) [allocation failure].
2017-03-07 11:01:00-08:00|97ms|Chunk: 1024 (26624 total)
[34651:0x102004600] 73664 ms: Scavenge 286.7 (322.4) -> 282.1 (327.4) MB, 8.5 / 0 ms (+ 13.5 ms in 144 steps since last GC) [allocation failure].
[34651:0x102004600] 73733 ms: Scavenge 291.7 (327.4) -> 287.5 (334.4) MB, 8.6 / 0 ms (+ 14.3 ms in 151 steps since last GC) [allocation failure].
2017-03-07 11:01:00-08:00|104ms|Chunk: 1024 (27648 total)
[34651:0x102004600] 73804 ms: Scavenge 297.8 (334.4) -> 293.3 (338.4) MB, 10.5 / 0 ms (+ 14.1 ms in 159 steps since last GC) [allocation failure].
2017-03-07 11:01:00-08:00|97ms|Chunk: 1024 (28672 total)
[34651:0x102004600] 73872 ms: Scavenge 302.8 (338.4) -> 298.3 (344.4) MB, 7.6 / 0 ms (+ 13.9 ms in 143 steps since last GC) [allocation failure].
[34651:0x102004600] 73941 ms: Scavenge 308.0 (344.4) -> 303.6 (349.4) MB, 8.4 / 0 ms (+ 13.6 ms in 152 steps since last GC) [allocation failure].
2017-03-07 11:01:01-08:00|101ms|Chunk: 1024 (29696 total)
[34651:0x102004600] 74018 ms: Scavenge 313.6 (349.4) -> 309.3 (355.4) MB, 8.6 / 0 ms (+ 23.7 ms in 154 steps since last GC) [allocation failure].
[34651:0x102004600] 74033 ms: Mark-sweep 309.6 (355.4) -> 304.5 (356.4) MB, 11.8 / 0 ms (+ 172.5 ms in 2123 steps since start of marking, biggest step 0.7 ms) [GC interrupt] [GC in old space requested].
2017-03-07 11:01:01-08:00|104ms|Chunk: 1024 (30720 total)
[34651:0x102004600] 74106 ms: Scavenge 320.1 (356.4) -> 314.0 (356.4) MB, 6.5 / 0 ms [allocation failure].
2017-03-07 11:01:01-08:00|63ms|Chunk: 1024 (31744 total)
[34651:0x102004600] 74139 ms: Scavenge 320.3 (356.4) -> 317.1 (364.4) MB, 9.1 / 0 ms [allocation failure].
[34651:0x102004600] 74194 ms: Scavenge 328.9 (364.4) -> 323.5 (367.4) MB, 6.2 / 0 ms [allocation failure].
2017-03-07 11:01:01-08:00|71ms|Chunk: 1024 (32768 total)
I'm not super familiar with gc tracing, so let me know if I should be running a different command. From the logs, it looks like the gc pause is very minimal, but I may be misinterpreting.
Where does the streaming begin in the driver? Is it streaming the rows straight from the db, and then parsing them in the stream? Or does it need to buffer all the rows, parse, and then convert to stream? Because that could explain the pause I guess.
from node-hdb.
It doesn't look to be the GC.
I am trying to reproduce the problem, but do not see such pauses.
Another guess, is the query slow. If you run it in HANA Studio do you see pauses. Could you analyze the query on the HANA side? What happens if you read the data from a LOB type.
The driver isn't caching batches of data, it should stream the results right out. You shouldn't have to tweak the driver.
from node-hdb.
Ok, I reran the queries to a server closer to the database, and things are streaming right out. Before, the database was being accessed through a VPN. So, I guess the VPN is the bottleneck here. Seems obvious now....
By the way, this problem was happening at any FETCH_SIZE
, but the lower the FETCH_SIZE
, the shorter the pause.
Just so I understand, can you explain why a slower connection / lower bandwidth (i.e. my VPN) would cause pauses in the stream? I would expect to see every chunk come slower. In the end I guess it's equivalent, I'm just curious and want to learn more about the driver internals.
from node-hdb.
Related Issues (20)
- [rl-reuse_tool-3] Violation against OSS Rules of Play HOT 1
- [rl-reuse_tool-4] Violation against OSS Rules of Play HOT 1
- typescript definitions HOT 2
- SSL support? HOT 3
- TSL handling HOT 1
- README: broken link in product page
- Connection parameters? HOT 2
- Installation issue with npm install -g @sap/hana-client HOT 6
- Invalid parameter: 'parameter (5)' contains different types.
- hdb and hana-client comparison wrt CAP HOT 2
- Data corruption for NCLOB fields
- IpAddress instead of Hostname
- SAP/hana-client architecture issue HOT 4
- Decimal number like .8 cannot be accepted
- Support timestamp seconds precisions with more than 3 fractional digits HOT 1
- Apache License terms contains placeholder names
- Crashes with LOBs with Duplicated Alias
- Server says client version is not compatible
- Insert of blob field from stream.Readable hangs
- Error: "hdb" couldn't be found HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from node-hdb.