Coder Social home page Coder Social logo

Comments (7)

alexpenev-s avatar alexpenev-s commented on August 18, 2024

Hi,
How do you fetch the data - do you use streaming (take a look at the 2 links below)
https://github.com/SAP/node-hdb#streaming-results
https://github.com/SAP/node-hdb#streaming-large-objects

Best regards

from node-hdb.

surjikal avatar surjikal commented on August 18, 2024

Yeah, I'm using streaming. I'm doing something like:

var FETCH_SIZE = Math.pow(2,14) // 16384
  , READ_SIZE  = Math.pow(2,10); // 1024
...
client.execute(sql, function(err, resultSet) {
  resultSet.setFetchSize(FETCH_SIZE);
  var stream = resultSet.createArrayStream(READ_SIZE);
  stream.pipe(...);
});

Once I hit FETCH_SIZE rows, the stream seems to pause for a bit (up to a minute), and then it resumes. I'd like to reduce this pause if possible.

from node-hdb.

surjikal avatar surjikal commented on August 18, 2024

I am logging the stream in my application. I've bolded the rows that show the pause.

timestamp time delta message total rows
2017-03-06 15:11:03-08:00 1ms client.execute called 0
2017-03-06 15:11:09-08:00 5174ms execute callback 0
2017-03-06 15:11:33-08:00 23924ms Chunk: 1024 1024
2017-03-06 15:11:33-08:00 75ms Chunk: 1024 2048
2017-03-06 15:11:33-08:00 79ms Chunk: 1024 3072
2017-03-06 15:11:33-08:00 79ms Chunk: 1024 4096
2017-03-06 15:11:33-08:00 91ms Chunk: 1024 5120
2017-03-06 15:11:33-08:00 85ms Chunk: 1024 6144
2017-03-06 15:11:33-08:00 91ms Chunk: 1024 7168
2017-03-06 15:11:33-08:00 92ms Chunk: 1024 8192
2017-03-06 15:11:33-08:00 92ms Chunk: 1024 9216
2017-03-06 15:11:33-08:00 82ms Chunk: 1024 10240
2017-03-06 15:11:33-08:00 70ms Chunk: 1024 11264
2017-03-06 15:11:33-08:00 61ms Chunk: 1024 12288
2017-03-06 15:11:33-08:00 71ms Chunk: 1024 13312
2017-03-06 15:11:34-08:00 62ms Chunk: 1024 14336
2017-03-06 15:11:34-08:00 69ms Chunk: 1024 15360
2017-03-06 15:11:34-08:00 73ms Chunk: 1024 16384
2017-03-06 15:11:57-08:00 23576ms Chunk: 1024 17408
2017-03-06 15:11:57-08:00 83ms Chunk: 1024 18432
2017-03-06 15:11:57-08:00 85ms Chunk: 1024 19456
2017-03-06 15:11:57-08:00 76ms Chunk: 1024 20480
2017-03-06 15:11:58-08:00 84ms Chunk: 1024 21504
2017-03-06 15:11:58-08:00 93ms Chunk: 1024 22528
2017-03-06 15:11:58-08:00 95ms Chunk: 1024 23552
2017-03-06 15:11:58-08:00 97ms Chunk: 1024 24576
2017-03-06 15:11:58-08:00 105ms Chunk: 1024 25600
2017-03-06 15:11:58-08:00 106ms Chunk: 1024 26624
2017-03-06 15:11:58-08:00 63ms Chunk: 1024 27648
2017-03-06 15:11:58-08:00 67ms Chunk: 1024 28672
2017-03-06 15:11:58-08:00 62ms Chunk: 1024 29696
2017-03-06 15:11:58-08:00 69ms Chunk: 1024 30720
2017-03-06 15:11:58-08:00 63ms Chunk: 1024 31744
2017-03-06 15:11:58-08:00 63ms Chunk: 1024 32768
2017-03-06 15:12:15-08:00 16079ms Chunk: 1024 33792
2017-03-06 15:12:15-08:00 90ms Chunk: 1024 34816
2017-03-06 15:12:15-08:00 90ms Chunk: 1024 35840
2017-03-06 15:12:15-08:00 90ms Chunk: 1024 36864
2017-03-06 15:12:15-08:00 92ms Chunk: 1024 37888
2017-03-06 15:12:15-08:00 85ms Chunk: 1024 38912
2017-03-06 15:12:15-08:00 103ms Chunk: 1024 39936
2017-03-06 15:12:15-08:00 90ms Chunk: 1024 40960
2017-03-06 15:12:15-08:00 100ms Chunk: 1024 41984
2017-03-06 15:12:15-08:00 96ms Chunk: 1024 43008
2017-03-06 15:12:16-08:00 128ms Chunk: 1024 44032
2017-03-06 15:12:16-08:00 125ms Chunk: 1024 45056
2017-03-06 15:12:16-08:00 61ms Chunk: 1024 46080
2017-03-06 15:12:16-08:00 64ms Chunk: 1024 47104
2017-03-06 15:12:16-08:00 70ms Chunk: 1024 48128
2017-03-06 15:12:16-08:00 63ms Chunk: 1024 49152
2017-03-06 15:12:35-08:00 19411ms Chunk: 1024 50176
2017-03-06 15:12:35-08:00 79ms Chunk: 1024 51200
2017-03-06 15:12:35-08:00 86ms Chunk: 1024 52224
.....

from node-hdb.

alexpenev-s avatar alexpenev-s commented on August 18, 2024

Could you enable GC tracing, from the looks of it - it might be that the GC is kicking-in.

from node-hdb.

surjikal avatar surjikal commented on August 18, 2024

Here is the same log but with gc tracing enabled (--trace_gc).

2017-03-07 11:00:02-08:00|1ms|Executing query...
[34651:0x102004600]    16576 ms: Scavenge 124.2 (161.3) -> 112.3 (161.3) MB, 6.6 / 0 ms [allocation failure].
2017-03-07 11:00:08-08:00|5275ms|SQL execution started...


[34651:0x102004600]    56743 ms: Scavenge 126.2 (161.3) -> 115.7 (161.3) MB, 29.8 / 0 ms [allocation failure].

2017-03-07 11:00:43-08:00|35799ms|First data chunk received after 35.80s

[34651:0x102004600]    56762 ms: Scavenge 118.8 (161.3) -> 117.3 (161.3) MB, 4.4 / 0 ms [allocation failure].
[34651:0x102004600]    56776 ms: Scavenge 119.3 (161.3) -> 118.6 (161.3) MB, 2.6 / 0 ms [allocation failure].
[34651:0x102004600]    56787 ms: Scavenge 120.3 (161.3) -> 119.7 (162.3) MB, 2.5 / 0 ms [allocation failure].
[34651:0x102004600]    56803 ms: Scavenge 121.6 (162.3) -> 120.9 (163.3) MB, 2.8 / 0 ms (+ 2.4 ms in 29 steps since last GC) [allocation failure].

2017-03-07 11:00:43-08:00|66ms|Chunk: 1024 (1024 total)

[34651:0x102004600]    56817 ms: Scavenge 122.6 (163.3) -> 122.0 (164.3) MB, 2.3 / 0 ms (+ 1.7 ms in 27 steps since last GC) [allocation failure].
[34651:0x102004600]    56832 ms: Scavenge 123.9 (164.3) -> 123.1 (165.3) MB, 2.8 / 0 ms (+ 2.3 ms in 30 steps since last GC) [allocation failure].
[34651:0x102004600]    56843 ms: Scavenge 125.0 (165.3) -> 124.2 (166.3) MB, 2.7 / 0 ms (+ 2.0 ms in 29 steps since last GC) [allocation failure].
[34651:0x102004600]    56880 ms: Scavenge 129.0 (166.3) -> 127.3 (167.3) MB, 3.0 / 0 ms (+ 6.2 ms in 75 steps since last GC) [allocation failure].
[34651:0x102004600]    56902 ms: Scavenge 130.1 (167.3) -> 129.1 (171.3) MB, 3.8 / 0 ms (+ 2.0 ms in 44 steps since last GC) [allocation failure].

2017-03-07 11:00:43-08:00|103ms|Chunk: 1024 (2048 total)

[34651:0x102004600]    56962 ms: Scavenge 139.1 (171.3) -> 134.9 (173.3) MB, 5.3 / 0 ms (+ 7.8 ms in 157 steps since last GC) [allocation failure].

2017-03-07 11:00:44-08:00|87ms|Chunk: 1024 (3072 total)

[34651:0x102004600]    57010 ms: Scavenge 140.9 (173.3) -> 138.1 (178.3) MB, 6.5 / 0 ms (+ 6.5 ms in 93 steps since last GC) [allocation failure].
[34651:0x102004600]    57085 ms: Scavenge 149.9 (178.3) -> 144.6 (181.3) MB, 6.8 / 0 ms (+ 13.2 ms in 186 steps since last GC) [allocation failure].

2017-03-07 11:00:44-08:00|92ms|Chunk: 1024 (4096 total)

[34651:0x102004600]    57145 ms: Scavenge 153.7 (181.3) -> 149.7 (188.3) MB, 8.7 / 0 ms (+ 11.6 ms in 143 steps since last GC) [allocation failure].

2017-03-07 11:00:44-08:00|91ms|Chunk: 1024 (5120 total)

[34651:0x102004600]    57210 ms: Scavenge 160.2 (188.3) -> 155.5 (193.3) MB, 7.9 / 0 ms (+ 11.7 
ms in 165 steps since last GC) [allocation failure].
[34651:0x102004600]    57274 ms: Scavenge 164.7 (193.3) -> 160.3 (198.3) MB, 7.0 / 0 ms (+ 8.9 ms in 142 steps since last GC) [allocation failure].

2017-03-07 11:00:44-08:00|91ms|Chunk: 1024 (6144 total)

[34651:0x102004600]    57341 ms: Scavenge 170.2 (198.3) -> 165.8 (204.3) MB, 8.6 / 0 ms (+ 13.4 ms in 154 steps since last GC) [allocation failure].

2017-03-07 11:00:44-08:00|96ms|Chunk: 1024 (7168 total)

[34651:0x102004600]    57385 ms: Mark-sweep 170.1 (204.3) -> 141.3 (200.4) MB, 10.7 / 0 ms (+ 97.7 ms in 1341 steps since start of marking, biggest step 0.6 ms) [GC interrupt] [GC in old space requested].
[34651:0x102004600]    57445 ms: Scavenge 154.3 (200.4) -> 148.5 (201.4) MB, 5.3 / 0 ms [allocation failure].

2017-03-07 11:00:44-08:00|76ms|Chunk: 1024 (8192 total)

[34651:0x102004600]    57488 ms: Scavenge 157.0 (201.4) -> 153.3 (205.4) MB, 8.3 / 0 ms [allocation failure].

2017-03-07 11:00:44-08:00|65ms|Chunk: 1024 (9216 total)

[34651:0x102004600]    57538 ms: Scavenge 164.2 (205.4) -> 159.3 (208.4) MB, 6.9 / 0 ms [allocation failure].

2017-03-07 11:00:44-08:00|65ms|Chunk: 1024 (10240 total)

[34651:0x102004600]    57585 ms: Scavenge 168.3 (208.4) -> 164.0 (213.4) MB, 7.5 / 0 ms [allocation failure].
[34651:0x102004600]    57636 ms: Scavenge 173.9 (213.4) -> 169.5 (219.4) MB, 9.0 / 0 ms [allocation failure].

2017-03-07 11:00:44-08:00|73ms|Chunk: 1024 (11264 total)

[34651:0x102004600]    57684 ms: Scavenge 179.6 (219.4) -> 175.1 (223.4) MB, 9.7 / 0 ms [allocation failure].

2017-03-07 11:00:44-08:00|66ms|Chunk: 1024 (12288 total)

[34651:0x102004600]    57730 ms: Scavenge 184.9 (223.4) -> 180.4 (227.4) MB, 6.8 / 0 ms [allocation failure].
[34651:0x102004600]    57778 ms: Scavenge 190.0 (227.4) -> 185.6 (232.4) MB, 7.4 / 0 ms [allocation failure].

2017-03-07 11:00:44-08:00|68ms|Chunk: 1024 (13312 total)

[34651:0x102004600]    57826 ms: Scavenge 195.4 (232.4) -> 191.1 (238.4) MB, 7.9 / 0 ms [allocation failure].

2017-03-07 11:00:44-08:00|65ms|Chunk: 1024 (14336 total)

[34651:0x102004600]    57873 ms: Scavenge 201.2 (238.4) -> 196.6 (242.4) MB, 8.9 / 0 ms [allocation failure].

2017-03-07 11:00:44-08:00|65ms|Chunk: 1024 (15360 total)

[34651:0x102004600]    57922 ms: Scavenge 206.2 (242.4) -> 201.6 (246.4) MB, 6.6 / 0 ms [allocation failure].
[34651:0x102004600]    57968 ms: Scavenge 211.3 (246.4) -> 207.0 (252.4) MB, 8.3 / 0 ms [allocation failure].

2017-03-07 11:00:45-08:00|70ms|Chunk: 1024 (16384 total)

--------- PAUSE ----------------------------

[34651:0x102004600]    71561 ms: Scavenge 217.4 (253.4) -> 210.6 (257.4) MB, 10.3 / 0 ms [allocation failure].
[34651:0x102004600]    72902 ms: Mark-sweep 222.5 (258.4) -> 212.8 (260.4) MB, 122.7 / 0 ms [allocation failure] [promotion limit reached].

2017-03-07 11:00:59-08:00|14927ms|Chunk: 1024 (17408 total)

[34651:0x102004600]    72949 ms: Scavenge 222.2 (260.4) -> 218.1 (264.4) MB, 11.6 / 0 ms [allocation failure].

2017-03-07 11:01:00-08:00|68ms|Chunk: 1024 (18432 total)

[34651:0x102004600]    72996 ms: Scavenge 228.3 (264.4) -> 223.7 (268.4) MB, 6.9 / 0 ms [allocation failure].
[34651:0x102004600]    73044 ms: Scavenge 232.9 (268.4) -> 228.6 (273.4) MB, 7.8 / 0 ms [allocation failure].

2017-03-07 11:01:00-08:00|68ms|Chunk: 1024 (19456 total)

[34651:0x102004600]    73091 ms: Scavenge 238.5 (273.4) -> 234.2 (279.4) MB, 7.9 / 0 ms [allocation failure].

2017-03-07 11:01:00-08:00|73ms|Chunk: 1024 (20480 total)

[34651:0x102004600]    73155 ms: Scavenge 244.3 (279.4) -> 239.7 (284.4) MB, 8.0 / 0 ms (+ 11.6 ms in 156 steps since last GC) [allocation failure].

2017-03-07 11:01:00-08:00|84ms|Chunk: 1024 (21504 total)

[34651:0x102004600]    73217 ms: Scavenge 249.3 (284.4) -> 244.7 (289.4) MB, 7.4 / 0 ms (+ 6.2 ms in 147 steps since last GC) [allocation failure].
[34651:0x102004600]    73276 ms: Scavenge 254.4 (289.4) -> 250.0 (295.4) MB, 7.5 / 0 ms (+ 8.6 ms in 150 steps since last GC) [allocation failure].

2017-03-07 11:01:00-08:00|90ms|Chunk: 1024 (22528 total)

[34651:0x102004600]    73339 ms: Scavenge 260.2 (295.4) -> 255.8 (300.4) MB, 8.8 / 0 ms (+ 10.6 ms in 158 steps since last GC) [allocation failure].

2017-03-07 11:01:00-08:00|89ms|Chunk: 1024 (23552 total)

[34651:0x102004600]    73402 ms: Scavenge 265.4 (300.4) -> 261.0 (306.4) MB, 7.5 / 0 ms (+ 11.8 ms in 148 steps since last GC) [allocation failure].
[34651:0x102004600]    73466 ms: Scavenge 270.6 (306.4) -> 266.2 (311.4) MB, 7.6 / 0 ms (+ 9.6 ms in 150 steps since last GC) [allocation failure].

2017-03-07 11:01:00-08:00|90ms|Chunk: 1024 (24576 total)

[34651:0x102004600]    73523 ms: Scavenge 276.0 (311.4) -> 271.7 (317.4) MB, 9.3 / 0 ms (+ 6.3 ms in 152 steps since last GC) [allocation failure].

2017-03-07 11:01:00-08:00|85ms|Chunk: 1024 (25600 total)

[34651:0x102004600]    73591 ms: Scavenge 281.8 (317.4) -> 277.2 (322.4) MB, 8.6 / 0 ms (+ 14.1 ms in 154 steps since last GC) [allocation failure].

2017-03-07 11:01:00-08:00|97ms|Chunk: 1024 (26624 total)

[34651:0x102004600]    73664 ms: Scavenge 286.7 (322.4) -> 282.1 (327.4) MB, 8.5 / 0 ms (+ 13.5 ms in 144 steps since last GC) [allocation failure].
[34651:0x102004600]    73733 ms: Scavenge 291.7 (327.4) -> 287.5 (334.4) MB, 8.6 / 0 ms (+ 14.3 ms in 151 steps since last GC) [allocation failure].

2017-03-07 11:01:00-08:00|104ms|Chunk: 1024 (27648 total)

[34651:0x102004600]    73804 ms: Scavenge 297.8 (334.4) -> 293.3 (338.4) MB, 10.5 / 0 ms (+ 14.1 ms in 159 steps since last GC) [allocation failure].

2017-03-07 11:01:00-08:00|97ms|Chunk: 1024 (28672 total)

[34651:0x102004600]    73872 ms: Scavenge 302.8 (338.4) -> 298.3 (344.4) MB, 7.6 / 0 ms (+ 13.9 ms in 143 steps since last GC) [allocation failure].
[34651:0x102004600]    73941 ms: Scavenge 308.0 (344.4) -> 303.6 (349.4) MB, 8.4 / 0 ms (+ 13.6 ms in 152 steps since last GC) [allocation failure].

2017-03-07 11:01:01-08:00|101ms|Chunk: 1024 (29696 total)

[34651:0x102004600]    74018 ms: Scavenge 313.6 (349.4) -> 309.3 (355.4) MB, 8.6 / 0 ms (+ 23.7 ms in 154 steps since last GC) [allocation failure].
[34651:0x102004600]    74033 ms: Mark-sweep 309.6 (355.4) -> 304.5 (356.4) MB, 11.8 / 0 ms (+ 172.5 ms in 2123 steps since start of marking, biggest step 0.7 ms) [GC interrupt] [GC in old space requested].

2017-03-07 11:01:01-08:00|104ms|Chunk: 1024 (30720 total)

[34651:0x102004600]    74106 ms: Scavenge 320.1 (356.4) -> 314.0 (356.4) MB, 6.5 / 0 ms [allocation failure].

2017-03-07 11:01:01-08:00|63ms|Chunk: 1024 (31744 total)

[34651:0x102004600]    74139 ms: Scavenge 320.3 (356.4) -> 317.1 (364.4) MB, 9.1 / 0 ms [allocation failure].
[34651:0x102004600]    74194 ms: Scavenge 328.9 (364.4) -> 323.5 (367.4) MB, 6.2 / 0 ms [allocation failure].

2017-03-07 11:01:01-08:00|71ms|Chunk: 1024 (32768 total)

I'm not super familiar with gc tracing, so let me know if I should be running a different command. From the logs, it looks like the gc pause is very minimal, but I may be misinterpreting.

Where does the streaming begin in the driver? Is it streaming the rows straight from the db, and then parsing them in the stream? Or does it need to buffer all the rows, parse, and then convert to stream? Because that could explain the pause I guess.

from node-hdb.

alexpenev-s avatar alexpenev-s commented on August 18, 2024

It doesn't look to be the GC.
I am trying to reproduce the problem, but do not see such pauses.

Another guess, is the query slow. If you run it in HANA Studio do you see pauses. Could you analyze the query on the HANA side? What happens if you read the data from a LOB type.
The driver isn't caching batches of data, it should stream the results right out. You shouldn't have to tweak the driver.

from node-hdb.

surjikal avatar surjikal commented on August 18, 2024

Ok, I reran the queries to a server closer to the database, and things are streaming right out. Before, the database was being accessed through a VPN. So, I guess the VPN is the bottleneck here. Seems obvious now....

By the way, this problem was happening at any FETCH_SIZE, but the lower the FETCH_SIZE, the shorter the pause.

Just so I understand, can you explain why a slower connection / lower bandwidth (i.e. my VPN) would cause pauses in the stream? I would expect to see every chunk come slower. In the end I guess it's equivalent, I'm just curious and want to learn more about the driver internals.

from node-hdb.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.