xDSL graphics throw error if not connected about tch-nginx-gui

[Ansuel]

can you test latest dev?

[meyergru]

With 7.13.18, there is now an error message in the GUI and this logread output:

Sun May 13 22:07:10 2018 daemon.err transformer[13316]: [truncated] getter(rpc.xdslctl., Us1BandInitial) threw an error: /usr/share/transformer/mappings/rpc/xdslctl.map:273: attempt to use a closed file
stack traceback:
        [C]: in function 'close'
        /usr/share/transformer/mappings/rpc/xdslctl.map:273: in function </usr/share/transformer/mappings/rpc/xdslctl.map:248>
        (tail call): ?
        [C]: in function 'xpcall'
        /usr/lib/lua/transformer/navigation.lua:169: in function 'get_parameter_value'
        /usr/lib/lua/transforSun May 13 22:07:10 2018 daemon.err nginx: 2018/05/13 22:07:10 [error] 13306#0: *9 lua entry thread aborted: runtime error: [string "/modals/diagnostics-xdsl-graphics-modal.lp"]:24: attempt to index a nil value
stack traceback:
coroutine 0:
        [string "/modals/diagnostics-xdsl-graphics-modal.lp"]:24: in function 'content'
        /usr/lib/lua/web/web.lua:249: in function 'process'
        content_by_lua(nginx.conf:163):4: in function <content_by_lua(nginx.conf:163):1>, client: 192.168.10.97, server: localhost, request: "GSun May 13 22:07:12 2018 daemon.notice wansensing: (L2Sense) runs L2Main.check(timeout)

[Ansuel]

try now

[meyergru]

Now there are no log errors, but the waiting popup keeps spinning forever:

[Ansuel]

https://www.youtube.com/watch?v=PGNiXGX2nLU

anyway...

give me the browser log... it should point to some javascript error

[meyergru]

Correct:

[Ansuel]

click on more information... on the right it should see something clickable with #:

[meyergru]

[Ansuel]

can you give me the output of this

transformer-cli get rpc.xdslctl.HlogScanBand

[meyergru]

Sure, without DSL connection, it is:

rpc.xdslctl.HlogScanBand [string] = 4094

Interesting, with a DSL connection, that gives:

rpc.xdslctl.HlogScanBand [string] = 4096

[Ansuel]

ok try now without and with dsl

[meyergru]

Did you upload a new GUI_dev.tar.bz2 to the forum repo? It still is 7.13.19?

[Ansuel]

check now

[meyergru]

Without DSL still spinning and this Javascript error (no logread entries):

[Ansuel]

now should be good

[meyergru]

Nope:

[Ansuel]

are you able to give me the javascript of the graph ?

should be something like

var SNRlen = SNRChart.data.datasets[0].data.length;
var BitsLen = BitsChart.data.datasets[0].data.length;

[meyergru]

How can I do that? I have tried Firefox Inspector Watch Expression, but that says:

SNRChart.data.datasets[0].data.length: "TypeError: SNRChart.data is undefined"

[meyergru]

Oh, with Chrome I got this:

And when I follow the link:

which should explain the missing ( before the condition...

[Ansuel]

try now

[Ansuel]

(anyway i'm modifying something so if the graph doesn't go with connection you know why...

[meyergru]

7.13.22:

Works without DSL (empty graphs).

Error with DSL (as you said):

No Javacript error, but logread:

Mon May 14 00:20:36 2018 daemon.err nginx: 2018/05/14 00:20:36 [error] 7760#0: *8 lua entry thread aborted: runtime error: [string "/modals/diagnostics-xdsl-graphics-modal.lp"]:28: attempt to index a nil value
stack traceback:
coroutine 0:
        [string "/modals/diagnostics-xdsl-graphics-modal.lp"]:28: in function 'content'
        /usr/lib/lua/web/web.lua:249: in function 'process'
        content_by_lua(nginx.conf:163):4: in function <content_by_lua(nginx.conf:163):1>, client: 192.168.7.2, server: localhost, request: "GET Mon May 14 00:21:31 2018 daemon.err nginx: 2018/05/14 00:21:30 [error] 7760#0: *10 lua entry thread aborted: runtime error: [string "/modals/diagnostics-xdsl-graphics-modal.lp"]:28: attempt to index a nil value
stack traceback:
coroutine 0:
        [string "/modals/diagnostics-xdsl-graphics-modal.lp"]:28: in function 'content'
        /usr/lib/lua/web/web.lua:249: in function 'process'
        content_by_lua(nginx.conf:163):4: in function <content_by_lua(nginx.conf:163):1>, client: 192.168.7.2, server: localhost, request: "GETMon May 14 00:22:40 2018 authpriv.info dropbear[7875]: Child connection from 192.168.7.2:56312
Mon May 14 00:22:40 2018 authpriv.notice dropbear[7875]: Pubkey auth succeeded for 'root' with key md5 71:fc:e6:9e:b1:33:9a:b0:05:55:a9:d2:3d:7b:d7:29 from 192.168.7.2:56312
Mon May 14 00:23:45 2018 daemon.err nginx: 2018/05/14 00:23:45 [error] 7760#0: *12 lua entry thread aborted: runtime error: [string "/modals/diagnostics-xdsl-graphics-modal.lp"]:28: attempt to index a nil value
stack traceback:
coroutine 0:
        [string "/modals/diagnostics-xdsl-graphics-modal.lp"]:28: in function 'content'
        /usr/lib/lua/web/web.lua:249: in function 'process'
        content_by_lua(nginx.conf:163):4: in function <content_by_lua(nginx.conf:163):1>, client: 192.168.7.2, server: localhost, request: "GET

[Ansuel]

yes yes ignore for now ;) i'm making some change the qln and hlog miss some data

[meyergru]

While you are at the graphs: The last one with the bits is scrollable and has a different width than all the others. Because it is scaled out like that, it also has gaps between some bars.

[Ansuel]

well limitation of the library that i use.... if i set a lower width the data disappear.

[Ansuel]

anyway now it should work

[meyergru]

It works now with DSL connected, but with DSL off still spinning:

I once had an error in PHP, when I output a boolean - matter-of-fact it showed that 'true' is represented by '1' but false not by '0', but by ''. So I had to use $var ? '1' : '0'.

With regard to the graphs: Why is only the last graph scrollable? Also a limitation? It would be better if all the graphs were synchronized, regardless of the absolute width.

[meyergru]

Wait!

I have bad news and good. First the bad:

There is another bug when DSL is connected which I have never noticed because of the wide Bits graph: A few of the graphs (namely HLog and QLN) have duplicated data from sample frequency #2800 upwards:

The data is truncated at the same frequency for SNR and Bits graphs:

I believe there are errors in xdslctl.map where the band ranges are calculated that cause these errors.

Now for the good news:

Apart from the truncated data, I have found a way to synchronize and shorten the Bits graph. Just use a 'line' chart with the same definitions as the first three charts and then fill the area. This is done like so:

var bits_ctx = document.getElementById("BitsChart").getContext("2d");\
        var BitsChart = new Chart(bits_ctx, {\
    type: "line",\
    data: {labels: labelLegend,\
        "datasets":[{"data":['); ngx.print(Bits); ngx.print('],\
        "pointBackgroundColor": pointBackgroundColor,\
        "backgroundColor":"#CCFFFF",\
        "lineTension":0.1,\
        "pointBorderWidth":"0.6",\
        "pointRadius":"0.6",\
        "fill":"start",\
    }]},\
    options: {\
                maintainAspectRatio: false,\
        scales: {\
            yAxes: [{\
                ticks: {\
                    beginAtZero:true,\
                }\
            }]\
        },\
                spanGaps:false,\
                showLines:true,\
                pointStyle:"cross",\
                legend: {display: false},\
                animation: {\
            duration: 0,\
        },\
        hover: {\
            animationDuration: 0,\
        },\
        responsiveAnimationDuration: 0,\
    }\
});\

And looks like this:

I also think that SNRlen and BitsLen should always be equal, so the whole 'backgroundColor' array could be removed if you make that change and correct the duplication band error.

[Ansuel]

you have 35b right?

[meyergru]

No, 17a, Deutsche Telekom VVDSL.

[Ansuel]

can you paste here

xdslctl info --pbParams

[meyergru]

xdslctl: ADSL driver and PHY status
Status: Showtime
Last Retrain Reason:    0
Last initialization procedure status:   0
Max:    Upstream rate = 40631 Kbps, Downstream rate = 132532 Kbps
Bearer: 0, Upstream rate = 40631 Kbps, Downstream rate = 109342 Kbps
Bearer: 1, Upstream rate = 0 Kbps, Downstream rate = 0 Kbps
Discovery Phase (Initial) Band Plan
US: (28,60) (871,1205) (1972,2771)
DS: (65,859) (1216,1961) (2793,3943)
Medley Phase (Final) Band Plan
US: (28,60) (871,1205) (1972,2771)
DS: (65,859) (1216,1961) (2793,3943)
VDSL Port Details               Upstream                Downstream
Attainable Net Data Rate:        40631 kbps             132532 kbps
Actual Aggregate Tx Power:      -   0.2 dBm                12.7 dBm
====================================================================================
VDSL Band Status        U0      U1      U2      U3      U4      D1      D2      D3
  Line Attenuation(dB):  3.0     12.9    19.9     N/A     N/A    8.6     17.9    27.7
Signal Attenuation(dB):  3.0     12.5    18.9     N/A     N/A    9.6     17.7    27.7
        SNR Margin(dB):  9.2     9.3     9.3      N/A     N/A    13.2    12.2    12.1
        TX Power(dBm):  -10.1   -2.0    -6.5      N/A     N/A    8.8     7.6     7.1
        kl0(dBx100):    180     180       N/A     N/A     N/A   673     681     719

Ahhh, I see. It ends at 2793. So the clipping is O.K., but not the duplication. Also, the graphs should be limited to 2793 instead of 4096.

[meyergru]

Nope. The Bits bins should end at 3943 (that graph is from a similar connection with a Fritzbox):

So, the data ranges are clipped and/or duplicated...

[Ansuel]

anyway i don't really like the blue background... we should find a way to color it..

[meyergru]

You can do it, but not with truncation. If you use "backgroundColor: pointBackgroundColor", the result is broken at this time. I think this is because of different lengths of arrays.

When you leave backgroundColor at default, it will be light grey. One can fine-tune the styling, the main idea is to remedy the limitations of the "bar" chart type.

One other possibility would be to use less samples (use only every Nth sample).

[Ansuel]

notice the same thing... actually, i should find a way to split all the graph

[meyergru]

I think the sheer amount of samples is what limits the bar chart type?

Maybe the "individual" coloring works better if the array lengths of data samples and coloring are in sync?

[Ansuel]

try this version and tell me what you see

[meyergru]

Oh, I have another idea: If you separate the dataset into upstream and downstream, you could overlay them and use one coloring for each set!!! That way, you would need no color attributes per data point, but only per set!

This way, the area coloring for "line" graph type should work!

[Ansuel]

yhea could work... but i can't find a way to show the data one after another.... (as i don't really like to fill the start of the data of Nan)

i think that this is the only way...

create 2 dataset... gonna be a lot of code to create the right dataset :( but is doable

[meyergru]

7.14.2: Still truncated / duplicated values and white background below Bits graph, but sometimes switches to transparent when cursor moves.

Yes, doing that changes the logic of the data acquisition, but it could be applied to the other graphs as well and should also be much faster... the whole timeout recoloring could be abandoned...

[Ansuel]

ok now we have 2 datasets one fro us band and one for ds

[meyergru]

Uploaded to forum repo already?

[Ansuel]

nope

[Ansuel]

if it works well you will love this...

hope for the best now it on the repo ;)

needs to find a way to speed up the loading...

[meyergru]

Nice, even the "DSL disconnected" bug is gone!

Two problems remain:

1. Still the wraparound / duplicate values at #2983:
   

2. A little slow (neglegible).

[Ansuel]

the slowness
the duplicate bug is very strange

i need the output of some command

[Ansuel]

transformer-cli get rpc.xdslctl.Us1BandFinal

transformer-cli get rpc.xdslctl.Us3BandFinal

transformer-cli get rpc.xdslctl.Ds3BandFinal

[meyergru]

I'd say they are as expected:

# xdslctl info --pbParams
xdslctl: ADSL driver and PHY status
Status: Showtime
Last Retrain Reason:    0
Last initialization procedure status:   0
Max:    Upstream rate = 40631 Kbps, Downstream rate = 131144 Kbps
Bearer: 0, Upstream rate = 40631 Kbps, Downstream rate = 109342 Kbps
Bearer: 1, Upstream rate = 0 Kbps, Downstream rate = 0 Kbps
Discovery Phase (Initial) Band Plan
US: (28,60) (871,1205) (1972,2771)
DS: (65,859) (1216,1961) (2793,3943)
Medley Phase (Final) Band Plan
US: (28,60) (871,1205) (1972,2771)
DS: (65,859) (1216,1961) (2793,3943)
VDSL Port Details               Upstream                Downstream
Attainable Net Data Rate:        40631 kbps             131144 kbps
Actual Aggregate Tx Power:      -   0.2 dBm                12.8 dBm
====================================================================================
VDSL Band Status        U0      U1      U2      U3      U4      D1      D2      D3
  Line Attenuation(dB):  3.0     12.9    19.9     N/A     N/A    8.6     17.9    27.7
Signal Attenuation(dB):  3.0     12.5    19.0     N/A     N/A    9.6     17.7    27.7
        SNR Margin(dB):  9.2     9.2     9.3      N/A     N/A    13.1    11.5    11.7
        TX Power(dBm):  -10.1   -2.0    -6.5      N/A     N/A    8.8     7.6     7.1
        kl0(dBx100):    180     180       N/A     N/A     N/A   673     681     719
# transformer-cli get rpc.xdslctl.Us1BandFinal
rpc.xdslctl.Us1BandFinal [string] = 60
# transformer-cli get rpc.xdslctl.Us3BandFinal
rpc.xdslctl.Us3BandFinal [string] = 2771
# transformer-cli get rpc.xdslctl.Ds3BandFinal
rpc.xdslctl.Ds3BandFinal [string] = 3943

[Ansuel]

transformer-cli get rpc.xdslctl.Ds4BandFinal
transformer-cli get rpc.xdslctl.Us4BandFinal

transformer-cli get rpc.xdslctl.Ds4ScannedBand
transformer-cli get rpc.xdslctl.Us4ScannedBand

[meyergru]

#transformer-cli get rpc.xdslctl.Ds4BandFinal
rpc.xdslctl.Ds4BandFinal [string] = 0
#transformer-cli get rpc.xdslctl.Us4BandFinal
rpc.xdslctl.Us4BandFinal [string] = 0
#transformer-cli get rpc.xdslctl.Ds4ScannedBand
rpc.xdslctl.Ds4ScannedBand [int] = 0
#transformer-cli get rpc.xdslctl.Us4ScannedBand
rpc.xdslctl.Us4ScannedBand [int] = 0

[Ansuel]

transformer-cli get rpc.xdslctl.HlogDs4
transformer-cli get rpc.xdslctl.HlogUs4

this?

[meyergru]

#transformer-cli get rpc.xdslctl.HlogDs4
rpc.xdslctl.HlogDs4 [string] =
#transformer-cli get rpc.xdslctl.HlogUs4
rpc.xdslctl.HlogUs4 [string] =

Both empty

[meyergru]

Does this help?

# transformer-cli get rpc.xdslctl.HlogUs1 | tr -cd ',' | wc -c
61
# transformer-cli get rpc.xdslctl.HlogUs2 | tr -cd ',' | wc -c
346
# transformer-cli get rpc.xdslctl.HlogUs3 | tr -cd ',' | wc -c
810
# transformer-cli get rpc.xdslctl.HlogDs1 | tr -cd ',' | wc -c
799
# transformer-cli get rpc.xdslctl.HlogDs2 | tr -cd ',' | wc -c
756
# transformer-cli get rpc.xdslctl.HlogDs3 | tr -cd ',' | wc -c
3943

I think the last one is strange?

[Ansuel]

O_O wtf !

[Ansuel]

transformer-cli get rpc.xdslctl.Ds3BandInitial ?

[meyergru]

#transformer-cli get rpc.xdslctl.Ds3BandInitial
rpc.xdslctl.Ds3BandInitial [string] = 2793
#transformer-cli get rpc.xdslctl.Ds3BandFinal
rpc.xdslctl.Ds3BandFinal [string] = 3943

[Ansuel]

OWWW OK I KNOW THE PROBLEM!

[Ansuel]

now should be good

[meyergru]

logread error:

Mon May 14 22:03:40 2018 daemon.err nginx: 2018/05/14 22:03:40 [error] 20453#0: *6 lua entry thread aborted: runtime error: [string "/modals/diagnostics-xdsl-graphics-modal.lp"]:86: unexpected symbol near 'then'
stack traceback:
coroutine 0:
        [C]: in function 'error'
        /usr/lib/lua/web/lp.lua:77: in function 'compile'
        /usr/lib/lua/web/lp.lua:153: in function 'load'
        /usr/lib/lua/web/web.lua:203: in function 'getContentRenderer'
        /usr/lib/lua/web/web.lua:240: in function 'process'
        content_by_lua(nginx.conf

[Ansuel]

my bad fixed

[meyergru]

If it was the superfluous "and" in line 86, then it is fixed...

[Ansuel]

yes was that ahhaha

[meyergru]

A huge step forward for mankind:

[Ansuel]

now pls check if it's the same of the one provided from dslstats or other graph

[meyergru]

Awesome, Ansuel! Good night...

[meyergru]

Concerning speed: the "BandPlanCondition" part is obsolete now.

[Ansuel]

??? explain pls oow ok you are right! but the thing that slow down it is the generation of the dummy value for the band... i should move them to javascript so that is the browser that creates the value and not the lua page that gets regenerated every time

[meyergru]

There is a section in /www/docroot/modals/diagnostics-xdsl-modal.lp that populates the variable BandPlanCondition which is now obsolete. It was used before, when you needed to set the backgroundColor for every single data element. That section, including the rpc call is not needed any more (starting at line #167 or so).

There are also two more arrays at line #258 that are leftovers: pointBackgroundColor and backgroundColor.

[Ansuel]

notice that... :) now i will try to move the generation of the graph from lua to javascript

[meyergru]

Do you think that is faster? Probably, because then you have a desktop CPU...

One thing is also that there are 32 rpcs, each of which has a turnaround overhead. When I do this, you can see it:

# time transformer-cli get rpc.xdslctl.HlogDs3 >/dev/null
real    0m 0.17s
user    0m 0.02s
sys     0m 0.01s

There is only 0.03s CPU time, but 0.17s in order to get the data via rpc. This times 32 is already 5s, that is half of the whole time to display the graphs now.

If you decompose the bands in Javascript, there would only be 5 rpcs(one to get the band limits and 4 for each data series).

[Ansuel]

at least with javascript i can generate the datasets after all the rpc data loads (and also the page) with settimeout

but you are right... we should find a way to speed up also that... would be good to underestand what is the real slowdown in the process... (and i think is the compilation of the lua page)

[meyergru]

If it is the compilation, it would also be good to combine the data at the topmost Javascript level in order to save the raw number of Lua calls (either by Javascript callbacks to Lua or by Lua calling rpcs, which are Lua again).

BTW: I saw that you snip off the header lines of the "xdslctl info" output via "sed -n" in the rpc code. That implies one more process fork per call. It would be better to do this by language means (i.e. skip n lines after file opening). Like this:

local function xdslband(n,dtype,bandtype)
        local line
        if ( bandtype == "Us" ) then
                line = "11"
        else
                line = "12"
        end
        local cmd = ("xdslctl info --pbParams")
        local f = assert(io.popen(cmd, 'r'))
        for i = 1, line do
                assert(f:read())
        end
        local s = assert(f:read())
        local r = "0"
        local c = 1
        f:close()
        s = string.gsub(s, '[\n\r]+', ' ')
        for w in string.gmatch (s, "%((.-)%)") do
                if c == n then
                        if dtype == "Initial" then
                                r = string.match(w,"(.-)%,")

[Ansuel]

umm so a for loop is less heavy then execute a direct command?

[meyergru]

Normally, yes.

Even with busybox, sed is not an internal command but a symlink to another busybox invocation. When you look at ps at the correct time, you should see a separate sed instance.

A system exec is a heavy operation: program loading from disk (albeit cached), parsing of ELF headers, relocation of external symbols including loading of dynamic libraries, connection of the pipe.
And after all that, sed has to parse its command line and then do essentially the same as the explicit loop above.

[Ansuel]

ummm made some changes, any idea to optimize it even more, just tell me ?

[meyergru]

It is now at ~8s instead of ~10s. I think you could only shave off another 4 seconds by actually calling each graph rpc only once instead of 8 times. As I said, each call is ~0.17s in the backend.

You would have to do the band separation all in the frontend in Javascript, though.

[Ansuel]

umm also i found another slow down...

i call the same data over and over again... like 30 times... could be that the problem.

[meyergru]

That is what I said. 8 bands (Us1-4, Ds1-4) times 4 graphs data (Hlog, SNR, QLN, Bits) = 32 calls times 0.17s = 5 seconds. If you do it only once for each graph and separate the band in Javascript, you should save ~4s.

[Ansuel]

ummm not talking about that... but about us and bs band for the for cicle

[meyergru]

Yes, I know. That is what I meant. 8 times for all bands times 4 different graphs. Do it once for every of the 4 graphs and split up the data into bands in Javascript.

[Ansuel]

try this version any improvement?

[meyergru]

7.14.33 = 7 seconds instead of 8.

[Ansuel]

ummm i found another bottleneck... i think... one sec...

[Ansuel]

try this

[meyergru]

3.2s! What was it? I doubt you have saved all the calls already...

[Ansuel]

make more test pls... this result it's too strange

[meyergru]

3287ms for the delivery when clicking on dianostics-xdsl-graphics-modal.lp. Around 6s until it's displayed.

[Ansuel]

see changes, i was requsting the same thing every time like an idiot...
what to optimize now (except moving all to javascript...)

[meyergru]

Subjectively, it is fast enough now. It is a diagnostic function after all - no need to animate it ;-)

[Ansuel]

only problem is underestand why is offset of 3-4 value...

[meyergru]

I have seen the strange code but did not understand it. I know, though, that it fixes the wraparound/cutoff problem at 2800.

[nclmrc]

Anche io ho quest'errore

[meyergru]

Yup, I did not try without DSL connection yet. logread shows:

Thu May 17 00:06:11 2018 daemon.err transformer[29861]: [truncated] getter(rpc.xdslctl., Ds1ScannedBand) threw an error: /usr/share/transformer/mappings/rpc/xdslctl.map:301: attempt to index local 's' (a nil value)
stack traceback:
        /usr/share/transformer/mappings/rpc/xdslctl.map:301: in function 'xdslband'
        /usr/share/transformer/mappings/rpc/xdslctl.map:328: in function </usr/share/transformer/mappings/rpc/xdslctl.map:323>
        (tail call): ?
        [C]: in function 'xpcall'
        /usr/lib/lua/transformer/navigation.luThu May 17 00:06:11 2018 daemon.err nginx: 2018/05/17 00:06:11 [error] 29851#0: *13 lua entry thread aborted: runtime error: [string "/modals/diagnostics-xdsl-graphics-modal.lp"]:12: attempt to index a nil value
stack traceback:
coroutine 0:
        [string "/modals/diagnostics-xdsl-graphics-modal.lp"]:12: in function 'content'
        /usr/lib/lua/web/web.lua:249: in function 'process'
        content_by_lua(nginx.conf:163):4: in function <content_by_lua(nginx.conf:163):1>, client: 192.168.10.97, server: localhost, request: "Thu May 17 00:06:13 2018 daemon.notice wansensing: (L2Sense) runs L2Main.check(timeout)

Should be easy to fix...

[Ansuel]

? explain better

[meyergru]

My timing experiments are happening on a connected device, so I did not test on my unconnected one (I have two DGAs).

On the unconnected device, there is the same error as initially: When DSL is not connected, the error above is shown on the GUI. Reason being that now you use a string to hold the parsed line for the band limits. And this is NULL with no connection, giving that logread error.

This is an error introduced by the refactorings for speed.

[Ansuel]

actually the error is about another thing
try this version

[meyergru]

Works both with connection & without.

[meyergru]

7.14.35 worked fine, so we can close this now.