nebraltd / hm-diag Goto Github PK
View Code? Open in Web Editor NEWHelium Miner Diagnostics
Home Page: https://nebra.io/hnt
License: MIT License
Helium Miner Diagnostics
Home Page: https://nebra.io/hnt
License: MIT License
In order to enrich the data in the dashboard, we need to send in some data that is only available on the device(s). We do however need to do this in a secure fashion. Here's how I am thinking this would work as of right now:
While this all sounds simple, the devil is in the details. Credentials will be the trickiest part to solve, but here's how I'm currently envisioning it working:
DASHBOARD_TOKEN
).
Look into implementing QR code onboarding using the diagnostics page
Would be useful to have the total current blockchain height and a percentage.
I.e.
12345/12345 (100%)
According to the local diagnostics the hotspot is running on hotspot verison 2021.06.10.0 but when pairing in the app it is still showing firmware version 2021.06.09.4
Open helium address link in new page
We want to make sure all shell scripts pass ShellCHeck, so let's add a CI run for this and fix them.
Request the same diagnostics information from the miner that the diagnostics function in the app uses.
This would then include:
If the hotspot thinks it is connected to the helium network.
Weather the hotspot is "dialable" (I'm guessing not relayed)
The height of the hotspot
And internet conneection type.
Does the "waiting for miner" show if the unit has not been asserted?
It would be good to see the region plan once it is set on the Diagnostics.
Was talking to Kevin and we had an idea that maybe we could add some error log view/download to the diagnostics to help with debugging, specifically the miner error log...
We had a customer who says they were having issues when port forwarding on a ubiquiti router due to port forwarding and UPnP being active. Thought maybe it might be useful to have a setting in diagnostics to deactivate upnp on the miner.
At the bottom it has "last updated" time. We should clarify that this is UTC
Currently the diagnostics shows UN123 when no frequency plan is detected. Make this more user friendly.
I deployed master to testnet, and it's now crashing with the following error:
diagnostics Traceback (most recent call last):
diagnostics File "/opt/utils.py", line 122, in writing_data
diagnostics with open(path, 'w') as file:
diagnostics FileNotFoundError: [Errno 2] No such file or directory: '/opt/nebraDiagnostics/html/diagnostics.json'
diagnostics
diagnostics During handling of the above exception, another exception occurred:
diagnostics
diagnostics Traceback (most recent call last):
diagnostics File "/opt/main.py", line 254, in <module>
diagnostics main()
diagnostics File "/opt/main.py", line 245, in main
diagnostics write_info_to_files(prod_diagnostics, diagnostics)
diagnostics File "/opt/main.py", line 128, in write_info_to_files
diagnostics utils.writing_data(path, data)
diagnostics File "/opt/utils.py", line 125, in writing_data
diagnostics raise FileNotFoundError(
diagnostics FileNotFoundError: Directory does not exist in the path: /opt/nebraDiagnostics/html/diagnostics.json
I have since reverted the update.
Sentry Issue: HM-DIAG-7
ValueError: invalid literal for int() with base 10: ''
File "main.py", line 146, in <module>
if(int(diagnostics['MH']) > (int(diagnostics['BCH'])-500)):
I think it can be very handy to show IPv4 and IPv6 addresses leased / assigned to the device for each interface on the diagnostic website.
THX
It looks like the diagnostics tool is running just fine, but we're getting this error:
diagnostics ERROR:root:[Errno 2] No such file or directory: '/sys/class/net/wlan0/address'
diagnostics grep: /sys/bus/usb/devices/*/idVendor: No such file or directory
diagnostics grep: /sys/bus/usb/devices/*/idVendor: No such file or directory
We need to improve the error handling for this.
Related to #20
also saw this one...
03.06.21 02:43:06 (+0100) diagnostics Traceback (most recent call last):
03.06.21 02:43:06 (+0100) diagnostics File "/opt/nebraDiagnostics/main.py", line 154, in <module>
03.06.21 02:43:06 (+0100) diagnostics diagnostics['BSP'] = round(((int(diagnostics['MH'])/int(diagnostics['BCH']))*100),3)
03.06.21 02:43:06 (+0100) diagnostics ZeroDivisionError: division by zero
03.06.21 02:43:10 (+0100) diagnostics rm: can't remove '/opt/nebraDiagnostics/html/index.html': No such file or directory
03.06.21 02:43:10 (+0100) diagnostics rm: can't remove '/opt/nebraDiagnostics/html/initFile.txt': No such file or directory
03.06.21 02:43:12 (+0100) diagnostics Diag Loop
Error log:
21.04.21 21:17:59 (+0100) diagnostics Traceback (most recent call last):
21.04.21 21:17:59 (+0100) diagnostics File "/opt/nebraDiagnostics/diagnosticsProgram.py", line 55, in <module>
21.04.21 21:17:59 (+0100) diagnostics diagnostics["RPI"] = open("/proc/cpuinfo")\
21.04.21 21:17:59 (+0100) diagnostics IndexError: list index out of range
I think I got the serial number by going to a specific line which works on the 3, 3+, CM3s and 4s but as the Zero is only single core it fails.
Need to tweak so it gets it slightly differently.
https://github.com/NebraLtd/hm-diag/blob/master/diagnostics-program/diagnosticsProgram.py#L55
Let's rewrite the diagnostics page to use Flask, which gives us a lot of free features out of the box. We should then serve this app using Gunicorn.
To keep things simple, let's ditch the Nginx container for the time being and serve Gunicorn on port 8000 with Docker exposing it on port 80. That way we don't need to run the container as root.
This is a per-requisite for #39.
Is it possible to show the "Miner Relayed" row in amber if the value is true and also indicate this in the main status "All Ok" is not an accurate reflection of this state as beaconing will not work.
I have found that the diagnostic page is the only place that shows the real time relayed state and that the explorer, Helium App, and dashboard all indicate that there is no relayed state even after many hours/days.
It may be also worth considering adding a link if the hotspot is relayed to a page that shows how to fix it.
Saw this on a unit today...
03.06.21 02:32:24 (+0100) diagnostics Diag Loop
03.06.21 02:32:25 (+0100) diagnostics ERROR:dbus.proxies:Introspect error on :1.94:/: dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NotSupported: org.freedesktop.DBus.Introspectable.Introspect
03.06.21 02:32:26 (+0100) diagnostics Frequency: US915
03.06.21 02:32:26 (+0100) diagnostics
We are already linting a little bit in a GitHub action (reference), but we're not fully linting.
To possibly tighten down security it could be worth changing to a better webserver software.
Currently this is using simplehttpd which has the benefit of being extremely lightweight.
However if it is secure enough needs to be quickly looked into as it might be perfectly fine.
It is just a HTTP Server, no post or get requests are being processed either.
Alternatives include:
Nginx
Lighttpd
Apache.
Sentry Issue: HM-DIAG-A
TimeoutError: [Errno 110] Operation timed out
File "urllib3/connection.py", line 159, in _new_conn
conn = connection.create_connection(
File "urllib3/util/connection.py", line 84, in create_connection
raise err
File "urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
NewConnectionError: <urllib3.connection.HTTPSConnection object at 0xf65e3058>: Failed to establish a new connection: [Errno 110] Operation timed out
File "urllib3/connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "urllib3/connectionpool.py", line 381, in _make_request
self._validate_conn(conn)
File "urllib3/connectionpool.py", line 976, in _validate_conn
conn.connect()
File "urllib3/connection.py", line 308, in connect
conn = self._new_conn()
File "urllib3/connection.py", line 171, in _new_conn
raise NewConnectionError(
MaxRetryError: HTTPSConnectionPool(host='api.helium.io', port=443): Max retries exceeded with url: /v1/blocks/height (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0xf65e3058>: Failed to establish a new connection: [Errno 110] Operation timed out'))
File "requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "urllib3/connectionpool.py", line 724, in urlopen
retries = retries.increment(
File "urllib3/util/retry.py", line 439, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
ConnectionError: HTTPSConnectionPool(host='api.helium.io', port=443): Max retries exceeded with url: /v1/blocks/height (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0xf65e3058>: Failed to establish a new connection: [Errno 110] Operation timed out'))
(1 additional frame(s) were not displayed)
...
File "requests/api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
Some of the recent changes broke master and we are no longer writings the files properly:
/opt/html # ls
bootstrap.min.css
/opt # ls diagnostics.json
diagnostics.json
/opt # ls initFile.txt
initFile.txt
Most likely, this was introduced in #68 as this was a large rewrite.
As the QR Codes are now generated elsewhere, we can remove this code to save some space.
To make it easier to configure 4g, add APN settings configuration
More context would be helpful for new developers to understand how things are structured and major design decisions.
@shawaj Reported & I noticed that the Diagnostics page has some rouge characters.
To-Do: Remove these characters.
Perhaps a button we can get people to press that downloads a txt file with diagnostics info to send us.
Similar to what's in the app
Top of androids pages are purple. Change CSS to match nebra blue.
I think it would be nice if the remote dashboard and the local dashboard converge to have similar features from a device info / device control perspective...
Things we could add:
Probably a bunch more we could add, just wanted to get the conversation started ๐
Currently we only show the sync percentage:
This can be confusing to people new to Helium as it's actually considered synced if it's within 500 blocks and this is how Helium display it on the explorer as shown here:
https://github.com/helium/explorer/blob/6c5539bc121b49f443838fad811d35100c3772a9/components/Hotspots/StatusPill.js#L33
We should add something similar so that we don't have people worrying that they aren't quite synced
Maybe "99.999% (Synced within 500 blocks)" or similar
I think it'd be quite handy to know the version of the software running on the device via the diagnostics page.
Helium are updating the "connected" metric in p2p_status to change "connected" to "well-connected" to reflect the fact that in principle 1 connection means it is connected but this check is really meant to determine if it has a good connection.
They will also add a "sessions" count as well.
We need to update the diagnostics to accommodate these changes once they are merged helium/miner#868
The sync percentage is showing as 1.0% when it should be 100%.
I have noticed it happen since the testnet 2021.06.24.4.
Blocked by #60.
I want to give the diagnostics page a fresh new look.
Here's how it currently looks:
@vladimirpoleshko can you start mocking up a new design that goes in line with the dashboard design? Doesn't have to have the exact same design but at least have a similar feel to it to make it consistent.
@shawaj has reported on mobile that the header text gets cut off.
To-do: Tweak font size to adjust to mobiles, or cut the text down.
Since a few releases I have noticed that the height status sometimes gets higher than the displayed blockchain height and shows as "Miner Is Still Loading" in the sync percentage. This has confused a few customers.
Can we still show a 100% sync status when the height status is higher than the reported blockchain height, at least if it is less than 20 blocks?
See below:
diagnostics Diagnostics loop started...
diagnostics Traceback (most recent call last):
diagnostics File "/opt/main.py", line 404, in <module>
diagnostics main()
diagnostics File "/opt/main.py", line 267, in main
diagnostics diagnostics["OK"] = str(pk_file[3])
diagnostics IndexError: list index out of range
Looks like this changes is the issue?
NebraLtd/hm-miner@6f5f5cb#diff-0e6697c77412680c0ae2160d7471440fc3df43cf171143ed589bdb8a8f4abd99R9
Like with WiFi routers I think we should password protect this page. Had a few people exposing it to the internet and being surprised to know that it's got sensitive info on it.
We can password protect perhaps with the last 6 digits of Mac address or RasPi serial.
And we could allow people to change the password but with a factory reset option to put it back to the default.
hm-diag/diagnostics-program/main.py
Lines 193 to 194 in 992d3ef
I think we are multiplying by 100 twice again here. This was fixed in #69 / #70 but the error seems to have been reintroduced?
Is there a way we can pass these back to the diag and to the dashboard?
I think the letter f at the beginning of this line is a typo...
hm-diag/diagnostics-program/main.py
Line 166 in 8d4c300
As if the miner isn't working the animal name variable wouldn't be generated this can be removed and save a good few mb.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.