andresp / cablemodem-status

Python based DOCSIS modem performance metrics collection tool using the built-in HTML portal of common devices.

Cable Modem Status Retriever

This script retrieves channel information from a cable modem and stores time series data to an InfluxDB as well as Event Logs in Loki. The data can be visualized using tools such as Grafana.

Supported Modems

• Motorola MB8600
• Netgear CM2000
• Technicolor XB7
• Arris International Touchstone TG3492, configured as Giga Connect Box for UPC.CH Switzerland

Dependencies

• Python3
• An InfluxDB 2.0 database
• Loki for Event Log collection

Installation

Build and install with pip

pip3 install -e .

Configuration

Edit configuration.ini and fill in your information for the InfluxDB and the modem.

Valid strings for ModemType:

• MotorolaMB8600
• NetgearCM2000
• TechnicolorXB7
• TouchstoneTG3492UPCCH

Executing the script

Test the script by executing it manually and verify it completes without error messages.

retriever

Development / Contributing

Install all dev dependencies: pip install .[dev] .

Run unittests: pytest tests

Set up recurring execution

Create a cron job (executes every 2 minutes):

*/2 * * * * /usr/local/bin/retriever 2>&1 > /dev/null

Docker

You can build your own image by running:

docker build -f "Dockerfile" -t cablemodemstatus:latest .

Once built, create a directory on your host for your configuration.ini file, such as /opt/cablemodemstatus. Then start a container referring to the configuration directory and mapping it to /app/data:

docker run -d --name cablemodemstatus --restart unless-stopped -v /opt/cablemodemstatus:/app/data cablemodemstatus:latest

You can monitor the container's status by running:

docker logs -f cablemodemstatus

Docker compose

If you are using Docker Compose, building and installation is even easier.

Assuming your configuration file is

/home/pi/config/configuration.ini

then all you need to add to your compose.yaml file is this:

  cablemodem_exporter:
     build: https://github.com/andresp/cablemodem-status.git
     command: retriever
     volumes:
       - /home/pi/config:/app/data:ro

Todo

• Implement modules for additional modems:
  • Arris S33
  • Arris SB8200
• Provide a Grafana Dashboard to visualize collected data
• Provide turnkey instructions to set up InfluxDB, Grafana and retrieval script
• Create Prometheus storage writer

cablemodem-status's Issues

Add support for the TC4400 modem

Login to CM2000 - Failing :: loginUrl = loginForm[0].get("action")

Doesn't look like the login function can reliability pull the cookie from the CM200.

INFO:app:Logging into modem
[] <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Added :: print(loginPage.select("#target"))
Traceback (most recent call last):
File "/Volumes/WinBucket/files/script-dev/CableModem/retriever.py", line 22, in wrapper
return job_func(*args, **kwargs)
File "/Volumes/WinBucket/files/script-dev/CableModem/retriever.py", line 42, in collectionJob
modem.login()
File "/Volumes/WinBucket/files/script-dev/CableModem/modems/netgear_cm2000.py", line 148, in login
loginUrl = loginForm[0].get("action")
IndexError: list index out of range

~~
Tried to loop through the login process until it is != [] .. Need to debug some more to see if I can figure out what is going on ..

while loginPage.select("#target") == []:
# response = []
time.sleep(10)
response = self.session.get(self.baseUrl, verify=False)
loginPage = BeautifulSoup(response.content, features="lxml")
print(loginPage.select("#target"))

~~

--|--

Cable Firmware Version	V8.01.02

Support for Influx DB 1.x

The InfluxDBClient has backwards compatibility built in for Influx DB 1.x. However, if you are not using https on your InfluxDB instance, and you set UseTls = False, things still won't work since https is hardcoded into the code.

To workaround this, change the format of the URL in line 91 from this:
influxUrl = "https://" + influxHost + ":" + influxPort
to this:
influxUrl = "http://" + influxHost + ":" + influxPort

Flux Query for Grafana and Visualizations

I managed to get this working with an XB8. It's not much different than XB7.

However, I'm very new to InfluxDB and Grafana.
Can you please share the queries and visualizations you're using for this use case?
I believe it's something you have listed as one the TODOs

Issues with Selenium and TouchstoneTG3492UPCCH even if not using this model

Based on the way this program is written, even if you are not using TouchstoneTG3492UPCCH then you don't need to utilize the Selenium functionality. However, when leveraging another model of the modem, the script will still error out.

Quick fix is to comment out the model number around line 99 of retriever.py

modems = {
    "MotorolaMB8600": MotorolaMB8600(config, dbClient, consoleLogger),
    "NetgearCM2000": NetgearCM2000(config, dbClient, consoleLogger),
    "TechnicolorXB7": TechnicolorXB7(config, dbClient, consoleLogger),
#    "TouchstoneTG3492UPCCH": TouchstoneTG3492UPCCH(config, dbClient, consoleLogger)
}

Error presented looks something like this:

[root@server01 cablemodem-status]$ python3 retriever.py 
INFO:app:Reading configuration
INFO:app:Connecting to InfluxDB
INFO:app:Connecting to Selenium remote
INFO:selenium.webdriver.remote.remote_connection:Could not connect to port 4444 on host seleniumhost
INFO:selenium.webdriver.remote.remote_connection:Could not get IP address for host: seleniumhost
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fbdd0a19080>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /wd/hub/session
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fbdd0a19390>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /wd/hub/session
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fbdd0a19400>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /wd/hub/session
Traceback (most recent call last):
  File "/home/root/.local/lib/python3.6/site-packages/urllib3/connection.py", line 175, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/home/root/.local/lib/python3.6/site-packages/urllib3/util/connection.py", line 72, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib64/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/root/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 710, in urlopen
    chunked=chunked,
  File "/home/root/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/home/root/.local/lib/python3.6/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib64/python3.6/http/client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/lib64/python3.6/http/client.py", line 974, in send
    self.connect()
  File "/home/root/.local/lib/python3.6/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/home/root/.local/lib/python3.6/site-packages/urllib3/connection.py", line 187, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fbdd0a19518>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "retriever.py", line 99, in <module>
    "TouchstoneTG3492UPCCH": TouchstoneTG3492UPCCH(config, dbClient, consoleLogger)
  File "/home/root/cablemodem-status/modems/touchstone_tg3492_upc_ch.py", line 23, in __init__
    self.browser = Remote(seleniumUri, DesiredCapabilities.CHROME)
  File "/home/root/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
    self.start_session(capabilities, browser_profile)
  File "/home/root/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/home/root/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 319, in execute
    response = self.command_executor.execute(driver_command, params)
  File "/home/root/.local/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 374, in execute
    return self._request(command_info[0], url, body=data)
  File "/home/root/.local/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 402, in _request
    resp = http.request(method, url, body=body, headers=headers)
  File "/home/root/.local/lib/python3.6/site-packages/urllib3/request.py", line 79, in request
    method, url, fields=fields, headers=headers, **urlopen_kw
  File "/home/root/.local/lib/python3.6/site-packages/urllib3/request.py", line 170, in request_encode_body
    return self.urlopen(method, url, **extra_kw)
  File "/home/root/.local/lib/python3.6/site-packages/urllib3/poolmanager.py", line 376, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/home/root/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 828, in urlopen
    **response_kw
  File "/home/root/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 828, in urlopen
    **response_kw
  File "/home/root/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 828, in urlopen
    **response_kw
  File "/home/root/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 788, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/root/.local/lib/python3.6/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='seleniumhost', port=4444): Max retries exceeded with url: /wd/hub/session (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fbdd0a19518>: Failed to establish a new connection: [Errno -2] Name or service not known',))
Exception ignored in: <bound method TouchstoneTG3492UPCCH.__del__ of <modems.touchstone_tg3492_upc_ch.TouchstoneTG3492UPCCH object at 0x7fbdd0a87c50>>
Traceback (most recent call last):
  File "/home/root/cablemodem-status/modems/touchstone_tg3492_upc_ch.py", line 28, in __del__
AttributeError: 'TouchstoneTG3492UPCCH' object has no attribute 'browser'
[root@server01 cablemodem-status]$