paregupt / mds_traffic_monitor

Cisco MDS monitoring using Grafana and InfluxDB

Cisco MDS Traffic Monitoring (MTM)

Cisco MDS 9000 monitoring using Grafana and InfluxDB

Global Inventory

Digital Optical Monitoring of SFP (Hottest, Coldest, Lowest RX and TX Power)

Digital Optical Monitoring of SFP

Top-10 ports (Traffic, Congestion, Errors, etc.)

Switchport monitoring

Port-Channel traffic distribution

and much more...

Architecture

The MTM collector (mds_traffic_monitor_*.py) pulls stats from Cisco MDS switches using NX-API. The stats are normalized and corrected before writing to InfluxDB. Finally, Grafana provides the visualization and use-cases.

• Data source: Cisco MDS Switches via NXAPI), read-only account is enough
• Data storage: InfluxDB, a time-series database
• Visualization: Grafana

Installation

• Tested OS: CentOS 7.x. Should work on other OS also.
• Python version: Version 3 only. Should be able to work on Python 2 also with minor modification.

DIY Installation

1. Install Telegraf
2. Install InfluxDB
3. Install Grafana
4. Download this repo in zip format or via git clone.

Configuration

Enable NX-API on MDS swithces via feature nxapi command. Enter the access details of the MDS switch in mds_group*.txt file. Refer to the file for more details on input format.

Try

$ python3 /usr/local/telegraf/mds_traffic_monitor*.py -h

Add to your telegraf.conf file as below

[[inputs.exec]]
   interval = "30s"
   commands = [
       "python3 /usr/local/telegraf/mds_traffic_monitor_high_frequency.py /usr/local/telegraf/mds_group_1.txt influxdb-lp -vv",
   ]
   timeout = "28s"
   data_format = "influx"

Create a different mds_group_.txt file for every MDS switch. Repeat the above seven lines in telegraf.conf as many times as the number of monitored MDS switches. The name of the mds_group.txt file can be changes to help you remember the switch.

Also update the global values like

  logfile = "/var/log/telegraf/telegraf.log"
  logfile_rotation_max_size = "10MB"
  logfile_rotation_max_archives = 5

Import the Grafana dashboard json files. That is all. Enjoy!

I haven't written detailed installation instructions yet. Nor do I have an OVA. If you are new to Grafana, InfluxDB and Telegraf, follow the steps from the UTM installation - Cisco UCS monitoring using Grafana, InfluxDB, Telegraf – UTM Installation. The MTM project follows the same design as the UTM project. Most of the steps are the same. Make sure to replace ucs_traffic_monitor.py by mds_traffic_monitor_*.py and ucs_domains_group_1.txt by mds_group_1.txt.

Looking for something similar to monitor Cisco UCS Servers?

Click here to check out Cisco UCS Traffic Monitoring (UTM).

mds_traffic_monitor's Issues

InfluxDB 2.0 Compatability

Just confirming this doesn't exist right now as I can't seem to get it to work. Are there any plans to make it compatible?

Data not showing in dashboards for cisco mds

I have multiple influx DB's on my instance. Where do I specify the db name for the script to write to?

Installed With UCS Traffic Monitor

Is it possible to install this with the UCS Traffic Monitor deployment?

Grafana doesn't show metrics

Hii,
I made all the install processes step by step .
In addition there is connection between MDS to Grafana.
The script succeed, but didn't created the new files.

Thanks
Aviv

mds_traffic....py file problem

Running the following command will result in an error.

# python3 /usr/local/telegraf/mds_traffic_monitor*.py -h
Traceback (most recent call last):
File "/usr/local/telegraf/mds_traffic_monitor_high_frequency.py", line 17, in
import requests
ModuleNotFoundError: No module named 'requests'

If I comment out line 17 of the .py file and run it again, I get the following error.

# python3 mds_traffic_monitor_high_frequency.py -h
Traceback (most recent call last):
File "mds_traffic_monitor_high_frequency.py", line 18, in
import urllib3
ModuleNotFoundError: No module named 'urllib3'

If I comment out both lines 17 and 18 of the py file and run it again, I don't get the error, but grafana doesn't show the information from MDS. What do you think might be causing this?

Cisco mds

MTM is featured in Pearson (Cisco Press) book on Congestion Management #100

I am thrilled to share that Pearson (Cisco Press) has published a book on Detecting, Troubleshooting, and Preventing Congestion in Storage Networks, and it is now available at a bookstore of your choice. Ed Mazurek (Principal Engineer in Cisco TAC) and I co-authored this book.

It is relevant for this community because, in many places, this book refers to case studies based on MTM. It also explains use cases and problems solved by MTM, which can't be solved otherwise.

With more than 700 pages of education, this book focuses on the practical use of technology. In addition to Fibre Channel, it also covers TCP storage networks (for iSCSI and NVMe/TCP) and lossless Ethernet networks (for RoCEv2). The most unique aspect of this book is the multiple case studies, which are based on our first-hand experience of working with customers around the world. You can learn step-by-step procedures to troubleshoot congestion in all types of storage networks.

Those who operate, design, or maintain storage networks will significantly benefit from it.

The book has the following nine chapters:

1. Introduction to congestion in storage networks
2. Understanding congestion in Fibre Channel fabrics
3. Detecting congestion in Fibre Channel fabrics
4. Troubleshooting congestion in Fibre Channel fabrics
5. Solving congestion by storage I/O performance monitoring
6. Preventing congestion in Fibre Channel fabrics
7. Congestion management in Ethernet storage networks
8. Congestion management in TCP storage networks
9. Congestion management in Cisco UCS Servers

This book is available at:

• Pearson (https://www.pearson.com/en-us/subject-catalog/p/detecting-troubleshooting-and-preventing-congestion-in-storage-networks/P200000009798/9780137887231)
• Amazon (https://a.co/d/jbFqyXC)
• Cisco Press (https://www.ciscopress.com/store/detecting-troubleshooting-and-preventing-congestion-9780137887231)
• Barnes and Nobel (https://www.barnesandnoble.com/w/detecting-troubleshooting-and-preventing-congestion-in-storage-networks-paresh-gupta/1140839033)

Grafana Version Tested

Hi paregupt,

I'm having a few issues to put the dashboards to work and I think that it's related to my grafana version. Could you please provide the versions that you tested it so I can compare here with mine?

Regards,
lzsilva-br

Include Port VSAN

Hi @paregupt

First of all I would like to thank you for your excellent work in creating this monitoring system for Cisco MDS equipment, it is helping me a lot!

I wonder if it would be possible to add the information of "vsan" to the graph of "Switch Port Inventory", it would be interesting to check if the port is in the correct VSAN.

Thanks!

Display active supervisor uptime

active supervisor uptime helps in detecting sup switchover in modular platforms directors

MDS9718-A# sh system uptime
System start time:          Sat Jan  9 16:28:56 2021
System uptime:              83 days, 16 hours, 26 minutes, 28 seconds
Kernel uptime:              83 days, 16 hours, 30 minutes, 3 seconds
Active supervisor uptime:   83 days, 16 hours, 26 minutes, 28 seconds