Airavat is a metric interceptor and a job watchdog for Spark Applications. It also features an interactive UI which shows all Spark Applications running, jobs and SQL Queries along with their metrics.
Is your feature request related to a problem? Please describe.
It's hard to track current number of active jobs across different applications, determine load etc.
Describe the solution you'd like
Prometheus is one of the industry standards for this, and can be easily integrated with Airavat
Describe alternatives you've considered
None
Additional context
Some metrics like current number of jobs can be Gauges (go up or down) while others can be summaries or histograms
Is your feature request related to a problem? Please describe.
A lot of code in the UI has come as a part of the CoreUI React Template, and is not being used in Airavat at all.
Describe the solution you'd like
This code needs to be removed