purestorage-openconnect / pure-fb-openmetrics-exporter Goto Github PK

I am trying to set this up on RHEL 7 using docker. I was able to get the docker images deployed but I have multiple endpoints to monitor with different API tokens. I want to set up a files_sd config to be able to add and remove endpoints. The top portion is with the previous pure_exporter, the bottom is the new way.

---
#- targets:
# - INT_IP_ADDR
# labels:
# env: prod
# authorization:
# credentials: SOME_API_KEY
# - INT_IP_ADDR
# labels:
# env: prod
# authorization:
# credentials: SOME_API_KEY

params:
endpoint: INT_IP_ADDR
authorization:
credentials: SOME_API_KEY
endpoint: INT_IP_ADDR
authorization:
credentials: SOME_API_KEY

I have set up the new exporter for one of the pure arrays. I noticed that all these client and filesystem level metrics are not available in the new exporter. Not available as part of any other new metrics either.

These are important filesystem and client level metrics. Can you please provide more information on why these are deprecated from the new exporter and if there is any plan to add these back?

purefb_client_performance_iops
purefb_client_performance_latency_usec
purefb_client_performance_opns_bytes
purefb_client_performance_throughput_bytes
purefb_filesystem_user_usage_bytes
purefb_filesystem_group_usage_bytes
purefb_filesystem_performance_iops
purefb_filesystem_performance_latency_usec
purefb_filesystem_performance_opns_bytes
purefb_filesystem_performance_throughput_bytes

i have kept the token in yaml filen and getting Target authorization token is missing issue.

address: XXX.XXX.XXX.XXX
api_token: XXXXXXXXXXXXXXXXXXXXXXXXXX

We need to be able to capture the array name and unique ID to correlate and tag all metrics in monitoring platforms. We implemented this for purefa_info which we used in the Dynatrace extension and require the same functionality for purefb_info.

When using the base Grafana dashboard - when comparing to the Purity GUI, it seems that the values are doubled. Please see screenshot.

Hi,

I am trying to run the exporter on a k8s cluster behind a load balancer. My prom is set up in another vm and now when i try to get the metrics from flashblade, it is always returning 400 bad request. When i run the same exporter on prom vm and let the prometheus vm communicate via internal ip, able to scrape the metrics.
not sure what is going wrong when scraping on a different kubernetes cluster. my prom config is

• job_name: 'purestorage-fb'
  scheme: https
  metrics_path: /metrics/array
  bearer_token:
  params:
  endpoint: ['pt-flashblade-001.com]
  static_configs:
  - targets: ["pure-exporter.com"]
  labels:
  location: germany
  site: frankfurt
  instance: pt-flashblade-001

whereas just changing the endpoint to the local container is working

• job_name: 'purestorage-fb'
  scheme: https
  metrics_path: /metrics/array
  bearer_token:
  params:
  endpoint: ['pt-flashblade-001.com]
  static_configs:
  - targets: ["pure-fb-exporter:9491"]
  labels:
  location: germany
  site: frankfurt
  instance: pt-flashblade-001

Thanks for any help

Bucket object count is available in /buckets and is useful for tracking for growth of an S3 store.

purefb_buckets_object_count

TODO:
metric name and symantec convention submission for review.

In this example the fullness of the utilisation gauge does not reflect 67.7%.

We need to name a user agent for this exporter similar to FlashArray PureStorage-OpenConnect/pure-fa-openmetrics-exporter#73

Hi,

These two metrics do not return the data within the 2 minutes scrape interval. What is the expected scrape time for these metrics?

purefb_file_system_usage_groups_bytes
purefb_file_system_usage_users_bytes

Other metrics are scrapped fine on the same instances.

Enhancement Request: Feature Request for Volume Exports Metrics

Regarding Volume Level Policy Metric from the Requested Enhancement.

Original Enhancement Request: [(https://github.com//issues/19)]

@genegr - I see purefb_nfs_export_rule policy metric but I do not see that policy name added to any of the filesystem/volume metric.

So the idea is to have one metric with all the policy rules and one of the filesystem metric should have one more label with policy name. That way I can join both the metrics to get all the exports rules for a given volume/filesystem.

Let me know if I missed looking at any filesystem metric. I looked for policy label and I could not find it on any filesystem metric.

Adding a screenshot from previous discussion on this thread.

mkdir -p out/bin
GO111MODULE=on go build -o out/bin/pure-fb-om-exporter cmd/fb-om-exporter/main.go
go: github.com/go-resty/resty/[email protected]: Get https://proxy.golang.org/github.com/go-resty/resty/v2/@v/v2.7.0.mod: Forbidden
make: *** [Makefile:23: build] Error 1

purefb_alerts_open metric has below labels:

• component_name
• component_type
• severity

1. Is there a way to capture more detailed Alert information - probably short summary and description as part of the labels?
2. If I want to collect more information about the metrics which are currently not part of the exporter, is there a way for me to build some object template for the data collection that exporter will export?

Similar to

PureStorage-OpenConnect/pure-fa-openmetrics-exporter#120
PureStorage-OpenConnect/pure-fa-openmetrics-exporter#121

Some of our customers who are able to access this repo, may not have access to the repos that we list in our go.mod as dependancies. This will more or less enable 'offline' builds of our binary.

Currently resty v2.10.0 has a vulnerability GHSA-xwh9-gc39-5298 that is showing in the quay scanner.

Resty now has a v2.11.0 that resolves this vulnerability. https://github.com/go-resty/resty/releases/tag/v2.11.0

Looking for information exposed via /file-system-replica-links to be included so we may track record of file systems replicated lag time and whether they are "protected" based on SLA.

Hi,

On 0.9.0 it seems the exporter does not export the metric purefb.array.space_bytes in the dimension type:array,space:capacity . This is required to report total space available on the flashblade, and needs to be monitored in case of blade or drive failure.

Here is the output I am getting from this metric:

# HELP purefb_array_space_bytes FlashBlade space in bytes
# TYPE purefb_array_space_bytes gauge
purefb_array_space_bytes{space="snapshots",type="array"} 0
purefb_array_space_bytes{space="snapshots",type="file-system"} 0
purefb_array_space_bytes{space="snapshots",type="object-store"} 0
purefb_array_space_bytes{space="total_physical",type="array"} 4.1013406407976e+13
purefb_array_space_bytes{space="total_physical",type="file-system"} 3.802964023162e+13
purefb_array_space_bytes{space="total_physical",type="object-store"} 2.983766176356e+12
purefb_array_space_bytes{space="unique",type="array"} 4.1013406407976e+13
purefb_array_space_bytes{space="unique",type="file-system"} 3.802964023162e+13
purefb_array_space_bytes{space="unique",type="object-store"} 2.983766176356e+12
purefb_array_space_bytes{space="virtual",type="array"} 5.4392994087991e+13
purefb_array_space_bytes{space="virtual",type="file-system"} 5.1398708366336e+13
purefb_array_space_bytes{space="virtual",type="object-store"} 2.994285721655e+12

Can we implement the same changes to purefb OME as purefa OME with regard to open alerts detail.
PureStorage-OpenConnect/pure-fa-openmetrics-exporter#97

Current cardinality
purefb_alerts_open{component_name="filesystem01",component_type="file-systems",severity="info"} 1

Example open alert from REST API

    {
      "name": "234",
      "index": 234,
      "flagged": true,
      "code": 1101,
      "severity": "info",
      "component_name": "filesystem01",
      "component_type": "file-systems",
      "state": "open",
      "created": 1707324042933,
      "updated": 1709728947957,
      "notified": 1707324163175,
      "summary": "File system 'filesystem01' approaching space quota",
      "description": "The 'alexusmb' file system is at 81% of its space quota of 400.00 G. Pure Storage Support is not notified of this alert. If you would like assistance from our support team, you can open a case by emailing [email protected]. As an INFO-level alert, there will be no reminder notifications unless the issue worsens.",
      "knowledge_base_url": "https://support.purestorage.com/?cid=Alert_1101",
      "action": "Remove data from the file system or increase its space quota.",
      "variables": {
        "CurrentUtilization": "0.814668344259262",
        "FileSystemName": "filesystem01"
      },

Suggest we expose

    {
      "action": "string",
      "code": 0,
      "component_name": "string",
      "component_type": "string",
      "created": 0,
      "description": "string",
      "knowledge_base_url": "string",
      "summary": "string",
    }

Can we add this metric to the exporter.
Details of the API endpoint are: https://code.purestorage.com/swagger/redoc/fb2.13-api-reference.html#tag/File-Systems/paths/~1api~12.13~1file-systems~1policies-all/get
This will allow hot users to be identified.

While investigating issue 49 I discovered OME latest and previous versions have never returned space metrics for

purefb_array_space_bytes{space="destroyed",type="file-system"}
purefb_array_space_bytes{space="destroyed_virtual",type="file-system"}
purefb_array_space_bytes{space="destroyed",type="object-store"}
purefb_array_space_bytes{space="destroyed_virtual",type="object-store"}

This would be helpful for more granular capacity reporting to understand how much capacity is waiting to be eradicated.
I'm not sure if this is a bug or an enhancement request.

Exposed metrics from OME v1.0.9 for purefb_array_space_bytes

# HELP purefb_array_space_bytes FlashBlade space in bytes
# TYPE purefb_array_space_bytes gauge
purefb_array_space_bytes{space="capacity",type="array"} 1.78628350604839e+14
purefb_array_space_bytes{space="capacity",type="file-system"} 1.78628350604839e+14
purefb_array_space_bytes{space="capacity",type="object-store"} 1.78628350604839e+14
purefb_array_space_bytes{space="empty",type="array"} 1.36556712710542e+14
purefb_array_space_bytes{space="snapshots",type="array"} 2.484569726e+09
purefb_array_space_bytes{space="snapshots",type="file-system"} 2.484569726e+09
purefb_array_space_bytes{space="snapshots",type="object-store"} 0
purefb_array_space_bytes{space="total_physical",type="array"} 4.2071637894297e+13
purefb_array_space_bytes{space="total_physical",type="file-system"} 3.8869690324535e+13
purefb_array_space_bytes{space="total_physical",type="object-store"} 3.201947569762e+12
purefb_array_space_bytes{space="unique",type="array"} 4.2067604064078e+13
purefb_array_space_bytes{space="unique",type="file-system"} 3.8867137439755e+13
purefb_array_space_bytes{space="unique",type="object-store"} 3.200466624323e+12
purefb_array_space_bytes{space="virtual",type="array"} 5.1417848040718e+13
purefb_array_space_bytes{space="virtual",type="file-system"} 4.8126804911104e+13
purefb_array_space_bytes{space="virtual",type="object-store"} 3.291043129614e+12

Output from debug shows metrics are being collected by REST API query.
GET /api/2.12/arrays/space?type=file-system

==============================================================================
2024/02/27 11:17:38.413972 DEBUG RESTY
==============================================================================
~~~ REQUEST ~~~
GET  /api/2.12/arrays/space?type=file-system  HTTP/1.1
HOST   : fb03
HEADERS:
        Accept: application/json
        Content-Type: application/json
        User-Agent: Pure_FB_OpenMetrics_exporter/1.0
        X-Auth-Token: d0c4a9bb-fae9-4e44-9706-c4d3c092d690
BODY   :
***** NO CONTENT *****
------------------------------------------------------------------------------
~~~ RESPONSE ~~~
STATUS       : 200
PROTO        : HTTP/1.1
RECEIVED AT  : 2024-02-27T11:17:38.41389486Z
TIME DURATION: 4.275515ms
HEADERS      :
        Cache-Control: no-cache, no-store, max-age=0, must-revalidate
        Connection: keep-alive
        Content-Security-Policy: frame-ancestors 'none'
        Content-Type: application/json;charset=UTF-8
        Date: Tue, 27 Feb 2024 11:17:38 GMT
        Expires: 0
        Pragma: no-cache
        Request-Id: 42886705
        Server: nginx
        Strict-Transport-Security: max-age=31536000; includeSubDomains;
        Vary: Origin, Access-Control-Request-Method, Access-Control-Request-Headers
        X-Content-Type-Options: nosniff
        X-Frame-Options: DENY
        X-Xss-Protection: 1; mode=block
BODY         :
{
   "continuation_token": null,
   "total_item_count": 1,
   "items": [
      {
         "name": "FB03",
         "id": "25cb9381-4be2-4ff8-b5c8-ada1bcd4a18c",
         "time": 1709032650000,
         "capacity": 178628350604839,
         "parity": 1.0,
         "space": {
            "virtual": 48126976205824,
            "unique": 38867280110911,
            "snapshots": 2484569726,
            "data_reduction": 1.2382388,
            "total_physical": 38869832995691,
            "total_provisioned": null,
            "destroyed_virtual": 68157440,
            "destroyed": 68315054,
            "available_provisioned": null,
            "available_ratio": null
         }
      }
   ]
}

/api/2.12/arrays/space?type=object-store

==============================================================================
2024/02/27 11:17:38.421215 DEBUG RESTY
==============================================================================
~~~ REQUEST ~~~
GET  /api/2.12/arrays/space?type=object-store  HTTP/1.1
HOST   : fb03
HEADERS:
        Accept: application/json
        Content-Type: application/json
        User-Agent: Pure_FB_OpenMetrics_exporter/1.0
        X-Auth-Token: d0c4a9bb-fae9-4e44-9706-c4d3c092d690
BODY   :
***** NO CONTENT *****
------------------------------------------------------------------------------
~~~ RESPONSE ~~~
STATUS       : 200
PROTO        : HTTP/1.1
RECEIVED AT  : 2024-02-27T11:17:38.421085927Z
TIME DURATION: 4.103292ms
HEADERS      :
        Cache-Control: no-cache, no-store, max-age=0, must-revalidate
        Connection: keep-alive
        Content-Security-Policy: frame-ancestors 'none'
        Content-Type: application/json;charset=UTF-8
        Date: Tue, 27 Feb 2024 11:17:38 GMT
        Expires: 0
        Pragma: no-cache
        Request-Id: 42886707
        Server: nginx
        Strict-Transport-Security: max-age=31536000; includeSubDomains;
        Vary: Origin, Access-Control-Request-Method, Access-Control-Request-Headers
        X-Content-Type-Options: nosniff
        X-Frame-Options: DENY
        X-Xss-Protection: 1; mode=block
BODY         :
{
   "continuation_token": null,
   "total_item_count": 1,
   "items": [
      {
         "name": "FB03",
         "id": "25cb9381-4be2-4ff8-b5c8-ada1bcd4a18c",
         "time": 1709032650000,
         "capacity": 178628350604839,
         "parity": 1.0,
         "space": {
            "virtual": 3291043129614,
            "unique": 3200466624323,
            "snapshots": 0,
            "data_reduction": 1.028301,
            "total_physical": 3201947569762,
            "total_provisioned": null,
            "destroyed_virtual": 1480945439,
            "destroyed": 1480945439,
            "available_provisioned": null,
            "available_ratio": null
         }
      }
   ]
}

Expecting metric to show the data which will be shown in this UI page - https://fb.ip/analysis/performance/users - Performance by Users (NFS) and Performance by Groups (NFS). As far as I have checked we are collecting file system performance data but not to granular level of users and group.

Please plan to add the same.

Currently only active alerts have values in purefb_alerts_open.

It would be useful to have a 0 value for the known levels of the severity dimension instead of null, if there are no other alerts for that severity.

Example,

Before:

purefb_alerts_open{component_name="example_file_system",component_type="file-systems",severity="warning"} 1.0
purefb_alerts_open{component_name="CH0.FB0",component_type="blades",severity="warning"} 1.0

After:

purefb_alerts_open{component_name="example_file",component_type="file-systems",severity="warning"} 1.0
purefb_alerts_open{component_name="CH0.FB0",component_type="blades",severity="warning"} 1.0
purefb_alerts_open{severity="critical"} 0.0
purefb_alerts_open{severity="info"} 0.0

Hello im trying to put run the container as podman
podman run -p 9491:9491 --name pure-fb-om-exporter quay.io/purestorage/pure-fb-om-exporter
2023/05/10 16:50:32 Error in token file: unknown arguments --host 0.0.0.0

can please have a look

Target authorization token is missing !!

i ran http://host:9491/metrics?apitoken=xxxxxxxxxxxxxxxxxxxxxxxxx&endpoint=xxxxxxxxxx

can you give me the solution .

In the previous pure-exporter, there is relabel configuration to support multiple targets.

See below:
relabel_configs:

- source_labels: [__address__]
  target_label: __param_endpoint

- source_labels: [__pure_apitoken]
  target_label: __param_apitoken

- source_labels: [__address__]
  target_label: instance

- target_label: __address__
  replacement: [xxxxx] # CHANGE THIS (pure-exporter address)

In this new openmetrics exporter, what will the configuration looks like? looks like __param_apitoken is not supported anymore.

In 2021, I had initiated discussion with Pure team to get some answers on how to get the Array's total capacity from Purefb Metrics.

The ask was to get this metric purefb_array_capacity_bytes to show the capacity of the array. In the meantime, I had created the static capacity timeseries metric for dashboards to fill the gap.

I want to re-initiate the follow up on this so that we can get the Array capacity dynamically instead of statically.

def __init__(self, fb):
    self.fb = fb
    self.capacity = GaugeMetricFamily(**'purefb_array_capacity_bytes',**
                                   **'FlashBlade total capacity in bytes',**
                                   labels=[])

Can you please provide some information on this?

Does this metric exist today? if not, is there a way to calculate this based off of current available metrics? If not, is there an API I can use to build this metric?

--host is now --address the example needs to be updated. https://github.com/PureStorage-OpenConnect/pure-fb-openmetrics-exporter/blob/main/examples/config/k8s/pure-fb-exporter-deployment.yaml

I pulled the /metrics and I don't see a more granular look at the file system. Below is what I would like to see
purefb-01> purefs list
Name Size Virtual Hard Limit Source Created Protocols Writable Promotion Status
EA1__dss135 10T 3.60T True - 2024-01-11 14:55:53 CST nfsv3 True promoted
nfsv4.1
EA1__dss136 10T 0.00 True - 2024-02-07 11:36:21 CST nfsv3 True promoted
nfsv4.1
EA1__dss137 10T 0.00 True - 2024-02-07 11:37:10 CST nfsv3 True promoted
nfsv4.1
EA1__dss138 10T 0.00 True - 2024-02-07 11:37:55 CST nfsv3 True promoted
nfsv4.1
EA1__dss139 10T 0.00 True - 2024-02-07 11:40:39 CST nfsv3 True promoted
nfsv4.1
EA1__dss140 10T 0.00 True - 2024-02-07 11:41:20 CST nfsv3 True promoted
nfsv4.1
EA1__dss141 10T 0.00 True - 2024-02-07 11:42:04 CST nfsv3 True promoted
nfsv4.1
EA1__dss142 10T 0.00 True - 2024-02-07 11:42:44 CST nfsv3 True promoted
nfsv4.1
EA1__dss143 10T 0.00 True - 2024-02-07 11:43:24 CST nfsv3 True promoted
nfsv4.1
EA1__dss144 10T 0.00 True - 2024-02-07 11:44:10 CST nfsv3 True promoted

I have the attached, but I would like to have a more granular look especially Name,size virtual and maybe protocal

Hi,
you write in the guide that it is possible to create a new readonly user. I am sure this is not possible. On Flashblades there is only the local pureuser.
You can only add a LDAP-user for monitoring, add it to a ad-group and map this group to readonly. Or is there any method?

Regards