emc-openstack / storops Goto Github PK

Add following metrics for Unity Virtual Volume.
Note that virtual volume has two implementations: block & file.
The path and calculation is different for different kind of virtual volume.

• read_iops
• write_iops
• read_mbps
• write_mbps
• total_latency_ms

Related paths:

• block based vvol:
  • sp.*.storage.vvol.block.*.reads
  • sp.*.storage.vvol.block.*.writes
  • sp.*.storage.vvol.block.*.totalIoTime
  • sp.*.storage.vvol.block.*.reads
  • sp.*.storage.vvol.block.*.readBlocks
  • sp.*.storage.vvol.block.*.writes
  • sp.*.storage.vvol.block.*.writeBlocks
• file based vvol:
  • sp.*.storage.vvol.file.*.readIoTime
  • sp.*.storage.vvol.file.*.reads
  • sp.*.storage.vvol.file.*.writeIoTime
  • sp.*.storage.vvol.file.*.writes
  • sp.*.storage.vvol.file.*.reads
  • sp.*.storage.vvol.file.*.readBytes
  • sp.*.storage.vvol.file.*.writes
  • sp.*.storage.vvol.file.*.writeBytes

Formula:

• Total calls for each SP:

delta(reads) + delta(writes)

• totalIoTime for each SP

delta(readIoTime) + delta(writeIoTime)

• Total Latency:

((spa totalIoTime) + (spb totalTime)) / ((spa total calls) + (spb total calls))/1000

With the support of this commit, user can:

• add storops operations to a queue, and they can be executed periodically till operation executes correctly.
• unfinished items in the queue can be restored automatically after restart.

Hi Team,
Can we read the alert messages in VNX via Storops ? Any way by which we can display the warning messages of VNX ?

Thanks
Sunny

Reproduce steps:

luns = vnx.get_lun()
lun1 = luns[1]
In [10]: lun1 = luns[1]

In [11]: lun1.name
Out[11]: u'volume-a4c17859-ba14-4a8b-b5d8-e842cf9b513d'

In [12]: lun1.update()
Out[12]:
{
    "VNXLun": {
        "blocks_read": 0,
        "blocks_read_sp_a": 0,
        "blocks_read_sp_b": 0,
        "blocks_written": 0,
        "blocks_written_sp_a": 0,
        "blocks_written_sp_b": 0,
        "busy_ticks": 0,
        "busy_ticks_sp_a": 0,
        "busy_ticks_sp_b": 0,
        "consumed_capacity_gbs": 1.753,
        "current_owner": "VNXSPEnum.SP_B",
        "deduplication_state": "Off",
        "deduplication_status": "OK(0x0)",
        "default_owner": "VNXSPEnum.SP_B",
        "existed": true,
        "explicit_trespasses": 0,
        "explicit_trespasses_sp_a": 0,
        "explicit_trespasses_sp_b": 0,
        "hash": 8789294098397,
        "idle_ticks": 0,
        "idle_ticks_sp_a": 0,
        "idle_ticks_sp_b": 0,
        "implicit_trespasses": 0,
        "implicit_trespasses_sp_a": 0,
        "implicit_trespasses_sp_b": 0,
        "initial_tier": "Lowest Available",
        "is_compressed": true,
        "is_private": false,
        "is_thin_lun": true,
        "lun_id": 78,
        "name": "volume-9f0870dc-1ac0-4afb-8339-919816791119",
        "non_zero_request_count_arrivals": 0,
        "non_zero_request_count_arrivals_sp_a": 0,
        "non_zero_request_count_arrivals_sp_b": 0,
        "operation": "None",
        "performance": 100.0,
        "pool_name": "DAILY_2",
        "raid_type": "VNXPoolRaidType.RAID5",
        "read_requests": 0,
        "read_requests_sp_a": 0,
        "read_requests_sp_b": 0,
        "snapshot_mount_points": [],
        "state": "Ready",
        "status": "OK(0x0)",
        "sum_of_outstanding_requests": 0,
        "sum_of_outstanding_requests_sp_a": 0,
        "sum_of_outstanding_requests_sp_b": 0,
        "tiering_policy": "Lowest Available",
        "total_capacity_gb": 2.0,
        "user_capacity_gbs": 2.0,
        "write_requests": 0,
        "write_requests_sp_a": 0,
        "write_requests_sp_b": 0,
        "wwn": "60:06:01:60:12:90:38:00:84:58:3E:1B:02:DE:E6:11"
    }
}

In [13]: lun1.name
Out[13]: u'volume-9f0870dc-1ac0-4afb-8339-919816791119'

You can see, the name changed

Link aggregation support includes:
Create link aggregation
Modify link aggregation
Delete link aggregation

Add a new API get_fc_port in UnitySystem to get all the fc ports on the system.

Add following operations for unity consistency groups.

• list cg.
• create cg with initial member.
• delete cg.
• add/remove cg member.
• take cg snapshot.
• remove cg snapshot.
• restore cg snapshot.

Add API to allow user to retrieve/modify/set the NTP and DNS server for Unity.

Change the default value back to False.

Add keep_for parameter in lun.create_snap and snap.create interfaces

Hi, would be real useful to be able to set the HLU when using attach_alu against a storage group.

I use boot LUNs, and host-based storage groups, where the HLUs all need to match.

Link aggregation is not supported in pre-Falcon release. We should handle the link aggregation for different releases.

Add license info in the requirement document.

This is a planned content:

1. Update README to include the capacity report feature
2. Refine the round_it utilities.

Add the extend method for UnityLun to allow expansion.

Attaching snap to host only supports from Unity OE version 4.1.

Although before the version 4.1, attaching snap to host would succeed, the IO to the snap will fail.
We need to add version check to only allow attaching snap to host in version 4.1 or later.

Hi Team,
I have sucessfully installed storops and also haveing naviseccli on the RHEL6.7.
Every time I am getting message "No handlers could be found for logger "storops.vnx.navi_command"

And it get stucks, sometimes i get the output but sometimes not.

[root@Test ~]# python2.7
Python 2.7.10 (default, May 27 2016, 05:55:08)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-17)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

>>> from storops import VNXSystem
>>> vnx = VNXSystem('10.xx.xx.xx', 'test_user', 'password')
>>> vnx
No handlers could be found for logger "storops.vnx.navi_command"

Thanks
Sunny

Hi Team,
I am looking for getting information on all types storage capacity of VNX. Attached image shows "Storage capacity summary" on Unisphere Dashboard of VNX. It shows 3 types of capacity in our environment.
"Free Storage pool capacity": 274812.95 GB( I am able to get/calculate this correctly Via Storops)
"Free raw disk capacity": 14552.92 GB
"Used Capacity": 78257.92 GB

My Query is : How to get info on "Used" & "Free raw disk capacity" using storops.
I did some analysis as shown below.

>>>pools = vnx.get_pool()
>>> pools.available_capacity_gbs
[260031.885, 14781.063]

i.e Addition of these Two pools is 274812.948 , This is correctly shown in the Dashboard of storage for "Free Storage pool capacity".
But whenever i try to get consumed capacity, it shows the capacity of actual luns which are allocated from the two pools.

>>> pools.consumed_capacity_gbs
[842.142, 243.976]

Below are some links which can be checked/referred on what exactly i am looking for.

How does Unisphere calculates Capacity on Dashboard
.
OR Is there any way we can get logical_capacity, raw_capacity, used_capacity, free_capacity as shown here How to compile capacity and used capacity of an EMC VNX via CLI?

You can also check Available space and Used space

Please let me know if you have any questions.

Thanks
Sunny

Add entry point/method for UnityIoLimitPolicy on UnitySystem.

The RestAPI of Unity supports nested query which can be used to fetch the properties of the associated object in one request. For example.

In order to get the iqn of an iscsi portal we need to send two requests:

"/api/instances/iscsiPortal/if_4?compact=True&fields=id,ipAddress,iscsiNode"
{
"content": {
"instanceId": "root/emc:EMC_UEM_IPInterfaceLeaf%InstanceID=if_4",
"iscsiNode": {
"id": "iscsinode_spa_eth2"
},
"ipAddress": "10.244.213.177",
"id": "if_4"
}
}

"/api/instances/iscsiPortal/iscsinode_spa_eth2?compact=True&fields=id,name"

With the nested query, only one single request is needed :

"/api/instances/iscsiPortal/if_4?compact=True&fields=ipAddress,iscsiNode,iscsiNode.name"
{
"content": {
"instanceId": "root/emc:EMC_UEM_IPInterfaceLeaf%InstanceID=if_4",
"iscsiNode": {
"id": "iscsinode_spa_eth2",
"name": "iqn.1992-04.com.emc:cx.fnm00150600267.a0"
},
"ipAddress": "10.244.213.177",
"id": "if_4"
}
}

The unity will return the name property in the iscsiNode along with id when it see the extra field "iscsiNode.name" in the query fields.

Hi Team,
Is there any direct method by which i can get all the Luns which belongs to particular pool.
I am using below method for now, but this gives me detailed ouput of luns.

>>>vnx = VNXSystem('10.xx.xx,xx', 'user', 'password')
>>>luns = vnx.get_lun()
>>>luns2 =  list(luns)
>>>type(luns2)
<type 'list'>
>>>len(luns2)   <- This gives me number 200+ luns which belongs to all the pools.
>>>luns3=filter(lambda x: x.pool_name=="Pool_1", luns2)

Variable luns3 gives me a big output with all details of around 12 luns which belongs to this pool "Pool_1". But i only need lun names.
Is there any direct way to achieve for getting list of luns which belongs to the particular pool ?

Thanks
Sunny

Add metric total_latency_ms and queue_length for unity disk.
Required metrics:

• sp.*.physical.disk.*.busyTicks
• sp.*.physical.disk.*.sumArrivalQueueLength
• sp.*.physical.disk.*.reads
• sp.*.physical.disk.*.writes
• sp.*.physical.coreCount

Formula:

• Total calls for each SP:

  delta(reads) + delta(writes)

• Total Latency for each SP:

  ((delta(busyTicks) * (delta(sumArrivalQueueLength)/(total calls))) / (total calls)) / coreCount

• Total Latency (ms):

  ((spa total latency) * (spa total calls) + (spb total latency) * (spb total calls)) / ((spa total calls) + (spb total calls)) / 1000

• Queue Length for each SP:

  (delta(sumArrivalQueueLength)/(total calls)) / coreCount

• Average Queue Length (count):

  ((spa Queue Length) * (spa total calls) + (spb Queue Length) * (spb total calls)) / ((spa total calls) + (spb total calls))

Porting the Unity metrics collecting framework to VNX and allow basic collection of metrics.

Storops doesn't configure the logging by default. The user need to configure the logging by themselves in order to see the log messages. It is not convenient when we try storops in console such as python or ipython. A simple API to enable the console logging will be very useful when we try the storops in console for testing or POC purpose.

Hi Team,
Is there any way by which i can get Host lists and allocated luns of VNX through Storops?

Thanks
Regards

Sunny

Storops doesn't support to verify the Unity array's certificate.

When cancelling migration, a error message may come out:
Migration cannot be cancelled because data sychronization is completed (0x714a8021)

This is to add a VNXLunSyncCompletedError to differentiate it from general error.

Python 3.6 is released. Add new CI configuration for python 3.6.

The performance data for Unity arrays are retrieved by register/subscribe realtime metrics data.
Add support for realtime metrics data subscribe and retrieval.
Also need to support the list of available metrics on the array.

Add following metrics for Unity Storage Processor.

• reads/writes
  • nfsv2_read_iops
  • nfsv2_write_iops
  • nfsv3_read_iops
  • nfsv3_write_iops
  • nfsv4_read_iops
  • nfsv4_write_iops
  • smb1_read_iops
  • smb1_write_iops
  • smb2_read_iops
  • smb2_write_iops
• response time
  • nfsv2_read_response_time_ms
  • nfsv2_write_response_time_ms
  • nfsv3_read_response_time_ms
  • nfsv3_write_response_time_ms
  • nfsv4_read_response_time_ms
  • nfsv4_write_response_time_ms
  • smb1_read_response_time_ms
  • smb1_write_response_time_ms
  • smb2_read_response_time_ms
  • smb2_write_response_time_ms

Related Paths:

• sp.*.nfs.v2.op.*.totalTime
• sp.*.nfs.v2.io.*.reads
• sp.*.nfs.v2.io.*.writes
• sp.*.nfs.v3.op.*.totalTime
• sp.*.nfs.v3.io.*.reads
• sp.*.nfs.v3.io.*.writes
• sp.*.nfs.v4.op.*.totalTime
• sp.*.nfs.v4.io.*.reads
• sp.*.nfs.v4.io.*.writes
• sp.*.cifs.smb1.op.*.totalTime
• sp.*.cifs.smb1.basic.reads
• sp.*.cifs.smb1.basic.writes
• sp.*.cifs.smb2.op.*.totalTime
• sp.*.cifs.smb2.basic.reads
• sp.*.cifs.smb2.basic.writes

Formula:

• response time

delta(totalTime)/delta(calls)/1000

• reads/writes per second

delta(reads or writes) / delta time

We are attempting to package storops to support EMC Cinder drivers in RDO and Red Hat OpenStack. It is important for us to run unit tests to ensure the packaged library works, since we can't test it against hardware in packaging build environments.

storops-0.4.6.zip on pypi does not contain data to run unit tests. A number of unit tests will fail with:

OSError: file: /home/eharney/src/storops-pypi/storops-0.4.6/test/vnx/testdata/block_output/security_-certificate_-getLevel.txt not found.

This is available in the storops git tree but is not packaged in the source zip.

Other tests also fail with:

FileNotFoundError: [Errno 2] No such file or directory: 'naviseccli'

which seems to indicate that the unit tests are attempting to execute "naviseccli". This program is not available.

Please include the required test data in the storops source zip, and mock out calls to external CLI utilities.

The dependency of lxml is causing various installation problems for storops on both Windows and Linux platform. While in fact we only utilize it for xml parsing for VNX file which is almost not used.
Remove this dependency and replace it with native python xml library to remove this installation trouble.

Add filters to UnityEthernetPortList, UnityFcPortList, UnityIscsiPortalList, and UnityInitiatorPathList.

VNX use naviseccli nqm to manage VNX QoS,

We need to add related functionality in storops as well.

Below files should be ignored by git.

test/connection/.requests_mock.py.swp
test/connection/.test_client.py.swp
test/connection/.test_connector.py.swp

The persisted metrics csv file now saves to the same file for each resource kind. This enhancement has two requirement:

• Open a new persisted csv file when the file size reaches limit.
• Limit the maximum number of csv files per each resource kind. Remove the oldest one if we reach the limit.

Both limits should be configurable from UnitySystem.

Storops needs to handle following output for storagegroup -list:

note: the issue is that: no space between SP A and 1

2017-01-13 01:30:45.569 429086 DEBUG storops.vnx.navi_command [-] command complete: /opt/Navisphere/bin/naviseccli -h 192.168.46.205 -user sysadmin -password *** -scope global storagegroup -messner -list -host -iscsiAttributes -gname wcmsc2-l-rh-cmp-2.hqplan.lab, time consumed (s): 1.71068286896, output:·
Storage Group Name:    wcmsc2-l-rh-cmp-2.hqplan.lab                                    
Storage Group UID:     BB:CD:9D:89:12:D9:E6:11:8D:24:82:E6:BF:48:3A:C7                 
HBA/SP Pairs:                                                                          
                                                                                       
  HBA UID                                          SP Name     SPPort                  
  -------                                          -------     ------·                 
  iqn.1994-05.com.redhat:wcmsc2-l-rh-cmp-2.hqplan.lab:c3bf2f4ccb SP A1                 
Host name:             wcmsc2-l-rh-cmp-2.hqplan.lab                                    
SPPort:                A-1v0                                                           
Initiator IP:          N/A                                                             
TPGT:                  6                                                               
ISID:                  N/A                                                             
                                                                                       
  iqn.1994-05.com.redhat:wcmsc2-l-rh-cmp-2.hqplan.lab:c3bf2f4ccb SP B1                 
Host name:             wcmsc2-l-rh-cmp-2.hqplan.lab                                    
SPPort:                B-1v0                                                           
Initiator IP:          N/A                                                             
TPGT:                  10                                                                                                                                                        
ISID:                  N/A                                                             
                                                                                       
  iqn.1994-05.com.redhat:wcmsc2-l-rh-cmp-2.hqplan.lab:c3bf2f4ccb SP A3                 
Host name:             wcmsc2-l-rh-cmp-2.hqplan.lab                              
SPPort:                A-3v0                                                     
Initiator IP:          N/A                                                       
TPGT:                  8                                                         
ISID:                  N/A                                                       
                                                                                 
  iqn.1994-05.com.redhat:wcmsc2-l-rh-cmp-2.hqplan.lab:c3bf2f4ccb SP B3           
Host name:             wcmsc2-l-rh-cmp-2.hqplan.lab                              
SPPort:                B-3v0                                                     
Initiator IP:          N/A                                                       
TPGT:                  12                                                        
ISID:                  N/A                                                       
                                                                                 
                                                                                 
HLU/ALU Pairs:                                                                   
                                                                                 
  HLU Number     ALU Number                                                      
  ----------     ----------                                                      
    196             24                                                           
Shareable:             YES _

Add absolute limit of throughput and bandwidth for Unity LUN.
Allow user to update and view these thresholds from LUN properties.

Support modifying the Speed and MTU setting of the Ethernet port

An error JobFailed was reported without any extra information when creating NFS share get failed.
JobStateError: Job failed in FAILED.

The error message could be enhanced to include the message that why the job failed which is reported by the RESTAPI as below:

   "messages": [
                    {
                        "errorCode": 108008449, 
                        "messages": [
                            {
                                "locale": "en_US", 
                                "message": "The specified file system size is too small. (Error Code:0x6701401)"
                            }
                        ]
                    }
                ],

The current implementation is that: if SP is not passed in (is None) to create NAS server, storops will use the first item of SP list.

This feature is to auto balance the SP of creating NAS server when SP is not passed in.
SPB will be chosen only when the existing NAS servers on SPA are more than the ones on SPB. Otherwise, SPA will be chosen.

When I get a VNXPort instance, there is no way to display this instance in a friendly name.

Hi Team,
From the VNX Dashboard, we can view Name and other details assigned to the Storage box.

How can we view the same in storops

Thanks
Sunny

Hi Team,
I am facing an weird issue that whenever i run the command 'vnx' for first time OR getting any information by it for example "vnx.model" throws some error messages on console and Next time by running same command gives me proper output till the session is running.
am i missing anything ? Is there any other way to pull basic details like serial no. , model and name of VNX ?

[root@Test ~]# python2.7
Python 2.7.10 (default, May 27 2016, 05:55:08)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-17)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from storops import VNXSystem
>>> import logging
>>> logging.basicConfig(level=logging.WARNING)
>>> vnx = VNXSystem('10.xx.xx.xx', 'test_user', 'password')
>>> vnx
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/storops/lib/common.py", line 50, in __repr__
    return self.json(indent=4)
  File "/usr/local/lib/python2.7/site-packages/storops/lib/common.py", line 47, in json
    return json.dumps(self.get_dict_repr(), indent=indent, sort_keys=True)
  File "/usr/local/lib/python2.7/site-packages/storops/lib/common.py", line 42, in get_dict_repr
    props = self._get_properties(dec)
  File "/usr/local/lib/python2.7/site-packages/storops/lib/resource.py", line 155, in _get_properties
    value = getattr(self, name)
  File "/usr/local/lib/python2.7/site-packages/storops/lib/resource.py", line 184, in __getattr__
    ret = self._get_property_from_raw(item)
  File "/usr/local/lib/python2.7/site-packages/storops/vnx/resource/__init__.py", line 70, in _get_property_from_raw
    value = super(VNXCliResource, self)._get_property_from_raw(item)
  File "/usr/local/lib/python2.7/site-packages/storops/lib/resource.py", line 197, in _get_property_from_raw
    self.update()
  File "/usr/local/lib/python2.7/site-packages/storops/vnx/resource/system.py", line 144, in update
    self.update_nodes_ip()
  File "/usr/local/lib/python2.7/site-packages/storops/vnx/resource/system.py", line 139, in update_nodes_ip
    self._cli.set_ip(self.spa_ip, self.spb_ip, self._get_cs_ip())
  File "/usr/local/lib/python2.7/site-packages/storops/vnx/resource/system.py", line 183, in _get_cs_ip
    return VNXDomainNodeList.get_cs_ip(self.serial, self._cli)
  File "/usr/local/lib/python2.7/site-packages/storops/vnx/resource/vnx_domain.py", line 83, in get_cs_ip
    node = dnl.get_node(serial)
  File "/usr/local/lib/python2.7/site-packages/storops/vnx/resource/vnx_domain.py", line 76, in get_node
    'domain node "{}" not found'.format(node_id))
storops.exception.VNXObjectNotFound: domain node "APM00XXXXXXXX" not found

Next time running the same command gives proper output.

>>> vnx
{
    "VNXSystem": {
        "agent_rev": "7.33.9 (1.5)",
        "existed": true,
        "hash": 8733XXXXX,
        "model": "VNX5600",
        "model_type": "Rackmount",
        "name": "X",
        "revision": "05.33.009.5",
        "serial": "APM00XXXXXXXX"
    }
}

Thanks
Regards

Sunny

A tenant is not able to be deleted when it contains hosts. This enhancement is to add an API on the tenant object to delete the tenant along with its hosts.

Add Unity LUN metrics total_latency_ms and queue_length.

Related paths:

• sp.*.storage.lun.*.totalIoTime
• sp.*.storage.lun.*.currentIOCount
• sp.*.storage.lun.*.busyTime
• sp.*.storage.lun.*.idleTime
• sp.*.storage.lun.*.reads
• sp.*.storage.lun.*.writes

Formula:

• Total latency (ms):

((spa totalIoTime) + (spb totalTime)) / ((spa total calls) + (spb total calls))/1000

• Average Queue Length for each SP (count):

delta(currentIOCount) * (delta(busyTime)/(delta(idleTime) + delta(busyTime)))

• Average Queue Length (count):

((spa Queue Length) * (spa total calls) + (spb Queue Length) * (spb total calls)) / ((spa total calls) + (spb total calls))

For ports on IO modules, there is no property parent_storage_processor.

Set the property manually.

As title. Add these two property to ease the approach of getting relationship between host and LUNs.

• Remove cobertura-clover-transform from test-requirements.
• Remove lxml dependency from VNX file UT.