openmainframeproject / feilong Goto Github PK

@mfriesenegger @jmertic
We have do some investigates, but didn't find a good solution for this issue, need some help and discuss from team and linux foundation.
This block the github action to access the Vicom infinity Z infrastructure.

The doc of CRUD of fcp_template and the attach_volume/detach_volume APIs are missing, need to add them.

Currently, ECKD disk support in Feilong is handled the same as FBA disk, where the image is of the filesystem level (for FBA disks this is the only sensible approach). The Linux DASD driver for ECKD supports a different mode, called raw-track-access mode. In this mode, the raw count-key-data records of the disk can be accessed.

If this support were introduced, the following benefits could be achieved:

• No need to dasdfmt a disk before writing an image to the disk, which would save a lot of time when creating a guest;
• Images of other OSes (z/OS, z/VSE, z/VM) could be supported by Feilong.

Feilong would need to track the type of an ECKD image, so that an image created using raw-track-access mode would be correctly restored. I am not aware that this could be done automatically, so it would be up to the user registering the image with Feilong to correctly "mark" it with the appropriate type.

I'm happy to start looking at how this could be implemented, since I have experience working with raw-track-access mode as part of the z/VM ESI project.

As discussed with @mfriesenegger at VM Workshop 2023.

https://cloudlib4zvm.readthedocs.io/en/latest/architecture.html#internal-architecture shows "IUCV" as the way the Feilong connector and SMAPI communicate.

According to "Update BYOL definition about IUCV" section on https://cloudlib4zvm.readthedocs.io/en/latest/quickstart.html#setup-for-z-vm-sdk-daemon, IUCV is rather used for the communication between Feilong and the deployed guests.

Which one is correct?

We once deployed Feilong z/VM without the IUCV authorizations by accident, and it was still comunicating fine with SMAPI, so I suspect it's the schema that is wrong, while the explanations in step 3 of 3.1.1 are correct.

How does Failong communicate with SMAPI by the way? I suspect with TCP/IP sockets?

Or is this an option that can be set somewhere?

The resizeECKD function fails because parted has change the resize option to resizepart. Here is the proposed fix.

--- /opt/zthin/bin/unpackdiskimage.orig 2020-07-16 23:30:12.000000000 -0400
+++ /opt/zthin/bin/unpackdiskimage      2020-09-03 15:11:59.102983614 -0400
@@ -399,7 +399,7 @@
       printError "When recreating the VTOC, 'fdasd' to  ${devNode} returns rc: $rc, out: $out"
       return 1
     fi
-    out=`parted ${devNode} resize $partnum 100%`
+    out=`parted ${devNode} resizepart $partnum 100%`
     rc=$?
     if (( rc != 0 )); then
       printError "When resizing the dasd, 'parted' to  ${devNode} returns rc: $rc, out: $out" 

I would like to set up root's password at deployment time.

Looking at the code of nova zvm connector, they pass it via admin_pass in the metadata file of the config disk. So I tried to do the same:

 {
   "uuid": "NAME",
+  "admin_pass": "ADMIN_PASS",
   "public_keys": {
     "Public Key": "PUBLIC_KEY"
   },
(...)

I'd like to contribute a fix to this problem, but there's something I do not understand. When I call the "Deploy Guest" API call, it uploads the "transport file" and then sends it to the VM's punch. Then it's used by cloud-init inside the Linux guest. But I could not find the code that manages this option.

PUT /guests/{userid}/disks says

Configure additional disks for a guest

and the request representation makes it look like it is modifying certain properties of one or more, but not necessarily all, of the guests already-existing disks (probably created earlier in time via POST on the same resource, but possibly also via other means like via the guest's creation request). Is this what's going on?

The problem with that behavior is that partial modifications (contrasted with complete replacements) of the request URI conflict with RFC 7231's requirements on HTTP PUT, specifically

The fundamental difference between the POST and PUT methods is highlighted by the different intent for the enclosed representation. The target resource in a POST request is intended to handle the enclosed representation according to the resource's own semantics, whereas the enclosed representation in a PUT request is defined as replacing the state of the target resource. Hence, the intent of PUT is idempotent and visible to intermediaries, even though the exact effect is only known by the origin server.

Reference: https://cloudlib4zvm.readthedocs.io/en/latest/restapi.html#smapi-healthy

1. In the doc, one reads:

7.4.1. Report healthy of SMAPI

but correct English would rather be "Report health of SMAPI".

2. In the doc, one reads:

* Request:

     `X-Admin-Token	header	string	Admin-token, which is stored in /etc/zvmsdk/token.dat. You need
put this into header to request for a auth-token. Then you can send a request with this auth-token.`

I think this is meant for another API call, and it should rather be:

* Request:

None

3. In the doc, one reads:

smapi_healthy	body	dict	Status of SMAPI healthy

but the API returns a dictionary named SMAPI instead of smapi_healthy. Which one is wrong?

4. In the doc, one reads:

continueousFail	body	integer	The count of continueous fail.

and this is consistent with what the API returns, but correct English is "continuous", not "continueous" (no "e").

5. In the doc, one reads:

healthy	body	integer	Whether SMAPI is healthy.

but the API returns a boolean instead of an integer. Which one is wrong?

6. This paragraph 7.4.1 lacks an example. Here is one:

{
  "SMAPI": {
    "totalSuccess": 0,
    "totalFail": 0,
    "lastSuccess": "",
    "lastFail": "",
    "continueousFail": 0,
    "healthy": true
  }
}

7. The return value from the API above lacks the "usual" fields: rs, overallRC, modID, rc, and output.

Is that intentional?

I hope that helps.

Here are the CRUD methods related to minidisks handling:

• Create: POST /guests/{userid}/disks
• Read: (none)
• Update: PUT /guests/{userid}/disks
• Delete: DELETE /guests/{userid}/disks

We are missing a Read function for GET /guests/{userid}/disks.

Here is how it appears in the source code of the python client library:

zvmconnector> grep /disks restclient.py -B1

def req_guest_create_disks(start_index, *args, **kwargs):
    url = '/guests/%s/disks'

def req_guest_delete_disks(start_index, *args, **kwargs):
    url = '/guests/%s/disks'

def req_guest_config_minidisks(start_index, *args, **kwargs):
    url = '/guests/%s/disks'

To fix this, I wrote a PR to add the missing API call.

Hello, all.
In the wrapperutils.c file, I believe that the EBCDIC <-> ASCII conversation tables are not quite right. Here is my "fixed" ones:

static unsigned char a2e[256] = {
0, 1, 2, 3, 55, 45, 46, 47, 22, 5, 37, 11, 12, 13, 14, 15,
16, 17, 18, 19, 60, 61, 50, 38, 24, 25, 63, 39, 28, 29, 30, 31,
64, 90, 127, 123, 91, 108, 80, 125, 77, 93, 92, 78, 107, 96, 75, 97,
// 64, 79, 127, 123, 91, 108, 80, 125, 77, 93, 92, 78, 107, 96, 75, 97,
240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 122, 94, 76, 126, 110, 111,
124, 193, 194, 195, 196, 197, 198, 199, 200, 201, 209, 210, 211, 212, 213, 214,
215, 216, 217, 226, 227, 228, 229, 230, 231, 232, 173, 173, 224, 189, 95, 109,
// 215, 216, 217, 226, 227, 228, 229, 230, 231, 232, 233, 74, 224, 90, 95, 109,
121, 129, 130, 131, 132, 133, 134, 135, 136, 137, 145, 146, 147, 148, 149, 150,
151, 152, 153, 162, 163, 164, 165, 166, 167, 168, 169, 192, 106, 208, 161, 7,
32, 33, 34, 35, 36, 21, 6, 23, 40, 41, 42, 43, 44, 9, 10, 27,
48, 49, 26, 51, 52, 53, 54, 8, 56, 57, 58, 59, 4, 20, 62, 225,
65, 66, 67, 68, 69, 70, 71, 72, 73, 81, 82, 83, 84, 85, 86, 87,
88, 89, 98, 99, 100, 101, 102, 103, 104, 105, 112, 113, 114, 115, 116, 117,
118, 119, 120, 128, 138, 139, 140, 141, 142, 143, 144, 154, 155, 156, 157, 158,
159, 160, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183,
184, 185, 186, 187, 188, 189, 190, 191, 202, 203, 204, 205, 206, 207, 218, 219,
220, 221, 222, 223, 234, 235, 236, 237, 238, 239, 250, 251, 252, 253, 254, 255
};

static unsigned char e2a[256] = {
0, 1, 2, 3, 156, 9, 134, 127, 151, 141, 142, 11, 12, 13, 14, 15,
16, 17, 18, 19, 157, 133, 8, 135, 24, 25, 146, 143, 28, 29, 30, 31,
128, 129, 130, 131, 132, 10, 23, 27, 136, 137, 138, 139, 140, 5, 6, 7,
144, 145, 22, 147, 148, 149, 150, 4, 152, 153, 154, 155, 20, 21, 158, 26,
32, 160, 161, 162, 163, 164, 165, 166, 167, 168, 91, 46, 60, 40, 43, 33,
38, 169, 170, 171, 172, 173, 174, 175, 176, 177, 33, 36, 42, 41, 59, 94,
// 38, 169, 170, 171, 172, 173, 174, 175, 176, 177, 93, 36, 42, 41, 59, 94,
45, 47, 178, 179, 180, 181, 182, 183, 184, 185, 124, 44, 37, 95, 62, 63,
186, 187, 188, 189, 190, 191, 192, 193, 194, 96, 58, 35, 64, 39, 61, 34,
195, 97, 98, 99, 100, 101, 102, 103, 104, 105, 196, 197, 198, 199, 200, 201,
202, 106, 107, 108, 109, 110, 111, 112, 113, 114, 203, 204, 205, 206, 207, 208,
209, 126, 115, 116, 117, 118, 119, 120, 121, 122, 210, 211, 212, 91, 214, 215,
// 209, 126, 115, 116, 117, 118, 119, 120, 121, 122, 210, 211, 212, 213, 214, 215,
216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 93, 230, 231,
// 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231,
123, 65, 66, 67, 68, 69, 70, 71, 72, 73, 232, 233, 234, 235, 236, 237,
125, 74, 75, 76, 77, 78, 79, 80, 81, 82, 238, 239, 240, 241, 242, 243,
92, 159, 83, 84, 85, 86, 87, 88, 89, 90, 244, 245, 246, 247, 248, 249,
48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 250, 251, 252, 253, 254, 255
};

Hope this is of some help here.
DJ

Hi,

I recently updated Feilong to 1.6.3, but it didn't work. I found out that in the database some columns were missing in the table FCP. I looked at the code, there is a check for a non existing table, but there is no check if the table matches.

Is it possible to add a check if the database is consistent for the columns it needs to have.

The columns missing were:
wwpn_npiv varchar(16) COLLATE NOCASE
wwpn_phy varchar(16) COLLATE NOCASE

I added them by hand.

Running Sphinx v1.8.5
loading translations [en]... done

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/cloudlib4zvm/envs/latest/lib/python2.7/site-packages/sphinx/cmd/build.py", line 303, in build_main
    args.tags, args.verbosity, args.jobs, args.keep_going)
  File "/home/docs/checkouts/readthedocs.org/user_builds/cloudlib4zvm/envs/latest/lib/python2.7/site-packages/sphinx/application.py", line 228, in __init__
    self.setup_extension(extension)
  File "/home/docs/checkouts/readthedocs.org/user_builds/cloudlib4zvm/envs/latest/lib/python2.7/site-packages/sphinx/application.py", line 449, in setup_extension
    self.registry.load_extension(self, extname)
  File "/home/docs/checkouts/readthedocs.org/user_builds/cloudlib4zvm/envs/latest/lib/python2.7/site-packages/sphinx/registry.py", line 475, in load_extension
    raise ExtensionError(__('Could not import extension %s') % extname, err)
ExtensionError: Could not import extension ext.restapi_parameters (exception: No module named yaml)

Extension error:
Could not import extension ext.restapi_parameters (exception: No module named yaml)

build doc failed due to yaml not found ,this is caused by commit
612b855

Deprecation warnings are raised due to invalid escape sequences. This can be fixed by using raw strings or escaping the literals. pyupgrade also helps in automatic conversion : https://github.com/asottile/pyupgrade/

find . -iname '*.py' | grep -Ev 'vendor|example|doc|tools|sphinx' | xargs -P4 -I{} python3.8 -Wall -m py_compile {}
./smtLayer/getVM.py:306: DeprecationWarning: invalid escape sequence \*
  results['response'] = re.sub('\*DVHOPT.*', '', results['response'])
./smtLayer/getVM.py:334: DeprecationWarning: invalid escape sequence \*
  results['response'] = re.sub('\*DVHOPT.*', '', results['response'])
./smtLayer/vmUtils.py:152: DeprecationWarning: invalid escape sequence \.
  match = re.search('Reason code (.+?)\.', output)
./smtLayer/vmUtils.py:814: DeprecationWarning: invalid escape sequence \w
  search_pattern = '(^HCP\w\w\w045E|^HCP\w\w\w361E)'.encode()
./smtLayer/vmUtils.py:1123: DeprecationWarning: invalid escape sequence \w
  match = re.search('(^HCP\w\w\w045E|^HCP\w\w\w361E)', out)
./smtLayer/changeVM.py:889: DeprecationWarning: invalid escape sequence \w
  if re.search('(^HCP\w\w\w040E)', results['response']):
./zvmsdk/utils.py:232: DeprecationWarning: invalid escape sequence \s
  if re.search('\s', cidr):
./zvmsdk/utils.py:555: DeprecationWarning: invalid escape sequence \w
  strfail = '(^HCP\w\w\w003E|^HCP\w\w\w040E|' + \
./zvmsdk/utils.py:556: DeprecationWarning: invalid escape sequence \w
  '^HCP\w\w\w026E|^HCP\w\w\w045E)'
./zvmsdk/utils.py:559: DeprecationWarning: invalid escape sequence \w
  strfail = '(^HCP\w\w\w003E|^HCP\w\w\w040E|^HCP\w\w\w026E)'
./zvmsdk/utils.py:560: DeprecationWarning: invalid escape sequence \w
  strok = '(^%s|^HCP\w\w\w045E|^HCP\w\w\w361E)' % userid
./zvmsdk/sdkwsgi/validation/parameter_types.py:188: DeprecationWarning: invalid escape sequence \s
  'pattern': '^([0-9a-fA-F]{,4})(\s+[0-9a-fA-F]{,4}){,2}$'}
./zvmsdk/sdkwsgi/validation/parameter_types.py:235: DeprecationWarning: invalid escape sequence \w
  'pattern': '^https?:/{2}|^file:/{3}\w.+$'
./zvmsdk/sdkwsgi/validation/parameter_types.py:247: DeprecationWarning: invalid escape sequence \-
  'pattern': '^[a-zA-Z0-9\-]+\@([0-9]{1,3}(.[0-9]{1,3}){3}$|'
./zvmsdk/sdkwsgi/validation/parameter_types.py:248: DeprecationWarning: invalid escape sequence \-
  '[a-zA-Z0-9\-]+(.[a-zA-Z0-9\-]){1,}$)'
./zvmsdk/sdkwsgi/validation/parameter_types.py:256: DeprecationWarning: invalid escape sequence \w
  'pattern': '^(\w{,8})$'
./zvmsdk/sdkwsgi/validation/parameter_types.py:268: DeprecationWarning: invalid escape sequence \*
  {'pattern': '\*'},
./zvmsdk/sdkwsgi/validation/parameter_types.py:288: DeprecationWarning: invalid escape sequence \s
  'pattern': '^(\s*\w{1,8}\s*)(,\s*\w{1,8}\s*){0,}$'
./zvmsdk/sdkwsgi/validation/parameter_types.py:295: DeprecationWarning: invalid escape sequence \s
  'pattern': '^(\s*\w{1,8}\s*)(,\s*\w{1,8}\s*){0,}$'
./zvmsdk/sdkwsgi/validation/parameter_types.py:308: DeprecationWarning: invalid escape sequence \w
  'pattern': '^\w+:\w+$'
./zvmsdk/sdkwsgi/validation/parameter_types.py:315: DeprecationWarning: invalid escape sequence \w
  'pattern': '^\w+:\w+$',
./zvmsdk/sdkwsgi/validation/parameter_types.py:329: DeprecationWarning: invalid escape sequence \w
  'disk_pool': {'type': 'string', 'pattern': '^\w+:\w+$'}
./zvmsdk/tests/fvt/test_suites.py:33: DeprecationWarning: invalid escape sequence \W
  re.sub('\W', '_',
./zvmsdk/tests/fvt/test_utils.py:522: DeprecationWarning: invalid escape sequence \w
  if re.search('(^HCP\w\w\w003E)', output):

It would be great if Feilong supported zOS lifecycle, where zOS is 2nd level zVM guest:

1. IPL zOS,
2. Stop zOS
3. invoke MVS commands (D IPLINFO, ...) and capture their response

My understanding is that Feilong runs a zlinux under zVM, what it is resource consumption of this image (average CPU utilization, Memory, Disk). In my case I would ideally avoid extra VM with zlinux to reduce memory and CPU resources because I'm running zVM under zD&T (zPDT)

I already have a POC version implemented off the Feilong project here is an architecture diagram:

• In my case I can have many instances of zVM each representing a parallel sysplex, that could be provisioned by a developer, therefore I prefer to avoid zlinux on each zVM and instead have the majority of components off the zVM.
• Ideally, I would like to combine LXC and observer into a single VM to minimize overhead of infrastructure
• LXC is used to send commands such as: Q NAMES, CP SEND CP ZOS1 CP IPL ..., CP SEND CP ZOS1 VINPUT VMSG D IPLINFO, CP SET OBSERVER ...
• OBSERVER - IUCV2Socket - capture console messages sent from zOS vms and forward them via a network socket to a CLI that runs on linux, POC is based on
  • Managing z/OS guests on z/VM with IBM Operations Manager for z/VM interesting article describing how to interact with zOS master console using CP VINPUT VMSG xxx commands and how to capture responses using CP SET OBSERVER ...
  • Sample REXX program to listen on a TCPIP socket
  • Sample REXX program to trap IUCV messages using WAKEUP

What happens

According to the Feilong 1.6.5 documentation:

max_mem_kb	body	integer	The maximum memory in KBytes can be allocated for this guest.

However Feilong returns value 4194304.0, which is not an integer, but a floating point number, in case of a maximum memory of 4 GiB.

This produces an error when you try to unmarshal that value.

What should happen

GetGuestInfo() should return an integer for field max_mem_kb.

The other field mem_kb is an integer as expected.

Attach volume says in the parameter table that assigner_id is specified in (presumably, the request URI) path, but the example shows it within the connection dictionary.

Detach volume has the same discrepancy. Detach's parameters and example look like an exact copy of Attach's, which seems suspicious. I'm not sure how specifying is_root_volume on a Detach Volume request helps, or would be used. It might be that detach should say that extra/unknown parameters are ignored (that seems like it would be a useful statement at a higher level "all Feilong APIs...", if it's true, since unknown parameters could just as likely result in a request being rejected).

Currently the documentation still references z/VM Cloud Connector - probably should update to Feilong

I had Feilong 1.4 running and wanted to upgrade. 1th I took the latest version, so now I have version 1.6, but also has an error, although on the same call but different message. I show the 1.6 version now.

I have a guest TEST6, not running. I want to show his definition by running http://CC-server/guests/TEST6
The error is: "Unexpected internal error in ZVM SDK, error: Could not find the userid: Unexpected internal error in ZVM SDK, error: Command "sudo vmcp q TEST6" Error: init() got an unexpected keyword argument 'timeout'"

Calling sudo on command line gives:
sudo vmcp q TEST6
HCPCQU045E TEST6 not logged on
Error: non-zero CP response for command 'Q TEST6': #45

smcli works:
/opt/zthin/bin/smcli Image_Query_DM -T test6
USER TEST6 LBYONLY 1024M 2G G
INCLUDE LINUXFEI
COMMAND DEF STOR RESERVED 1024M
CPU 00 BASE
IPL CMS
MACHINE ESA 4
MDISK 0100 FB-512 138600032 25165824 LNXOS5 MW
MDISK 1300 9336 81760160 2097152 LNXOS6 MR
MDISK 1301 9336 1200032 20480 LNXOS1 MR
*DVHOPT LNK0 LOG1 RCM1 SMS0 NPW1 LNGAMENG PWC20201030 CRCXX

I made zthin and sdkserver from this 1.6 version.
Running Feilong on Suse 15 SP1.

I think we are ready for rename python-zvm-sdk to feilong, @jmertic please go ahead to do it. Thanks.

GET /guests says: List names of all the guests created by Feilong.

This is ambiguous: it ignores the running/not guest distinction, and (possibly) the registered/not one.

Running/not is cross-platform since some of Feilong's guest APIs specifically say they manipulate directory entries (defined domains, in libvirt terms), others specifically say they operate on running guests (active domains), and many like this one leave it up to the reader's imagination whether the guest resource represents a directory entry, running guest, or both.

Registered/not appears to be a Feilong-specific consideration, given that I don't find it in libvirt's concepts or z/VM documentation (but Feilong does document register and deregister APIs). In this specific case, the phrase "created by Feilong" might be intended to cover "registered as Feilong-managed -and- created by Feilong", but other cases certainly exist given the presence of the register API. For example, a guest might be created by Feilong and then later deregistered; under my "might..." reading such a guest should not be returned, but under a strict reading of the existing documentation it should be returned.

I see the same running/not ambiguity in "more than a few" of the current APIs. Would you prefer that I add the list here, or open a separate issue for each one linking here for the full explanation?

If IPv6 is enabled on the Linux guest, the "Get guest adapters info" API call [1] has a tendancy to return a link-local IPv6 address.

This is an address that starts with fe80: and that is unusable from any host that is not directly on same (switched) link.

=> If there is a router between the client and the guest, the address returned by "get guest adapters info" is unusable.

The problem is even deeper: a host adapter can have several IP addresses; several IPv4 and several IPv6. The current API method assumes each adapter is identified by only one address.

Typical output is:

    "output":
    {
        u'adapters': [{u'lan_owner': u'SYSTEM',
                       u'adapter_address': u'1000',
                       u'lan_name': u'VSC11590',
                       u'adapter_status': u'02',
                       u'mac_address': u'02:55:36:5D:48:57',
                       u'mac_ip_version': u'4',
                       u'mac_ip_address': u'9.152.85.152'}]
    }

but the last fields should be a list of addresses, ideally.

How to fix it?

SMAPI call Virtual_Network_Adapter_Query_Extended returns something like:

mac_count=7 

mac_address=01005E000001
mac_address_type=01
mac_status=00

mac_address=023AFC42002A
mac_address_type=00
mac_status=00
mac_ip_version=4
mac_ip_address=10.145.214.42

mac_address=023AFC42002A
mac_address_type=00
mac_status=00
mac_ip_version=6
mac_ip_address=2a07:de40:b208:2:3a:fcff:fe42:2a

mac_address=023AFC42002A
mac_address_type=00
mac_status=00
mac_ip_version=6
mac_ip_address=fe80::3a:fcff:fe42:2a

mac_address=333300000001
mac_address_type=01
mac_status=00

mac_address=3333FF42002A
mac_address_type=01
mac_status=00

mac_address=3333FFC48072
mac_address_type=01
mac_status=00

mac_info_end

The current code in smtclient.py overwrites every IP address in this output with the next one 😿 .

• Solution 1 (breaks compatibility):

  return a full list with all this information [ {mac1, ip1, iptype1}, {mac2, ip2, iptype2} ...];
  this is the best solution as it matches both SMAPI and network reality.

• Solution 2: (keeps compatibility):

  refrain from returning a link-local address
  for example by always stopping at first valid IP address (seems to be the IPv4).

I can provide a PR for whatever solution you prefer (including not listed).

[1] https://cloudlib4zvm.readthedocs.io/en/latest/restapi.html#get-guest-adapters-info

It might be useful to use both a cloud-init parameters disk and a network.doscript script to initialize a guest.

In that case, the only way I have found to do that is to specify "transportfiles:" twice in the json used by DeployGuest.

However, RFC 8259 discourages the use of duplicate keys in a json file. As a matter of fact, several json serializers / deserializers cannot cope with them.

Did I miss something? Or should the value of transportfiles be an array of strings rather than a single string?

I got the following error when I tried to bring up the sdkserver service. There are no FCP devices on my VM system. The smapi SYSTEM_WWPN_QUERY call returned RC=4 (warning return code) with an empty output buffer, but the exception on the python layer was taken due to RC<>0. It looks to me that RC=4 from the SYSTEM_WWPN_QUERY call should not prevent sdkserver from up running.

[root@linux076 zvmsdk]# vmcp q fcp all
An active FCP was not found.
A free FCP was not found.
An offline FCP was not found.
An FCP Agent was not found.

[root@linux076 zvmsdk]# journalctl -u sdkserver
Aug 30 16:33:19 linux076.lvn.broadcom.net systemd[1]: Started zVM SDK API server.
Aug 30 16:33:45 linux076.lvn.broadcom.net sudo[4093]: zvmsdk : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/vmcp query userid
Aug 30 16:33:46 linux076.lvn.broadcom.net sudo[4097]: zvmsdk : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/vmcp query userid
Aug 30 16:33:47 linux076.lvn.broadcom.net sudo[4101]: zvmsdk : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/vmcp query userid
Aug 30 16:33:47 linux076.lvn.broadcom.net sudo[4105]: zvmsdk : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/opt/zthin/bin/smcli System_WWPN_Query --addRCheader -T LINUX076 -k OWNER=YES
Aug 30 16:33:49 linux076.lvn.broadcom.net python3.6[4085]: detected unhandled Python exception in '/usr/local/bin/sdkserver'
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: Traceback (most recent call last):
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: File "/usr/local/lib/python3.6/site-packages/zVMCloudConnector-1.6.3-py3.6.egg/zvmsdk/utils.py", line 421, in log_and_reraise_smt_request_failed
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: yield
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: File "/usr/local/lib/python3.6/site-packages/zVMCloudConnector-1.6.3-py3.6.egg/zvmsdk/smtclient.py", line 291, in get_fcp_info_by_status
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: results = self._request(rd)
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: File "/usr/local/lib/python3.6/site-packages/zVMCloudConnector-1.6.3-py3.6.egg/zvmsdk/smtclient.py", line 140, in _request
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: raise exception.SDKSMTRequestFailed(results, msg)
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: zvmsdk.exception.SDKSMTRequestFailed: SMT request failed. RequestData: 'getvm LINUX076 fcpinfo all YES', Results: '{'overallRC': 8, 'rc': 4, 'rs': 28, 'errno': 0, 'strErro>
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: During handling of the above exception, another exception occurred:
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: Traceback (most recent call last):
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: File "/usr/local/bin/sdkserver", line 4, in
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: import('pkg_resources').run_script('zVMCloudConnector==1.6.3', 'sdkserver')
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 654, in run_script
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: self.require(requires)[0].run_script(script_name, ns)
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 1434, in run_script
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: exec(code, namespace, namespace)
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: File "/usr/local/lib/python3.6/site-packages/zVMCloudConnector-1.6.3-py3.6.egg/EGG-INFO/scripts/sdkserver", line 24, in
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: sdkserver.start_daemon()
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: File "/usr/local/lib/python3.6/site-packages/zVMCloudConnector-1.6.3-py3.6.egg/zvmsdk/sdkserver.py", line 264, in start_daemon
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: server = SDKServer()
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: File "/usr/local/lib/python3.6/site-packages/zVMCloudConnector-1.6.3-py3.6.egg/zvmsdk/sdkserver.py", line 42, in init
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: self.sdkapi = api.SDKAPI()
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: File "/usr/local/lib/python3.6/site-packages/zVMCloudConnector-1.6.3-py3.6.egg/zvmsdk/api.py", line 99, in init
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: self._volumeop = volumeop.get_volumeop()
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: File "/usr/local/lib/python3.6/site-packages/zVMCloudConnector-1.6.3-py3.6.egg/zvmsdk/volumeop.py", line 60, in get_volumeop
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: _VolumeOP = VolumeOperatorAPI()
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: File "/usr/local/lib/python3.6/site-packages/zVMCloudConnector-1.6.3-py3.6.egg/zvmsdk/volumeop.py", line 85, in init
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: VolumeOperatorAPI._fcp_manager_obj = FCPVolumeManager()
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: File "/usr/local/lib/python3.6/site-packages/zVMCloudConnector-1.6.3-py3.6.egg/zvmsdk/volumeop.py", line 1666, in init
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: self.fcp_mgr = FCPManager()
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: File "/usr/local/lib/python3.6/site-packages/zVMCloudConnector-1.6.3-py3.6.egg/zvmsdk/volumeop.py", line 478, in init
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: self.sync_db()
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: File "/usr/local/lib/python3.6/site-packages/zVMCloudConnector-1.6.3-py3.6.egg/zvmsdk/volumeop.py", line 487, in sync_db
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: self._sync_db_with_zvm()
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: File "/usr/local/lib/python3.6/site-packages/zVMCloudConnector-1.6.3-py3.6.egg/zvmsdk/volumeop.py", line 914, in _sync_db_with_zvm
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: fcp_dict_in_zvm = self.get_fcp_dict_in_zvm()
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: File "/usr/local/lib/python3.6/site-packages/zVMCloudConnector-1.6.3-py3.6.egg/zvmsdk/volumeop.py", line 799, in get_fcp_dict_in_zvm
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: fcp_dict_in_zvm = self.get_all_fcp_pool(smt_userid)
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: File "/usr/local/lib/python3.6/site-packages/zVMCloudConnector-1.6.3-py3.6.egg/zvmsdk/volumeop.py", line 738, in get_all_fcp_pool
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: all_fcp_info = self._get_all_fcp_info(assigner_id)
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: File "/usr/local/lib/python3.6/site-packages/zVMCloudConnector-1.6.3-py3.6.egg/zvmsdk/volumeop.py", line 490, in _get_all_fcp_info
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: fcp_info = self._smtclient.get_fcp_info_by_status(assigner_id, status)
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: File "/usr/local/lib/python3.6/site-packages/zVMCloudConnector-1.6.3-py3.6.egg/zvmsdk/smtclient.py", line 291, in get_fcp_info_by_status
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: results = self._request(rd)
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: File "/usr/lib64/python3.6/contextlib.py", line 99, in exit
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: self.gen.throw(type, value, traceback)
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: File "/usr/local/lib/python3.6/site-packages/zVMCloudConnector-1.6.3-py3.6.egg/zvmsdk/utils.py", line 428, in log_and_reraise_smt_request_failed
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: raise exception.SDKSMTRequestFailed(err.results, msg)
Aug 30 16:33:51 linux076.lvn.broadcom.net sdkserver[4085]: zvmsdk.exception.SDKSMTRequestFailed: Failed to query fcp info of 'LINUX076'. SMT error: SMT request failed. RequestData: 'getvm LINUX076 fcpinfo all YES', Results: '{'ove>
Aug 30 16:33:51 linux076.lvn.broadcom.net systemd[1]: sdkserver.service: Main process exited, code=exited, status=1/FAILURE
Aug 30 16:33:52 linux076.lvn.broadcom.net systemd[1]: sdkserver.service: Failed with result 'exit-code'.

Thanks
Hongjie Yang

Hi team,

To increase the space used for specific mount points on linux, we generated a new gold image, but when we try create a new guest basead on the new image, unpackdiskimage report incorrectly the size volume. In this case bellow, both z/vm guests and image generated has 50000 cylinders, 20032 is the size for old image. I can't find where he is getting this info to adjust.. any help?

[2023-12-06 14:22:20] [DEBUG] Get msg to handle: {u'rs': 3, u'overallRC': 300, u'modID': 10, u'rc': 300, u'output': u'', u'errmsg': u"Failed to deploy image to userid: 'RFSD1010', unpackdiskimage failed with rc: 3, error: \nERROR: Target disk is too small for specified image, required:20032 cylinders, given:50000 cylinders"}

[2023-12-06 14:22:20] [INFO] The msg <{u'rs': 3, u'overallRC': 300, u'modID': 10, u'rc': 300, u'output': u'', u'errmsg': u"Failed to deploy image to userid: 'RFSD1010', unpackdiskimage failed with rc: 3, error: \nERROR: Target disk is too small for specified image, required:20032 cylinders, given:50000 cylinders"}> lead to return internal error

[2023-12-06 14:22:20] [INFO] 172.20.32.204 "POST /guests/rfsd1010/action" status: 500 length: 270 headers: [('Content-Length', '270'), ('Content-Type', 'application/json; charset=UTF-8'), ('cache-control', 'no-cache')] exc_info: None
[2023-12-06 14:22:20] [DEBUG] [Thread-152] No more item in request queue, worker will exit now.

GET /host/diskpool talks about "disk pools", which are not a native z/VM concept. Libvirt has storage pools, "storage" means memory (to CP) or disk (to DirMaint, which usually calls it DASD), so I understand that a different term is needed. Since Feilong is z/VM-specific, consumers need to be clear about what the API maps to; my best would be a DirMaint DASD Group, but there might be another animal in z/VM's disk menagerie that I have not run across yet.

In section https://cloudlib4zvm.readthedocs.io/en/latest/restapi.html#guest-register , one reads:

meta   body  string  The metadata of the vm to be registered.

and

{
   "action": "register_vm",
   "meta": '{"os_version": "sles12sp3"}',
   "net_set": "1",
}

The field meta cannot be both a string and a dictionary. Which one is it?

According to the source code, I think it is rather a string, but I am not 100% sure. If so, what would be a correct example?

The problem

When trying to use "Attach Volume" Feilong API call (https://cloudlib4zvm.readthedocs.io/en/latest/restapi.html#attach-volume) one gets an error message like:

{HTTP status: 404, body: {"overallRC": 404, "modID": 30, "rc": 404, "rs": 1, "errmsg":
 "FCP with id: fc04 does not exist.", "output": ""}

After debugging, it appears it is because the information about the FCP device was not stored in the database at the startup of zvmsdk. This is turn is caused by:

[2024-01-17 16:56:18] [ERROR] Failed to get PCHID for the CHPID '60' with command
 '['sudo', '/sbin/vmcp', 'query chpid', '60', 'pchid']'. Check the z/VM userid 'FEILONG' on the z/VM
  'S390ZL0C' is authorized to run the CP command: 'QUERY CHPID yy PCHID'. 

The solutions

The remediation is to add privilege class B to the ZVM connector VM (Feilong).

Ideally, Feilong should use something else than vmcp query chpid XX pchid to query zVM, because this command requires class B.

If that is not technically possible, the issue should at least be documented in Feilong's documentation. I suggest adding:

6. Add privilege class B

If you are planning to use Fiber Channel volumes, you need to add privilege class B to your BYOL:

replace G with BG at the end of the first definition line (starting with USER).

to section 3.1.1. Preparation on BYOL (https://cloudlib4zvm.readthedocs.io/en/latest/quickstart.html#preparation-on-byol), and changing final note:

Note Preparation step 2 and step 3 require to logoff then re-logon the BYOL to make
the updates become effective.

to:

Note Preparation step 2 , step 3, and step 6 require to logoff then re-logon the BYOL to make
the updates become effective.

doc/source/quickstart.rst lines 146-182 documents using a build project. The build project is no longer available at the URL in step 1. Should the information in this section be rewritten to explain how to use make to build and install the zthin components or the buildrpm project be incorporated into feilong?

Currently, the NICDEF statement is repeated twice in couple_nic_to_vswitch of zvmsdk/smtclient.py and anything after NICDEF XXX like MACID is removed.

A simple fix would be

                    v = ent
                    if vlan_id < 0:
                        v += " LAN SYSTEM %s" % vswitch_name
                    else:
                        v += " LAN SYSTEM %s VLAN %s PORTTYPE %s" \
                             % (vswitch_name, vlan_id, port_type)

                    new_user_direct[-1] = v

Attempting to add a minidisk using EDEV space through smcli. For example:

zaddedev 121 154 1g -v
...
Adding EDEV 154 of size 1G using pool ZSDED to D2S00121 directory entry
zActionCmd command: sudo /usr/local/sbin/smcli Image_Disk_Create_DM -T D2S00121 -u 2 -m MR -v 154 -t 9336 -a AUTOG -r ZSDED -z 2097152 -f 2
.. stalls here for 2-3 minutes...
ERROR: sudo /usr/local/sbin/smcli Image_Disk_Create_DM -T D2S00121 -u 2 -m MR -v 154 -t 9336 -a AUTOG -r ZSDED -z 2097152 -f 2 returned 1
Adding a disk to D2S00121's directory entry... Started Return Code: 592 Reason Code: 111074641 Description: Asynchronous operation started - product-specific asynchronous operation ID : 111074641

The 154 MDISK is not added to the user directory entry.

POST /guests allows a list of dedicate_vdevs when creating a guest. The associated request sample omits it, so the format of each entry is ambiguous (string? dict?). DirMaint 7.2's DIRM DEDICATE command documentation suggests that a system device name (rdev or volid) is minimally required for each dedicated device, unless this API carries an implicit assumption that the corresponding directory entry statements would be populated out of band (I find zero other mentions of "dedicated" in the entire REST API doc page).

It's also unclear how any dedicated devices would behave if their vdevs are included on the 3 /guests/{userid}/disks method calls. Presumably (by omission) Feilong does not support adding dedicated disks after the guest is created. Modify and delete might apply regardless of how the disk was created, however.

The TAC has recommended SPDX usage and simplified copyright notices for it's hosted projects ( see https://github.com/openmainframeproject/tac/blob/master/process/contribution_guidelines.md#license-specification ). We are seeing larger usage of this pattern as well outside of OMP project ( https://www.linuxfoundation.org/blog/2020/01/copyright-notices-in-open-source-software-projects/ ).

For this project, the recommendation would be to change headers to this for code...

#  Copyright Contributors to the Feilong Project.
#  SPDX-License-Identifier: Apache-2.0

and this for documentation

#  Copyright Contributors to the Feilong Project.
#  SPDX-License-Identifier: CC-BY-4.0

There are numerous files with IBM copyright notices; if they are to be removed it must be by an IBM employee.

Let me know of any questions/concerns.