ocp-power-automation / ocp4-upi-powervs

OpenShift on Power Virtual Server

Table of Contents

• Table of Contents
• Introduction
• Getting Started With PowerVS
• Automation Host Prerequisites
• PowerVS Prerequisites
• OCP Install
• Contributing

Introduction

The ocp4-upi-powervs project provides Terraform based automation code to help with the deployment of OpenShift Container Platform (OCP) 4.x on IBM® Power Systems™ Virtual Server on IBM Cloud.

This project leverages the helpernode ansible playbook internally for OCP deployment on IBM Power Systems Virtual Servers (PowerVS).

!!! Note For bugs/enhancement requests etc. please open a GitHub issue

!!! Note Use the main branch to install any versions of OCP starting from 4.6, including pre-release versions.

Getting Started With PowerVS

• Power Systems Virtual Servers(IBM Cloud Docs)
• IBM Power Systems in the Multicloud(Youtube video)
• PowerVS (Youtube video)

Automation Host Prerequisites

The automation needs to run from a system with internet access. This could be your laptop or a VM with public internet connectivity. This automation code have been tested on the following Operating Systems:

• Mac OSX (Darwin)
• Linux (x86_64/ppc64le)
• Windows 10

Follow the guide to complete the prerequisites.

PowerVS Prerequisites

Follow the guide to complete the PowerVS prerequisites.

OCP Install

Follow the quickstart guide for OCP installation on PowerVS.

Contributing

Please see the contributing doc for more details. PRs are most welcome !!

ocp4-upi-powervs's Issues

Reserve external network address for the use of public VIP for HA clusters

Similar to #44 but this issue is to support external IP to be used as a VIP for API/Ingress endpoints which will point to the keepalived master bastion.

Create T-Shirt size clusters configuration for deployment

To easy the first time experience for those not familiar with OCP or PowerVS and reduce the amount of the necessary configurations we could provide predefined flavors: small, medium, large, x-large. This option could be set from the command line, during the execution:

terraform apply -var-file var.tfvars -size=small

Suggestions:

small

bastion     = {memory      = "16",   processors  = "1",    "count"   = 1}
bootstrap   = {memory      = "32",   processors  = "0.5",  "count"   = 1}
master      = {memory      = "32",   processors  = "0.5",  "count"   = 3}
worker      = {memory      = "32",   processors  = "0.5",  "count"   = 2}

medium

bastion     = {memory      = "16",   processors  = "1",    "count"   = 1}
bootstrap   = {memory      = "32",   processors  = "0.5",  "count"   = 1}
master      = {memory      = "32",   processors  = "0.5",  "count"   = 3}
worker      = {memory      = "32",   processors  = "1",  "count"   = 4}

large

bastion     = {memory      = "16",   processors  = "1",    "count"   = 1}
bootstrap   = {memory      = "32",   processors  = "0.5",  "count"   = 1}
master      = {memory      = "32",   processors  = "1",  "count"   = 3}
worker      = {memory      = "64",   processors  = "1",  "count"   = 6}

x-large

bastion     = {memory      = "16",   processors  = "1",    "count"   = 1}
bootstrap   = {memory      = "32",   processors  = "0.5",  "count"   = 1}
master      = {memory      = "32",   processors  = "1",  "count"   = 3}
worker      = {memory      = "64",   processors  = "2",  "count"   = 8}

Delete bastion node after master and worker nodes

The bastion node is not automatically destroyed with the cluster.

Create Terraform vars for test/dev cluster, CP4D, etc for release branches

dhcpd is always running on env2 interface

The DHCP configuration using the helpernode is always using the default interface name env2. This is because of ip r command is returning multiple lines for the sed command to fail with below error:

module.install.null_resource.config (remote-exec): sed: -e expression #1, char 45: unknown command: `.'

Creating Bastion manually with Public IP and using the same for Install

For cases, where we see higher failure rate with public network reachability, can we ask the end-user to create Bastion with public IP as a pre-requisite and validate if the public IP is reachable before standing up OCP using TF automation ?

Understand this is not the best experience, however this will ensure a working cluster is available.

Fix DirectLink interface bug in template

module.install.null_resource.powervs_config[0] (remote-exec): sed: -e expression #1, char 33: unknown command: `.'
module.install.null_resource.powervs_config[0] (remote-exec): Running powervs specific nodes configuration playbook...

Could not deploy OCP in lon06 zone

I could not deploy OCP 4.6 in London06 zone. All nodes except bastion are in a reboot loop, failing to download ign file.

Set http fetch timeout for ignition hosted on HTTP sever

Ignition fetch stage continue to GET the HTTP file with default timeout set to 0 (indefinite).

Need to set the time to proper value in the automation.

Note: Ignition provider does not allow to change the timeout. Need to use string manipulation to add timeout in rendered ignition string.

Allow selecting which IP to use to SSH into the cluster

By default we are using the public IP to SSH into the cluster. This covers the case where we assume someone is running this automation from a remote source, outside PowerVS. Eventually someone will have a full deployment only within PowerVS using private IPs, without using any route outside this environment.

I would be a good feature if we can select via TF which IP we want to use: bastion_public_ip or bastion_private_ip.

Support for OCP 4.6 installation

Going forward to support OCP 4.6 we need to:

Update the ignition config spec to v3.1.0 which is included in https://github.com/community-terraform-providers/terraform-provider-ignition.

Ignition resource changes will be needed.

Set specific git clone commands for each branch

To avoid misunderstandings regarding using specific versions of this code, it would be good to add in the docs the following:

Clone the correct branch based on the OpenShift version you would like to install:

For OCP 4.6:

git clone --single-branch --branch release-4.6 https://github.com/ocp-power-automation/ocp4-upi-powervs.git

For OCP 4.5:

git clone --single-branch --branch release-4.5 https://github.com/ocp-power-automation/ocp4-upi-powervs.git

doc update suggestions

Creating this issue to leave some items in for later...

Intermittent issue while getting public IP for ssh connections

Public IP data source not working at initial step. Could be issue populating the public vlan at the API side.

    {
      "module": "module.prepare",
      "mode": "data",
      "type": "ibm_pi_instance_ip",
      "name": "bastion_public_ip",
      "each": "list",
      "provider": "provider.ibm",
      "instances": [
        {
          "index_key": 0,
          "schema_version": 0,
          "attributes": {
            "external_ip": "158.176.146.37",
            "id": "c67f6948-8f24-486d-8ade-62560886403e",
            "ip": "192.168.139.37",
            "ipoctet": "37",
            "macaddress": "fa:5c:f9:80:7f:21",
            "network_id": "c67f6948-8f24-486d-8ade-62560886403e",
            "pi_cloud_instance_id": "729b4527-6709-4ecb-8803-e0f1709ea945",
            "pi_instance_name": "yus-ha-bastion-0",
            "pi_network_name": "yus-ha-pub-net",
            "type": "fixed"
          }
        },
      ]
    },

This is only first time, subsequent apply works without any issues and the data resource is populated. But the bastion node resource has the external IP.

Support for local registry setup

• Add support to configure the local registry on the bastion node.

Bastion node does not connect over public IP and does not have internet access

On PowerVS we use both private and public network for the bastion. External IP is used to connect to the bastion from the TF terminal.

Actual Behaviour: The bastion node does not connect via external IP on specific zone.

Excepted Behaviour: The external IP should work as the only way to connect via the TF terminal.

Additional Info: The bastion node always connects via external IP when using the UI console. Also, we don't need to provide the public network information while creating the node.

Leveraging VM pinning in Power VS

https://cloud.ibm.com/docs/power-iaas?topic=power-iaas-power-iaas-faqs#pinning

Image upgrade fails when providing the upgrade_image

console:

module.install.null_resource.upgrade[0]: Creating...
module.install.null_resource.upgrade[0]: Provisioning with 'file'...
module.install.null_resource.upgrade[0]: Provisioning with 'remote-exec'...
module.install.null_resource.upgrade[0] (remote-exec): Connecting to remote host via SSH...
module.install.null_resource.upgrade[0] (remote-exec):   Host: 169.48.X.X
module.install.null_resource.upgrade[0] (remote-exec):   User: root
module.install.null_resource.upgrade[0] (remote-exec):   Password: false
module.install.null_resource.upgrade[0] (remote-exec):   Private key: true
module.install.null_resource.upgrade[0] (remote-exec):   Certificate: false
module.install.null_resource.upgrade[0] (remote-exec):   SSH Agent: false
module.install.null_resource.upgrade[0] (remote-exec):   Checking Host Key: false
module.install.null_resource.upgrade[0] (remote-exec): Connected!
module.install.null_resource.upgrade[0] (remote-exec): Running ocp upgrade playbook...
module.install.null_resource.upgrade[0] (remote-exec): Using /root/ocp4-playbooks/ansible.cfg as config file
module.install.null_resource.upgrade[0] (remote-exec): [WARNING]: Found both group and host with same name: bootstrap

module.install.null_resource.upgrade[0] (remote-exec): PLAY [Upgrade OCP cluster] *****************************************************

module.install.null_resource.upgrade[0] (remote-exec): TASK [Gathering Facts] *********************************************************
module.install.null_resource.upgrade[0] (remote-exec): ok: [test-ocp-6f2c-bastion-0]

module.install.null_resource.upgrade[0] (remote-exec): TASK [include_role : ocp-upgrade] **********************************************
module.install.null_resource.upgrade[0] (remote-exec): fatal: [test-ocp-6f2c-bastion-0]: FAILED! => {"msg": "The conditional check 'upgrade_version != \"\"' failed. The error was: error while evaluating conditional (upgrade_version != \"\"): 'upgrade_version' is undefined\n\nThe error appears to be in '/root/ocp4-playbooks/playbooks/upgrade.yaml': line 5, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n  tasks:\n    - include_role:\n      ^ here\n"}

module.install.null_resource.upgrade[0] (remote-exec): PLAY RECAP *********************************************************************
module.install.null_resource.upgrade[0] (remote-exec): test-ocp-6f2c-bastion-0    : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0



Error: error executing "/tmp/terraform_1158875228.sh": Process exited with status 2

Adopt Terraform 0.13

Terraform validate is failing with the latest terraform 0.13 release, this issue to make the necessary change to work properly on latest terraform version.

Revert the change here once we address this issue - ppc64le-cloud/test-infra#29

/assign @yussufsh @Prajyot-Parab

Verify whether or not IPs that belongs to the public network created by the automation are being used

Once the cluster is up and running it will be natural to expand the resources around it and eventually use the same pub network created to deploy the cluster for another VMs. For instance, I manually created a new NFS server in the pub network created by the TF automation. However, when deleting the cluster (specifically the network), it fails because of the IPs of that network is allocated.

To ensure this error does not happen, we should verify during the deletion of the network if there is any port allocated to a VM which was not created by the automation and either leave the network there or ask about forcing the resources deleting completely.

Error:

Error: {"description":"an error has occurred; please try again: unable to delete subnet for network 4f9be979-bafd-4478-bbfd-3023c9a43171 for cloud instance dad1780f13c54b8a9f7ea55778e965e0: Expected HTTP response code [] when accessing [DELETE https://192.168.24.136:9696/v2.0/subnets/87e3804c-301c-4b2e-8d69-0655a4d26d14], but got 409 instead\n{\"NeutronError\": {\"message\": \"Unable to complete operation on subnet 87e3804c-301c-4b2e-8d69-0655a4d26d14: One or more ports have an IP allocation from this subnet.\", \"type\": \"SubnetInUse\", \"detail\": \"\"}}","error":"internal server error"}

Not wait for long to timeout during bastion destroy

Currently it is waiting for 15m in destroy provisioner for bastion pi_instance. In case the bastion was not accessible over ssh TF waits to timeout after 15m.
We should exit in less time since ssh is not working which could mean the machine was never accessible. We can ignore the destroy commands in this case.

Use rhel_username instead of root in playbook inventory files

The user for bastion nodes "ansible_user=root" is hardcoded in inventory templates.

• modules/5_install/templates/helpernode_inventory
• modules/5_install/templates/install_inventory

rh subscription failure should fail the installation

Currently if the subscription-manager subscribe step fails, the automation continues. Since no packages are installed, the bastion setup will not complete, which prevents the cluster installation. I see that the subscription manager registration is done via a series of shell commands inside an inline block. From what I've read, this will only fail if the very last command fails. I've experimented with dropping a set -e into the top of the inline section, but that still did not seem to cause the registration failure to halt the automation.

Restriction on bastion count to 2 not working for HA

Maximum we should have 2 bastions in HA mode. The check is still with count <= 3.

Intermittent error getting public network details using ibm_pi_network data source

Error seen during TF apply getting data source for public network.

module.prepare.ibm_pi_volume.volume[0]: Creation complete after 16s [id=fc0db58c-a3d3-433c-8175-e9f5f814d83c/1d065ef7-5583-405e-b865-8a7511685d2e]
Error: {"description":"an error has occurred; please try again: unable to get network, subnet, and possibly public network cidr for network 0a91e6a7-63c0-490b-8855-8d590eb7528b on cloud instance a007eb5664ab4188a236468c86e51f16","error":"internal server error"}
  on modules/1_prepare/prepare.tf line 49, in data "ibm_pi_network" "public_network":
  49: data "ibm_pi_network" "public_network" {

Remove the health check for the bastion during create

When creating the bastion node we check for pi_health_status and allow the creation to complete when the value is WARNING.

We have seen issues that the bastion fails to connect via ssh with the root cause of going into grub/maintenance prompt. Removing the health status check will help for the TF to fail on the creation and not while attempting to ssh.

Need to keep health check to default just for the bastion node.

Add RSCT deployment to OCP nodes

Run the project from IBM Power systems

Similar to the work done in ocp-power-automation/ocp4-upi-powervm#33 and ocp-power-automation/ocp4-upi-powervm#166 we need the documentation updated to use the automation on Power.

Support running terraform again with bootstrap count as 0

Currently terraform apply will fail if we delete the bootstrap node and taint the config & install resources.
To overcome this we have similar change in helpernode. The TF changes is needed to not set bootstrap details in playbook vars.

Update playbook tag to fix chrony conf file overwrite issue

Install playbook execution failing with default chrony config

Install module is failing with default chrony config ie: chrony_config=true and chrony_config_servers=[].

module.install.null_resource.install (remote-exec): TASK [ocp-config : Configure chrony to synchronize with ntp servers] ***********
module.install.null_resource.install (remote-exec): fatal: [****-9433-bastion-0]: FAILED! => {"msg": "Invalid data passed to 'loop', it requires a list, got this instead: . Hint: If you passed a list/dict of just one element, try adding wantlist=True to your lookup invocation or use q/query instead of lookup."}

This is introduced with the validation PR #133 where the type of chrony_config_servers mentioned is...

type    = list(object({
       server  = string,
       options = string
   }))

Allow bastion node to use pi_health_status as an input variable

This issues is to perform bastion health status check with an optional variable. That way one can set the value as OK/WARNING if you don't want to wait till health status is ACTIVE for bastion.

The default will be OK as before with no change in the flow. We can decide later if we want to change the default to WARNING instead for bastion.

Note that the other cluster nodes are already using WARNING as health_status as they need the ignition file to boot and RMC pods are creating after deployment. We cannot use OK as health_status for them.

Make de NFS provisioner configuration optional

As an user deploying my own cluster I would like to avoid getting the NFS provisioner configured out of the box without explicitly setting it via vat.tf. In a production deployment I will probably use an already existing or a new NFS server and will include some customization for my deployment to meet some requirements.

I had to execute the following before installing a customized one:

oc delete all -l app=nfs-client-provisioner -n nfs-provisioner
oc delete project nfs-provisioner

Document steps to create a custom bastion instance via PowerVS UI

This task is to create a guide with detailed steps to be performed while creating a bastion instance via PowerVS UI and using the same for OCP cluster deployment.

error with terraform destroy-time provisioner variable access

I bumped up my terraform version from 0.12.29 to 0.13.x and hit this:

> terraform init                                               
Initializing modules...                                                                             
There are some problems with the configuration, described below.                                    
                                                                                                    
The Terraform configuration must be valid before initialization so that                             
Terraform can determine which modules and providers need to be installed.
                                                                                                    
Error: Invalid reference from destroy provisioner                                                   
                                                                                                    
  on modules/1_prepare/prepare.tf line 93, in resource "ibm_pi_instance" "bastion":                 
  93:             user        = var.rhel_username                                                   
                                                                                                    
Destroy-time provisioners and their connection configurations may only                              
reference attributes of the related resource, via 'self', 'count.index', or                         
'each.key'.                                                                                         
                                                                                                    
References to other resources during the destroy phase can cause dependency
cycles and interact poorly with create_before_destroy.

Confirmed that moving back to 0.12 made this go away.

I think this is the same as hashicorp/terraform#23679

Create Terraform vars for test/dev cluster, CP4D, etc

TF is failing on deleting the cloud-init package

module.prepare.null_resource.rhel83_fix[0] (remote-exec): Transaction Summary
module.prepare.null_resource.rhel83_fix[0] (remote-exec): ========================================
module.prepare.null_resource.rhel83_fix[0] (remote-exec): Remove  1 Package

module.prepare.null_resource.rhel83_fix[0] (remote-exec): Freed space: 3.0 M
module.prepare.null_resource.rhel83_fix[0] (remote-exec): Running transaction check
module.prepare.null_resource.rhel83_fix[0] (remote-exec): Transaction check succeeded.
module.prepare.null_resource.rhel83_fix[0] (remote-exec): Running transaction test
module.prepare.null_resource.rhel83_fix[0] (remote-exec): Transaction test succeeded.
module.prepare.null_resource.rhel83_fix[0] (remote-exec): Running transaction
module.prepare.null_resource.rhel83_fix[0] (remote-exec):   Preparing        :                1/1
module.prepare.null_resource.rhel83_fix[0] (remote-exec):   Running scriptlet: cloud-init-1   1/1

module.prepare.null_resource.rhel83_fix[0] (remote-exec):   Erasing          : cloud [      ] 1/1
module.prepare.null_resource.rhel83_fix[0] (remote-exec):   Erasing          : cloud [=     ] 1/1
module.prepare.null_resource.rhel83_fix[0] (remote-exec):   Erasing          : cloud [==    ] 1/1
module.prepare.null_resource.rhel83_fix[0] (remote-exec):   Erasing          : cloud [===   ] 1/1
module.prepare.null_resource.rhel83_fix[0] (remote-exec):   Erasing          : cloud [====  ] 1/1
module.prepare.null_resource.rhel83_fix[0] (remote-exec):   Erasing          : cloud [===== ] 1/1
module.prepare.null_resource.rhel83_fix[0] (remote-exec):   Erasing          : cloud-init-1   1/1
module.prepare.null_resource.rhel83_fix[0] (remote-exec): warning: /etc/cloud/cloud.cfg saved as /etc/cloud/cloud.cfg.rpmsave

module.prepare.null_resource.rhel83_fix[0] (remote-exec):   Running scriptlet: cloud-init-1   1/1
module.prepare.null_resource.rhel83_fix[0] (remote-exec):   Verifying        : cloud-init-1   1/1
module.prepare.null_resource.rhel83_fix[0] (remote-exec): Installed products updated.

module.prepare.null_resource.rhel83_fix[0] (remote-exec): Removed:
module.prepare.null_resource.rhel83_fix[0] (remote-exec):   cloud-init-19.4-11.el8_3.1.noarch

module.prepare.null_resource.rhel83_fix[0] (remote-exec): Complete!
module.prepare.null_resource.rhel83_fix[0]: Creation complete after 8s [id=9048021131775509033]

Error: error executing "/tmp/terraform_1915964773.sh": Process exited with status 2

terraform destroy error

module.nodes.null_resource.remove_worker[0] (remote-exec): pod/redhat-marketplace-4mf5r evicted
module.nodes.null_resource.remove_worker[0] (remote-exec): pod/prometheus-adapter-79bd7d5c9-ffzjn evicted
module.nodes.null_resource.remove_worker[0]: Still destroying... [id=7928149752953593866, 30s elapsed]
module.nodes.null_resource.remove_worker[1]: Still destroying... [id=4203784824677227330, 30s elapsed]
module.nodes.ibm_pi_instance.master[2]: Still destroying... [id=fac4755e-8aff-45f5-8d5c-1d3b58b7a229/c141c210-316f-4a76-9a5a-7f9a9ddf3e16, 30s elapsed]
module.nodes.null_resource.remove_worker[1] (remote-exec): There are pending pods in node "worker-1" when an error occurred: error when evicting pod "router-default-6d99c496cc-rl2n4": rpc error: code = Unavailable desc = transport is closing
module.nodes.null_resource.remove_worker[1] (remote-exec): pod/router-default-6d99c496cc-rl2n4
module.nodes.null_resource.remove_worker[1] (remote-exec): Following errors occurred while getting the list of pods to delete:
module.nodes.null_resource.remove_worker[1] (remote-exec): cannot delete rpc error: code = Unavailable desc = transport is closing: openshift-machine-config-operator/machine-config-daemon-lb9qmerror: unable to drain node "worker-1", aborting command...

module.nodes.null_resource.remove_worker[1] (remote-exec): There are pending nodes to be drained:

Add Owners file

Add owners file for the project for Prow

`terraform destroy` leaves behind resources after a failed `terraform apply`

After a cluster creation attempt that fails, I've seen the terraform destroy command complete but still leave behind instances and volumes. The scenario seems to be that the cluster creation is attempted, instances are created, but never boot. The instances created include the bastion, bootstrap, workers, and masters. All of these are left behind in the case that none of them come up.

Document on how to use the RHEL developer subscription for the Bastion

https://www.cyberciti.biz/linux-news/red-hat-introduces-new-no-cost-rhel-option/

https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/registrations?client_id=rhd-web&redirect_uri=https%3A%2F%2Fdevelopers.redhat.com%2Fconfirmation&state=93c685fc-b7c9-47e6-8048-f0a6e4c1c909&response_mode=fragment&response_type=code&scope=openid&nonce=4b460df8-8838-4803-9684-a50f6ea0a79b

Reserve private network address for the use of VIP for HA clusters

For using a network address as a VIP in HA mode we need to create a network port. The network port will have a macaddress and network address which can be used as a VIP.

Above VIP can be used by passing to the helpernode playbook having HA support. (depends on redhat-cop/ocp4-helpernode#150)

Allow using PowerVS catalog images for bastion

Now we have CentOS 8.3 image as part of the image catalog from PowerVS.

The Terraform automation should be able to use the stock images to deploy the bastion node. We need to find the given variable rhel_image value in catalog images and if not found look at boot images.

This is also helpful for the infra/pvsadm automation which uses the 1_prepare module.

Terraform crash when resource does not have a public network

The terraform module will crash with index out of bound error as mentioned in IBM-Cloud/terraform-provider-ibm#1701

TF provider assume there will always be 1 and only 1 public network within a resource instance.

As observed on UI the public network is created automatically when a VM with public network is created. Also once the allocation pool is completely utilized it will create a new public network with different subnet. Still, TF provider will return the first indexed network.

Explore possibility to use region details in OCP node names

Having PowerVS region identifier in OCP nodes will make it easier when trying to find out the OCP nodes in PowerVS UI

Add instructions on release branches

Wildcard DNS search in resolv.conf prevents resolving service IPs

I provisioned a cluster using cluster_name = "nip.io", as suggested by the var.tfvars comment.

When attempting to deploy an operator using an internal registry we ran into the following DNS issue (credit to @dberg1 for writing this up):

The multicluster-operators-standalone-subscription pod is trying to resolve IP address for hostname multiclusterhub-repo.open-cluster-management.svc.cluster.local.
Here is the resolv.conf of the pod:

# oc rsh -n open-cluster-management multicluster-operators-standalone-subscription-7d7986f9f6-rnpvj
sh-4.4$ cat /etc/resolv.conf
search open-cluster-management.svc.cluster.local svc.cluster.local cluster.local ocp-dal12-hub.169.59.253.69.nip.io 169.59.253.69.nip.io
nameserver 172.30.0.10
options ndots:5

Since the hostname the pod is trying to resolve has less than 5 dots, it will not do a direct lookup first.
It will try all the suffixes listed in the search list. Since 169.59.253.69.nip.io is in the search list, it will always get the IP address of the bastion node (because nip.io is a wildcard DNS that will reply for any hostname under 169.59.253.69.nip.io). 169.59.253.69.nip.io should not be put in the search list.

NFS storage mount restrictive permissions inhibit nfs-storage-provisioner

I provisioned from the release-4.7 branch (at c98d7d6), and once the cluster was up the deployment/image-registry operator reported as degraded:

Cluster operator image-registry has been degraded for 10 minutes. Operator is degraded because Unavailable and cluster upgrades will be unstable.

status:
  conditions:
  - lastTransitionTime: "2021-03-03T13:52:28Z"
    lastUpdateTime: "2021-03-03T13:52:28Z"
    message: Deployment does not have minimum availability.
    reason: MinimumReplicasUnavailable
    status: "False"
    type: Available

The image-registry wasn't able to get a PersistentVolumeClaim:

ocp-dal12-hub.ibm-power-acm.com > oc describe PersistentVolumeClaim/registry-pvc
Name:          registry-pvc
Namespace:     openshift-image-registry
StorageClass:  nfs-storage-provisioner
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: nfs-storage
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       image-registry-5dd46bcfb6-crccn
Events:
  Type     Reason                Age                    From                                                                                      Message
  ----     ------                ----                   ----                                                                                      -------
  Normal   ExternalProvisioning  3m34s (x331 over 83m)  persistentvolume-controller                                                               waiting for a volume to be created, either by external provisioner "nfs-storage" or manually created by system administrator
  Normal   Provisioning          119s (x11 over 83m)    nfs-storage_nfs-client-provisioner-544bb74bd4-9tgxj_40e80696-59a2-417b-88c5-f4a4db4b5512  External provisioner is provisioning volume for claim "openshift-image-registry/registry-pvc"
  Warning  ProvisioningFailed    119s (x11 over 83m)    nfs-storage_nfs-client-provisioner-544bb74bd4-9tgxj_40e80696-59a2-417b-88c5-f4a4db4b5512  failed to provision volume with StorageClass "nfs-storage-provisioner": unable to create directory to provision new pv: mkdir /persistentvolumes/openshift-image-registry-registry-pvc-pvc-de4f6b47-fb49-4bb4-b996-669b8869d988: permission denied

Looks like a permissions issue, so I ssh'd into the bastion:

[root@ocp-dal12-hub-bastion-0 ~]# ls -ld /export/
drwxr-xr-x. 3 root root 4096 Mar  3 12:51 /export
[root@ocp-dal12-hub-bastion-0 ~]# chmod 777 /export/
[root@ocp-dal12-hub-bastion-0 ~]# ls -ld /export/
drwxrwxrwx. 3 root root 4096 Mar  3 12:51 /export/

And with the that PersistenVolume was created and clusteroperator/image-registry became available.

With latest terraform getting deprecated warnings for interpolation syntax

With TF v0.13.4 started getting the interpolation syntax warnings. This will show up during apply and validate commands.

# terraform validate

Warning: Interpolation-only expressions are deprecated

  on variables.tf line 230, in locals:
 230:     private_key_file    = "${var.private_key_file == "" ? "${path.cwd}/data/id_rsa" : "${var.private_key_file}" }"

Terraform 0.11 and earlier required all non-constant expressions to be
provided via interpolation syntax, but this pattern is now deprecated. To
silence this warning, remove the "${ sequence from the start and the }"
sequence from the end of this expression, leaving just the inner expression.

Template interpolation syntax is still used to construct strings from
expressions when the template includes multiple interpolation sequences or a
mixture of literal strings and interpolations. This deprecation applies only
to templates that consist entirely of a single interpolation sequence.

(and 7 more similar warnings elsewhere)

Success! The configuration is valid, but there were some validation warnings as shown above.

Version constraint on providers and Terraform binary

Need to put restrictions on PowerVS automation code for:

• Terraform provider plugin versions.
• Terraform version should be v0.12.X