Cannot upgrade cluster from 5.7.198.9494 to latest about service-fabric-cli

[judy-zz]

I ran pip3 install sfctl==99 to see a list of versions available. I got 1.0.1, 1.1.0, 1.2.0rc2, 2.0.0, and 3.0.0. I tried both 1.2.0rc2 and 2.0.0, in case one of them introduced the upgrade command and still talked to my cluster. No dice. 1.2.0rc2 doesn't have upgrade, and 2.0.0 gives me Bad Request.

[oanapl]

@judy, can you paste the verbose output from your command for sfctl 2.0.0?

Meanwhile, can you use Powershell cmdlets to upgrade the cluster to unblock yourself?

[judy-zz]

Sure. Downgraded to 2.0.0 and ran it:

sfctl cluster upgrade --verbose
Enter PEM pass phrase:
/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
Operation returned an invalid status code 'Bad Request'
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/knack/cli.py", line 125, in invoke
    cmd_result = self.invocation.execute(args)
  File "/usr/local/lib/python3.6/site-packages/knack/invocation.py", line 85, in execute
    cmd_result = parsed_args.func(params)
  File "/usr/local/lib/python3.6/site-packages/knack/commands.py", line 67, in __call__
    return self.handler(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/knack/commands.py", line 123, in _command_handler
    result = op(client, **command_args) if client else op(**command_args)
  File "/usr/local/lib/python3.6/site-packages/sfctl/custom_cluster_upgrade.py", line 126, in upgrade
    client.start_cluster_upgrade(upgrade_desc, timeout=timeout)
  File "/usr/local/lib/python3.6/site-packages/azure/servicefabric/service_fabric_client_ap_is.py", line 1400, in start_cluster_upgrade
    raise models.FabricErrorException(self._deserialize, response)
azure.servicefabric.models.fabric_error.FabricErrorException: Operation returned an invalid status code 'Bad Request'

[belmaiastar]

@judy, sfctl starting from 2.0.0 is for cluster version 6.0 and above so it wouldn't work for pre-6.0 clusters.
I've verified that 1.. sfctl doesn't have cluster upgrade feature, so basically you need to use other APIs to achieve this. e.g. PS like oanapl suggested

[judy-zz]

I was trying to get the cmdlets working yesterday. I think I was hitting the same issue, where the latest version of the SDK refused to talk to the older clusters.

I ended up destroying the clusters and recreating them. Not ideal, and not something we're going to be able to do when we're running in production soon. 😞

[belmaiastar]

@judy , sorry for the inconvenience. The fundamental issue with this is like I said - a specific version of sfctl is supposed to work on a corresponding version of cluster or above nowadays which is our "backward compatibility" guarantee. What sfctl is doing is calling cluster's REST API with a specific "api-version" in the body. That's why your sfctl can not help you upgrading v5.7 cluster.

By "trying to get the cmdlets working" you mentioned, did you mean you tried PowerShell? Is your cluster a secured cluster or no?

We will improve the following aspects:

1. Add a switch in sfctl to allow overriding "api-version" which allows talking to older version of cluster.
2. The cluster should return a verbose error message to tell the request is rejected because of version not supported and which version is the latest it can support.

[judy-zz]

Powershell against a secured cluster. I know it's worked in the past, with the entire connection string to tell it which certificate to use (by thumbprint) and so on. But I've upgraded the SDK since then...

Those improvements would help a lot. We really need to be able to effortlessly talk to older API versions, in order to send the command to upgrade to newer API versions. ;) Thank you!

[samedder]

@judy I don't know the best way to categorise this issue. I think we're going to make improvements but there isn't a current workaround in just the CLI.

I'll close the issue for now and we'll go from what @linggengmsft suggested.