Exponential backoff on CA failures about ztunnel

[hzxuzhonghu]

It is not limited, but looks https://crates.io/crates/backoff is the most popular one

[keithmattix]

/cc @MorrisLaw

[MorrisLaw]

Thank you Keith, I'll look into this!

[MorrisLaw]

Is there a retry library that maintainers recommend @howardjohn?

[MorrisLaw]

PR is ready for review again: #768

[MorrisLaw]

PR has been merged.

[keithmattix]

Had to revert; reopening

[MorrisLaw]

Had to revert due to an issue noticed in an automated Istio PR that tried updating to the latest stable ztunnel master. TestIntermediateCertificateRefresh failed due to missing cert. It looks like it wasn't refreshed in time or it could be a possible race condition.

Related areas of the code to look at as it relates to the integration test that were changed before and are relevant to the backoff PR, these may require modification in order to support this change:

• https://github.com/istio/ztunnel/pull/742/files
• istio/istio@1ee1a43

Goal is to reproduce the integration test failure with the original PR that has been reverted, then figure out how to update the integration test or the business logic itself so that the integration test no longer fails.

[MorrisLaw]

This has been merged back in. Found the issue in an integration test and fixed it. Also, confirmed that istio integration is happy with this latest version of ztunnel: https://github.com/istio/istio/pull/50079/files. We can close this for real now.