[engine] Smart Contract Upgrading Implementation (Milestone 1) about daml

[nickchapman-da]

• Update existing softFetch/softExercise integration tests to match latest required behaviour. i.e. it must work for any types which satisfy the compatibility relation. And specifically, work for downgrades. Fix existing implementation.
• Implement the compatibility relation (i.e code to drop/make-None) within importValue/translateValue, replacing the current code in SBPromoteAnyContract (which is not in the right place)
• Refuse to downgrade optional field values which are not None (test and imp)
• Control upgrading behaviour via engine-startup feature flag
• Add ResultNeedUpgradeVerification to engine/ledger interface (we might yet switch to ResultNeedCreate)
• Ensure changes support Ratko's canton level integration test PR. Specifically, so we can replace the related hack PR in the daml repo.
• Restructure contract caching in the engine
• make required calls to ResultNeedUpgradeVerification
• Add engine level unit tests for upgrading and downgrading.

[nickchapman-da]

interaction with contract-caching

Following the work to move the compatibility transformation into importValue it became clear there is a interaction with contract caching (in the engine) which needs to be resolved.

Contract caching prevents the engine from asking the ledger for the same contract twice. Currently the (LF) Value for the contract (received from the ledger) is first converted to an SValue by importValue before being cached.

importValue requires a type to do it's work. Previously (before #17095) it used the type associated with the contract data. Now, to support upgrades, we use the type coming from the code (fetch/exercise) we are running, and then up/down-grade as necessary. Potentially, it seems that the same contract value could be used (fetch/exercise) at two different types (type versions).

This does not play well with the caching of the SValue resulting from importValue

Remy and I discussed two potential solutions, presented here:
#17095 (comment)

A: cache an (LF) Value. Then convert to SValue on each import
B: cache SValue (like now). Then convert to a new SValue on each import.

We believe that B has too much subtlety and complexity, and scope for bugs.
We agreed that we should attempt A in the first instance.