Automated contract deployment about flow-cli

[psiemens]

I'm starting to sketch out a new format for flow.json that aims to address the following:

• Better management of accounts (related to #31)
• Automated contract deployment
• Emulator configuration
• Saved network configurations

{
  "emulator": {
    "default": {
      "port": 3569,
      "serviceKey": {
        "privateKey": "",
        "signatureAlgorithm": "ECDSA_P256",
        "hashAlgorithm": "SHA3_256"
      }
    },
    "advanced": {
      "extends": "default",
      "withFeesEnabled": true,
    }
  },
  "networks": {
    "emulator": {
      "host": "127.0.0.1:3569",
      "chain": "emulator"
    },
    "testnet": {
      "host": "access.testnet.nodes.onflow.org:9000",
      "chain": "testnet"
    }
  },
  "contracts": [
    {
      "destination": {
        "emulator": "emulator",
        "testnet": "testnet"
      },
      "source": {
        "KittyItems": "./kitty-items-cadence/cadence/kittyItems/contracts/KittyItems.cdc",
        "KittyItemsMarket": "./kitty-items-cadence/cadence/kittyItemsMarket/contracts/KittyItemsMarket.cdc",
        "Kibble": "./kitty-items-cadence/cadence/kibble/contracts/Kibble.cdc"
      }
    }
  ],
  "accounts": {
    "emulator": {
      "type": "memory",
      "address": "service",
      "keyIndex": 0,
      "signatureAlgorithm": "ECDSA_P256",
      "hashAlgorithm": "SHA3_256",
      "config": {
        "privateKey": ""
      }
    }
  }
}

Here are some examples of how I think this could be used:

flow start-emulator                    # start the emulator with default config profile
flow start-emulator --profile advanced # start the emulator with advanced config profile

flow deploy-contracts emulator         # deploy contracts to emulator 
flow deploy-contracts testnet          # deploy contracts to testnet
flow deploy-contracts emulator --watch # deploy contracts to emulator, watch for updates and redeploy

[janezpodhostnik]

Some contracts are also already deployed:
FlowFees, FlowServiceAccount, FlowStorageFees, FlowToken, FungibleToken
These addresses should be in the json event though they don't need a source.

Another thing I was thinking about is address replacement in scripts. So if I write import FlowFees from 0xFlowFees in a script 0xFlowFees will get replaced by the address of FlowFees which could be defined in the json.

[10thfloor]

This seems like something that cli users / project creators might not even need to look at. I think there are 2 concerns mixed into the idea here: 'project state' and 'project configuration'.

Information about project state, deployment statuses, addresses, accounts and keys ... etc could be saved to a local database that is managed by the cli. It could even be another json file, just hidden from the user and exportable / importable if the user wished to move their project between machines / environments.

If the cli had the notion of a 'project' we could eliminate the need for the flow.json file in the source code, or at least keep it minimal; used primarily as configuration that tells the cli how to connect to networks / emulator, and further configures the emulator with dev flags. (withFeesEnabled ...etc) or any other configuration a project might need, like how to connect to a 'dev wallet', or oracles ...etc.

[sideninja]

I was thinking of how this format could also work with scripts/transactions. There are multiple aspects to this problem:

• option to execute transaction and script files within CLI. This could be achieved very similar to contract deployment, as you would just have to list them under lets say transactions and scripts property, and then address replacement would work the same way
• ability to execute transactions and scripts within the code. After contracts are deployed it would be great if a program written in any language could benefit the work CLI does. I think this could be achieved by having kinda of a "state" json file were we would save locations of deployed contracts so any program could just import that file and benefit the work from CLI deployment - this probably should be discussed in a separate issue (I will open it and link it)

[joshuahannan]

You'll need a little more fine grained control for how you deploy, correct? Doing upgrades, creating new accounts, or deploying to an existing account. And should there be a different json organization for contracts and their accounts on emulator and testnet?
Maybe something resembling this?

  "contracts": [
    {
      "KittyItems": {
        "source": "./kitty-items-cadence/cadence/kittyItems/contracts/KittyItems.cdc",
        "emulator": "0x02949495949494"
        "testnet":  {
             status: "deployed"
             address: "0x285b3f3f393f3j9"
        },
      },
      "KittyItemsMarket": {
        "source": "./kitty-items-cadence/cadence/kittyItemsMarket/contracts/KittyItemsMarket.cdc",
        "emulator": ""
        "testnet": {
           "status": ""
      },
    }
  ],

Forgive me if my json doesn't make any sense, I just feel like there is more metadata about contracts to keep track of

[psiemens]

I think there are 2 concerns mixed in to the idea here: 'project state' and 'project configuration'.

Yep, I think this is at the root of the problem we're solving.

I assumed a few constraints when I was thinking about this:

• flow.json can always be safely committed to a project's repository (similar to package.json)
• Multiple developers of the same project may have different testnet accounts and/or private keys
  • For example, two devs working on Kitty Items may deploy the contracts to separate testnet accounts, rather than all devs sharing the same testnet account.
• If somebody deploys 3 contracts to a single account on the emulator, they will also want to deploy those same 3 contracts to a single account on testnet

I wasn't thinking about this as clearly as you @10thfloor, but I like what you're saying and would propose the following:

• Project configuration is the information required to configure and deploy a new instance of an application on the emulator, testnet or mainnet.
  • Always included in flow.json
  • Always committed to source control
• Project state is information about a specific deployed instance of a project, whether it is a developer's local emulator deployment or a team's shared testnet deployment.
  • Never included in flow.json
  • Never committed to source control

@joshuahannan is right, there isn't enough info in the above JSON to track deployments and accounts. But I think that missing info is state and not configuration.

To put it more concretely, I was experimenting with the idea of a flow.private.json that is always excluded from source control (e.g. in .gitignore) and includes information about a specific instance of a project. Every developer would have their own flow.private.json that is unique to their system.

Here's a modified (and truncated) version of the flow.json example I posted above, along with an accompanying flow.private.json example:

// flow.json

{
  "contracts": [
    {
      "destination": {
        "emulator": "emulator",
        "testnet": "testnet-kitty-items"
      },
      "source": {
        "KittyItems": "./kitty-items-cadence/cadence/kittyItems/contracts/KittyItems.cdc",
        "KittyItemsMarket": "./kitty-items-cadence/cadence/kittyItemsMarket/contracts/KittyItemsMarket.cdc",
        "Kibble": "./kitty-items-cadence/cadence/kibble/contracts/Kibble.cdc"
      }
    }
  ],
  "accounts": {
    "emulator": {
      "type": "memory",
      "address": "service",
      "network": "emulator",
      "keyIndex": 0,
      "signatureAlgorithm": "ECDSA_P256",
      "hashAlgorithm": "SHA3_256",
      "config": {
        "privateKey": ""
      }
    },
    "testnet-kitty-items": {
      "network": "testnet",
      "isPrivate": true
    }
  }
}

// flow.private.json

{
  "accounts": {
    "testnet-kitty-items": {
      "type": "memory",
      "address": "MY_UNIQUE_TESTNET_ADDRESS",
      "keyIndex": 0,
      "signatureAlgorithm": "ECDSA_P256",
      "hashAlgorithm": "SHA3_256",
      "config": {
        "privateKey": "MY_UNIQUE_TESTNET_PRIVATE_KEY"
      }
    }
  }
}

The idea here is that the public flow.json file says "every instance of this project should have a testnet-kitty-items account". The CLI would then parse flow.private.json and prompt the user to create an entry for testnet-kitty-items if it doesn't already exist.

[psiemens]

Some contracts are also already deployed:
FlowFees, FlowServiceAccount, FlowStorageFees, FlowToken, FungibleToken
These addresses should be in the json event though they don't need a source.

Great point. I have some ideas for how to solve this but need to give it some more thought.

Another thing I was thinking about is address replacement in scripts.

I have a general solution for this that's in the works. I'll share here when I have something to show!

[psiemens]

Early proof-of-concept that reads the following JSON format and uses the declared paths to rewrite local imports as on-chain addresses.

{
  "aliases": {
    "FungibleToken": {
      "emulator": "0xee82856bf20e2aa6",
      "testnet": "TESTNET_ADDRESS"
    }
  },
  "contracts": {
    "kitty-items": {
      "target": {
        "emulator": "emulator", 
        "testnet": "testnet" 
      },
      "source": {
        "NonFungibleToken": "./cadence/kittyItems/contracts/NonFungibleToken.cdc",
        "Kibble": "./cadence/kibble/contracts/Kibble.cdc",
        "KittyItems": "./cadence/kittyItems/contracts/KittyItems.cdc",
        "KittyItemsMarket": "./cadence/kittyItemsMarket/contracts/KittyItemsMarket.cdc"
      }
    }
  }
}

[psiemens]

I realized there's an edge case present in the above example. NonFungibleToken isn't available by default on the emulator and therefore needs to be loaded from source, but it is pre-deployed on tesnet. The configuration above assumes that it will also need to be deployed on testnet.

Maybe something like this could fix that?

{
  "contracts": {
    "nft": {
      "target": {
        "emulator": "emulator",
        "testnet": "0xTESTNET_NFT_ADDRESS"
      },
      "source": {
        "NonFungibleToken": "./cadence/kittyItems/contracts/NonFungibleToken.cdc"
      }
    },
    "kitty-items": {
      "target": {
        "emulator": "emulator",
        "testnet": "testnet"
      },
      "source": {
        "Kibble": "./cadence/kibble/contracts/Kibble.cdc",
        "KittyItems": "./cadence/kittyItems/contracts/KittyItems.cdc",
        "KittyItemsMarket": "./cadence/kittyItemsMarket/contracts/KittyItemsMarket.cdc"
      }
    }
}

[psiemens]

@sideninja I was thinking more about future versions of this feature and realized we'll eventually want the ability to deploy contracts with initialization arguments.

pub contract Foo {
  init(bar: String) {
    // ...
  }
}

Given the current schema we've been discussing, do you think it makes sense to allow for something like this?

{
  "deploy": {
    "testnet": {
      "account-1": [
        { 
          "name": "Foo",
          "args": ["Hello, World!"]
        }
      ],
      "account-2": ["Kibble", "KittyItems", "KittyItemsMarket"],
    }
  }
}

[sideninja]

@psiemens I agree this is important to envision. I like the proposed format, this could start to get intuitive as an advanced option for almost any param that you can pass an object instead of a string.