Coder Social home page Coder Social logo

eth-client-docker-image-builder's Introduction

eth-client-docker-image-builder

Automates docker builds for ethereum clients. The build process is scheduled every hour to check source repositories for new commits.

Build image on demand

Run the Build client workflow;

Run the Build tooling workflow;

Adding a new image to build on schedule

Add a new image to config.yaml file and it will be built on schedule from this workflow.

- source:
    repository: sigp/lighthouse # source repository to build from
    ref: stable # source repository branch/tag/commit to build from
  build_script: ./teku/build.sh # optional build script to run INSTEAD of the docker build & push (see below)
  target:
    tag: stable # tag to add to the docker image tag, this must be unique for each docker hub repository
    repository: ethpandaops/lighthouse # dockerhub target to deploy the built image
    dockerfile: ./lighthouse/Dockerfile # optional docker file to use, defaults to the source repository's Dockerfile

Output image tags

Take the following config;

- source:
    repository: sigp/lighthouse
    ref: stable
  target:
    tag: banana
    repository: ethpandaops/lighthouse

This would produce the following docker image tags;

# the tag by itself to have the latest build
ethpandaops/lighthouse:banana
# the tag and the source repository's commit hash
ethpandaops/lighthouse:banana-abcd123

How does the build_script work?

The build_script is a bash script that is run INSTEAD of the docker build & push. This is useful for clients that have a custom build process.

When the build_script is set, you must build and push the docker image yourself! Docker will already be logged in to the target repository. You should try to use the target_tag and target_repository environment variables to tag your image.

The following environment variables are available to the build_script;

  • source_repository - source repository to build from
  • source_ref - source repository branch/tag/commit to build from
  • target_tag - tag to add to the docker image tag
  • target_repository - dockerhub target to deploy the built image
  • target_dockerfile - optional docker file to use, defaults to the source repository's Dockerfile
  • source_git_commit_hash - the source repository's commit hash

Example build_script file;

#!/bin/bash

# helper to get source directory
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
cd ${SCRIPT_DIR}/../source

# do something here that requires this custom build script
# ...

# finally build with the tags from the dockerfile
docker build -t "${target_repository}:${target_tag}" -t "${target_repository}:${target_tag}-${source_git_commit_hash}" -f "../${target_dockerfile}" .

# push the image tags
docker push "${target_repository}:${target_tag}"
docker push "${target_repository}:${target_tag}-${source_git_commit_hash}"

Additional Configuration Files

Our image building process utilizes two additional configuration files: platforms.yaml and runners.yaml. These files help in determining the platforms for which docker images should be built and specifying the runners to use for those platforms, respectively.

This configuration determines the platforms for which each client will have a Docker image built.

Sample Content:

besu:
  - linux/amd64
lighthouse:
  - linux/amd64
  - linux/arm64

In the example above, the client 'besu' and 'lighthouse' are both configured to have Docker images built for the linux/amd64 platform. While 'lighthouse' is also configured to have Docker images built for the linux/arm64 platform.

This configuration maps platforms to GitHub Action runners. It tells our workflow which runner should be used when building a Docker image for a specific platform.

Sample Content:

linux/amd64: ubuntu-latest
linux/arm64: self-hosted

In this example, the platform linux/amd64 will use the ubuntu-latest runner, while darwin/arm64 will use the self-hosted runner.

Lint locally

Requirements;

# make sure yamale is installed
pip install yamale

# yamale lint
yamale -s schema.yaml config.yaml

# check unique target tag, should return []
yq 'group_by(.target.repository + ":" + .target.tag) | map(select(length>1))' config.yaml

eth-client-docker-image-builder's People

Contributors

barnabasbusa avatar parithosh avatar pk910 avatar prestonvanloon avatar samcm avatar savid avatar skylenet avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

eth-client-docker-image-builder's Issues

Rename nimbus to nimbus-eth2

This is to ensure we are staying consistent with their image names, so we can easily use it interchangeably in the future.

Latest lodestar:unstable images broken on kurtosis

Working:chainsafe/lodestar:sha256:22d93f375fa2d942f80ee433fca7941e148b2e4d5af5c75caaa859db1c6d74a9
Not working: ethpandaops/lodestar:sha256:b7d9483274157b79a496ff099832c719440b6e9fc2e32d6175057f0557a0b0d8

image tag: ethpandaops/lodestar:unstable-34d8955

Container just hangs, doesn't return anything for /node/identity api

There was an error executing Starlark code 
An error occurred executing instruction (number 72) at github.com/kurtosis-tech/eth-network-package/src/cl/lodestar/lodestar_launcher.star[120:35]:
  add_service(name="cl-2-lodestar-geth", config=ServiceConfig(image="ethpandaops/lodestar:unstable-97d0e46", ports={"http": PortSpec(number=4000, transport_protocol="TCP", application_protocol=""), "metrics": PortSpec(number=8008, transport_protocol="TCP", application_protocol=""), "tcp-discovery": PortSpec(number=9000, transport_protocol="TCP", application_protocol=""), "udp-discovery": PortSpec(number=9000, transport_protocol="UDP", application_protocol="")}, files={"/genesis": "cl-genesis-data"}, cmd=["beacon", "--logLevel=info", "--port=9000", "--discoveryPort=9000", "--dataDir=/consensus-data", "--paramsFile=/genesis/output/config.yaml", "--genesisStateFile=/genesis/output/genesis.ssz", "--eth1.depositContractDeployBlock=0", "--network.connectToDiscv5Bootnodes=true", "--discv5=true", "--eth1=true", "--eth1.providerUrls=http://{{kurtosis:0c4e39af5cc0480ea0dfcf563465d53e:ip_address.runtime_value}}:8545", "--execution.urls=http://{{kurtosis:bb7399b345d6499594144b1410ca3345:ip_address.runtime_value}}:8561", "--rest=true", "--rest.address=0.0.0.0", "--rest.namespace=*", "--rest.port=4000", "--nat=true", "--enr.ip=KURTOSIS_IP_ADDR_PLACEHOLDER", "--enr.tcp=9000", "--enr.udp=9000", "--subscribeAllSubnets=true", "--jwt-secret=/genesis/output/jwtsecret", "--metrics", "--metrics.address=0.0.0.0", "--metrics.port=8008", "--bootnodes={{kurtosis:4401ec01642c4dfa8001b25c9571fa37:extract.enr.runtime_value}}"], private_ip_address_placeholder="KURTOSIS_IP_ADDR_PLACEHOLDER", max_cpu=1000, min_cpu=50, max_memory=1024, min_memory=256, ready_conditions=ReadyCondition(recipe=GetHttpRequestRecipe(port_id="http", endpoint="/eth/v1/node/health"), field="code", assertion="IN", target_value=[200, 206], timeout="15m")))
  Caused by: Unexpected error occurred starting service 'cl-2-lodestar-geth'
  Caused by: An error occurred waiting for all TCP and UDP ports to be open for service 'cl-2-lodestar-geth' with private IP '172.16.12.13'; this is usually due to a misconfiguration in the service itself, so here are the logs:
  == SERVICE 'cl-2-lodestar-geth' LOGS ===================================
  Aug-11 15:18:35.967[]                 warn: --nat flag is set with no purpose
  Aug-11 15:18:36.102[]                 info: Lodestar network=testnet, version=v1.10.0, commit=
  Aug-11 15:18:36.164[]                 info: Connected to LevelDB database path=/consensus-data/chain-db
  Aug-11 15:18:39.922[]                 info: Initializing beacon from a valid checkpoint state slot=0, epoch=0, stateRoot=0x6ae190352e3327f9c79c0f60efd15ff06fee5c8636f82c603e9e988a12e3f89f, isWithinWeakSubjectivityPeriod=true
  
  == FINISHED SERVICE 'cl-2-lodestar-geth' LOGS ===================================
  Caused by: An error occurred while waiting for all TCP and UDP ports to be open
  Caused by: Unsuccessful ports check for IP '172.16.12.13' and port spec '{number:9000 transportProtocol:0 applicationProtocol:<nil> wait:0xc00059e1c8}', even after '240' retries with '500' milliseconds in between retries. Timeout '2m0s' has been reached
  Caused by: An error occurred while calling network address '172.16.12.13:9000' with port protocol 'TCP' and using time out '200ms'
  Caused by: dial tcp 172.16.12.13:9000: i/o timeout

Lodestar image is missing commit hash

As noted by @marioevz in ChainSafe/lodestar#6276 (Additional context) it looks like the Lodestar image unstable-linux-amd64 does not include the commit hash in version info. This makes it difficult to determine the exact version (code) that is running based on the logs.

Based on the workflows of this repo it looks like it uses the Dockerfile from the Lodestar repo to build the image but does not pass the COMMIT build arg which is required to write git data.

For reference, this is how we pass the commit hash in our build step

--build-arg COMMIT=$(git rev-parse HEAD)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.