Generic Workflow to create and Publish Docker images. about cortx-re

[gauravchaudhari02]

@shailesh-vaidya Here is the tested snippet for my suggested workflow

name: Base Docker Image for <component name>
on:
  push:
    branches: [ dev ]
    paths:
      - 'scripts/docker/s3server-devel/Dockerfile' # Mentioning dockerfile path is must here
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
     - uses: actions/checkout@v2
     - name: Build and push ${{ github.event.repository.name }} docker image
       uses: docker/[email protected]
       env:
          TOKEN: ${{ secrets.USER_TOKEN }}
          TAG: 7.7.1908
       with:
         # Path to the Dockerfile (Default is '{path}/Dockerfile')
         dockerfile: scripts/docker/s3server-devel/Dockerfile  # Define `dockerfile` variable only if your dockerfile isn't present in {path} 
         # Build Arguments for docker image build.
         build_args: 'token=${{ env.TOKEN }}'  # Define build arguments required in your dockerfile here
         username: ${{ github.actor }}
         password: ${{ env.TOKEN }}
         registry: docker.pkg.github.com
         repository: ${{ github.repository }}/${{ github.event.repository.name }}
         tags: ${{ env.TAG }}

[justinzw]

@shailesh-vaidya I just joined as a developer advocate on @johnbent 's team. I'm also new to CORTX and not familiar with the cortx-re module.

I'd love to help more and work with you on this. Could you help me understand more about these Docker images and how we can utilize them?

[shailesh-vaidya]

@justinzw Sure. May be we can have call and I will show you the way we are doing product releases currently using Jenkins and Docker.

[gauravchaudhari02]

@shailesh-vaidya, What is the difference between DEFAULT_PASSWORD and GITHUB_TOKEN?

[shailesh-vaidya]

@gauravchaudhari02 As discussed DEFAULT_PASSWORD is used to set password for user in container. Which was required for SSH connections in Jenkins. We will not require it in GitHub actions. I will remove it.

[gauravchaudhari02]

@shailesh-vaidya Would it be good to avoid set ENV step?
Instead use GitHub context variables github.repository, github.event.repository.name replacing new environment variables INPUT_REPOSITORY, IMAGE_NAME and move TAG environment variable to env section of step Build and push Docker images

[shailesh-vaidya]

I have updated workflow based on inputs. Please review.

  steps:
   - uses: actions/checkout@v2
   - name: Set ENV
     run: |
       echo ::set-env name=INPUT_REPOSITORY::$( echo $GITHUB_REPOSITORY | tr '[:upper:]' '[:lower:]')
       echo ::set-env name=IMAGE_NAME::cortx_centos
       echo ::set-env name=TAG::7.7.1908
   - name: Build and push Docker images
     uses: docker/[email protected]
     env:
         DEFAULT_PASSWORD: ${{ secrets.DEFAULT_PASSWORD }}
         REPO_NAME: ${{ github.event.repository.name }}
     with:
       # Path to the Dockerfile (Default is '{path}/Dockerfile')
       dockerfile: docker/cortx_components/centos/Dockerfile 
       # Build Arguments for docker image build. 
       build_args: 'CENTOS_RELEASE=7.7.1908'
       path: docker/cortx_components/centos/
       username: ${{ github.actor }}
       password: ${{ secrets.GITHUB_TOKEN }}
       registry: docker.pkg.github.com
       repository: ${{ env.INPUT_REPOSITORY }}/${{ env.IMAGE_NAME }}
       tags: ${{ env.TAG }}

[shailesh-vaidya]

@shailesh-vaidya Would it be good to avoid set ENV step?
Instead use GitHub context variables github.repository, github.event.repository.name replacing new environment variables INPUT_REPOSITORY, IMAGE_NAME and move TAG environment variable to env section of step Build and push Docker images

@gauravchaudhari02 Can you please test and add snippet from workflow here.

[shailesh-vaidya]

@gauravchaudhari02 As mentioned in #39 we can use ENTRYPOINT in Dockerfile so we will not need TOKEN: ${{ secrets.USER_TOKEN }} as build_arg. We can move logic of downloading third party packages in component wise build scripts. will this help?

[johnbent]

Hey Shailesh,

How can we have docker images if our code has kernel dependencies?

CC @xahmad

[shailesh-vaidya]

@johnbent To build artifacts we need required files, modules etc. in place and available for build scripts in build environment. This can be achieved using Docker with installing all required rpm's inside container. This helps in setting up isolated environments to build multiple components at same time on Docker Host (VM in our case). We can install kernel and kernel devel packages inside container required to build Motr and S3Server rpm's. We have created component specific Dockerfile's and using them to build artifacts in CI workflow

[johnbent]

Thanks @shailesh-vaidya ! So I think I understand. We create dockerfiles with all necessary dependencies inside that allow us to build motr and S3 rpm's.

Can we create dockerfiles with motr and S3 already built inside and use them to run CORTX? Or we can only use them to build but not run?

[shailesh-vaidya]

@johnbent AFAIK Kernel is shared between Docker containers so even through we create Dockerfile's with Motr + S3 installed. We will be able to run only one such container on VM at a time. IO operation of multiple Motr+S3 containers might cause issue. However this need to be validated practically.

[shailesh-vaidya]

@gauravchaudhari02 As you mentioned removed set ENV step .
Added path argument. This is due to directory structure in cortx-re repository. We can use it make workflow independent of repository directory structure. Also we do not need to provide Dockerfile path explicitly and copy of additional files like .repo files scripts can be handled.
Also need to give cortx_centos explicitly as repo name is not same as images as in S3Sever cases. I feel that much customization is ok. Please review.

Below is code snippet.

name: Base Docker Image CI

on:
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2
    - name: Build and push ${{ github.event.repository.name }} docker image
      uses: docker/[email protected]
      env:
          TOKEN: ${{ secrets.TOKEN }}
          TAG: 7.7.1908
      with:
        path: docker/cortx_components/centos/
        build_args: 'CENTOS_RELEASE=7.7.1908'  # Define build arguments required in your dockerfile here
        username: ${{ github.actor }}
        password: ${{ env.TOKEN }}
        registry: docker.pkg.github.com
        repository: ${{ github.repository }}/cortx_centos
        tags: ${{ env.TAG }}

[gauravchaudhari02]

@shailesh-vaidya

Also we do not need to provide Dockerfile path explicitly and copy of additional files like .repo files scripts can be handled.

Do you mean that we will provide path variable instead of giving dockerfile variable?
Are we planning to handle .repo file copy in scripts?

[shailesh-vaidya]

@gauravchaudhari02 - We need to provide .repo files and custom scripts for some of the components. like for CSM during Docker Image creation.

[shailesh-vaidya]

We can finalize below workflow as Generic one to build Docker Images. Let me know if anyone has any issues. If we are good we can add this as part of documentation.

steps:
 - uses: actions/checkout@v2
 - name: Build and push ${{ github.event.repository.name }} docker image
   uses: docker/[email protected]
   env:
       TOKEN: ${{ secrets.TOKEN }}
       TAG: 7.7.1908
   with:
     path: docker/cortx_components/centos/
     build_args: 'CENTOS_RELEASE=7.7.1908'  # Define build arguments required in your dockerfile here
     username: ${{ github.actor }}
     password: ${{ env.TOKEN }}
     registry: docker.pkg.github.com
     repository: ${{ github.repository }}/cortx-centos
     tags: ${{ env.TAG }}

[stale]

This issue/pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue/pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.