ocibuilder / ocibuilder

A tool to build OCI compliant images

ocibuilder - Build Open Container Initiative (OCI) compliant images the declarative way

What is the ocibuilder?

The ocibuilder offers a command line tool called the ocictl to build, push and pull OCI compliant images through declarative specifications, allowing you to pick between Buildah or Docker as the container build tool.

Features

• Specify docker or buildah as a build tool.
• Define multiple builds in single build configuration.
• Ability to templatize build stages.
• Multi-stage build support
• Parameterize build configuration at runtime with environment variable support.
• Supports distroless to produce lean images.
• Supports ansible roles as build stage.
• Supports build contexts like Local Filesystem, Git, S3, Google Cloud Storage, Azure Storage Blob, Aliyun OSS
• All basic features like registry login, pulling and pushing images from/to multiple registries.

Architecture

Install

Binary downloads of the ocictl are available on the Releases page.

You can use the install.sh script to install the latest version of ocictl:

curl https://raw.githubusercontent.com/ocibuilder/ocibuilder/master/install.sh | sh

This requires GOPATH to be set, with bin added to your PATH.

The latest images with Buildah and Docker pre-installed alongside the ocictl is available on our Dockerhub repository.

Read the full installation guide available in our docs.

Getting Started

To learn more about the ocibuilder and how to get started take a look at our quick start guide.

Documentation

View our complete documentation.

The ocibuilder.yaml specification file with all fields available and examples is documented here.

Roadmap

Take a look at our roadmap and features in developement here

Contribute

Please read the CONTRIBUTING.md for contributing guidelines.

License

Apache License Version 2.0, see LICENSE

References

Docker: https://github.com/docker

Buildah Commands: https://github.com/containers/buildah

ocibuilder's Issues

add support for image-diff comparisons

As a user I want to be able to see how my newly built image has changed on a layer by layer level, to prevent the introduction or modification of dependencies.

Integrate the container-diff tool by google into the ocibuilder.

Consider support for UMOCI

Is your feature request related to a problem? Please describe.
The feature is not related to an issue.

Describe the solution you'd like
evaluate adding support for umoci

API documentation generator

Is your feature request related to a problem? Please describe.
https://github.com/ahmetb/gen-crd-api-reference-docs

e.g https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.13/

buildah on error exit process

Describe the bug
Currently when buildah fails and execution, the error is logged with pty but the ocictl process is not exited.

Expected behavior
The process should execute on an internal buildah error

This fix is necessary for successfully completing the e2e testing suite.

go-fmt git commit hook

Add a git commit hook which runs gofmt when commiting anything.

We should run it as part of any local commit hook (husky?)

Originally posted by @michael-bowen-sc in #39

rename struct values to increase readability

This issues covers the renaming of method struct names to something more detailed.

Yeah I'm inclined to agree, I think a cleanup is in order for our struct vars to be renamed to something more readable. I'll create an issue.

Originally posted by @artbegolli in #84

default registry if no registry specified

If no registry is defined in login or push, we want to default to dockerhub registry and automatically input the registry registry.hub.docker.com

Milestone v0.1.1 Goals

• Implement a unified client for docker and buildah #58
• Support multiple build contexts #27
• Implement build spec initializer #84 #85
• Fix bugs #78 #82

replacing dep with go-modules

Go 1.13 supports the use of go modules for dependency management.

Migrate from dep and begin to use go-modules instead

adding of a quick-start guide

We want to add a quick-start guide for ocibuilder - giving them getting started instructions on how to install the tool and linking to documentation on how to begin using the tool.

Milestone v0.1.2 Goals

• Implement Operator #30 #96
• Documentation #93
• Cache flag for ocibuilder #83
• Support for image labelling #101
• Image Diff analysis and implementation #12
• Housekeeping #79
• Remote Overlays #106
• define grafeas API #91

buildah unprivileged support

Describe the bug
Attempting to build with Buildah IN A CONTAINER leads to two issues:

1. If the container has not been run as privileged, executing any commands with the builder set as buildah will throw a Operation not permitted error
2. Executing RUN commands in a Dockerfile, even in privileged mode, leads to the following error process exited with error: fork/exec /bin/sh: no such file or directorysubprocess exited with status 1

These errors DO NOT cause ocictl to fall over and exit, instead subsequent steps are attempted to be executed as well

Expected behavior
Buildah is able to fully build images, if an error is recognised the ocibuilder exits and returns a useful error message.

static site generation for docs

Add static site documentation for the ocibuilder/docs repository. Choices can be Hugo, Jekyll, Statik

passing logger to common utils

Current Behaviour:

logrus is used for logging for docker and buildah components - allowing the user to switch on debug mode with a command line flag.

Enhanced Behaviour:

We want to be able to pass this logger to our common utils such that a user can see debug logs in common utils

Support lifecycle hooks

Is your feature request related to a problem? Please describe.
At each phase in the lifecycle of a spec, the framework needs to have a mechanism to publish notifications to services like Slack, Mattermost, PagerDuty, Email etc.

Describe the solution you'd like
Create an interface called Publish(event *Event). Slack, PagerDuty etc will be specific implementations of Publish

Investigate buildpack for image generation

Details forthcoming

https://buildpacks.io/

ansible roles end-to-end installation workflow via ansible templates

Describe the solution you'd like
Implement end-to-end installation for ansible roles

Current State of Ansible roles installation

label support for images

Is your feature request related to a problem? Please describe.
I want to be able to label my images when they are built

Describe the solution you'd like
Allow defining labels in ocibuilder.yaml

Describe alternatives you've considered
N/A

Additional context
N/A

validate path correctness for params

Current a param can have a dest field which can be incorrect. If nothing is being replaced, the builder should error out and make the user aware of this.

buildah 'client-like' interface

Details forthcoming
#65

concurrent running build steps

Update the build step logic to be able to handle the running of multiple build steps concurrently.

Build API for Grafeas integration

Is your feature request related to a problem? Please describe.
The builder needs to have a clean way of storing metadata generated at the build time.

Describe the solution you'd like
Use Grafeas to store the metadata and also expose functionality to user to retrieve the metadata

move common utilities into libraries within pkg

logging for incorrect context

Describe the bug
For context is incorrect, the error message returned is unhelpful

To Reproduce
Steps to reproduce the behavior:

1. Specify invalid contextpath in spec.yaml
2. Run a build
3. See error

Expected behavior
An error specific to the context that has been used. Additional debug logging which displays the context.

pass to spec file, specifying context

Discuss whether when you pass a path to a spec file you

1. want the current directory to remain the expected context

OR

2. by passing a path to a spec file, you want the default context to live in the same place as the spec file.

support for image metadata

Is your feature request related to a problem? Please describe.
Initial support for image metadata. This is to help facilitate proper execution of buildah cmds using exec. Exec commands using pty aren't executed until io.ReadClose is read and outputted.

This is problematic as it makes it impossible to cleanup generated Dockerfiles, as it falls on the client to do this with no knowledge of the generated filenames.

First part is to create and ImageMeta type, which is able to store any built image metadata going forward. For the time being, this type will store the generated Dockerfile path.

Additionally, a CleanUp method is to be created under the Builder interface which will trigger a cleanup of all generated files.

adding docker daemon switch to spec

At the minute you can specify you builder using the --builder command line flag.

We want to also give the option to specify the builder in your spec.yaml file with the daemon boolean flag.

If the flag is set to true, docker will be used; otherwise buildah will be used.

install.sh install script

Is your feature request related to a problem? Please describe.
As a user I want to be able to install ocibuilder very easily.

Describe the solution you'd like
Create a homebrew package to install ocibuilder

Additional context
Add any other context or screenshots about the feature request here.

add cache flag to ocibuilder.yaml

Is your feature request related to a problem? Please describe.
We want to be able to specify whether or not the build being run will use the cache or not, in docker we have access to the --no-cache flag.

Describe the solution you'd like
Update both the ocibuilder.yaml specification api in types.go adding the no-cache flag to the build step struct.
Also we want to be able to override whatever is specified in the build spec using our own command line flag. This is a change to the ocictl build, introducing a new command line arg.

We also need to update the Docker and Buildah client implementations to support the new build flag.

bugfix(): fixing OpenAPI rule violations

At present, we get following OpenAPI rule violations:

API rule violation: list_type_missing,github.com/ocibuilder/ocibuilder/pkg/apis/ocibuilder/v1alpha1,BuildSpec,Steps
API rule violation: list_type_missing,github.com/ocibuilder/ocibuilder/pkg/apis/ocibuilder/v1alpha1,BuildSpec,Templates
API rule violation: list_type_missing,github.com/ocibuilder/ocibuilder/pkg/apis/ocibuilder/v1alpha1,BuildStep,Stages
API rule violation: list_type_missing,github.com/ocibuilder/ocibuilder/pkg/apis/ocibuilder/v1alpha1,BuildTemplate,Cmd
API rule violation: list_type_missing,github.com/ocibuilder/ocibuilder/pkg/apis/ocibuilder/v1alpha1,Command,Flags
API rule violation: list_type_missing,github.com/ocibuilder/ocibuilder/pkg/apis/ocibuilder/v1alpha1,Command,Value
API rule violation: list_type_missing,github.com/ocibuilder/ocibuilder/pkg/apis/ocibuilder/v1alpha1,DockerStep,Inline
API rule violation: list_type_missing,github.com/ocibuilder/ocibuilder/pkg/apis/ocibuilder/v1alpha1,OCIBuilderList,Items
API rule violation: list_type_missing,github.com/ocibuilder/ocibuilder/pkg/apis/ocibuilder/v1alpha1,OCIBuilderSpec,Login
API rule violation: list_type_missing,github.com/ocibuilder/ocibuilder/pkg/apis/ocibuilder/v1alpha1,OCIBuilderSpec,Params
API rule violation: list_type_missing,github.com/ocibuilder/ocibuilder/pkg/apis/ocibuilder/v1alpha1,OCIBuilderSpec,Push
API rule violation: list_type_missing,github.com/ocibuilder/ocibuilder/pkg/apis/ocibuilder/v1alpha1,Stage,Cmd
API rule violation: names_match,github.com/ocibuilder/ocibuilder/pkg/apis/ocibuilder/v1alpha1,BuildStep,Context
API rule violation: names_match,github.com/ocibuilder/ocibuilder/pkg/apis/ocibuilder/v1alpha1,K8sCreds,Password
API rule violation: names_match,github.com/ocibuilder/ocibuilder/pkg/apis/ocibuilder/v1alpha1,K8sCreds,Username
API rule violation: names_match,github.com/ocibuilder/ocibuilder/pkg/apis/ocibuilder/v1alpha1,Param,ValueFromEnvVariable

Update documentation

This issue is to track documentation changes with new features and enhancements:

• Support context #40

Support different build contexts

Is your feature request related to a problem? Please describe.
To build an image, you need the context/filesystem to work with. The CLI only supports local context meaning local filesystem which is not sufficient as there may be scenarios where the context is not available locally.

Describe the solution you'd like
Support context stored on -

• Local filesystem
• S3
• Git
• Azure Blob Storage
• GCP PubSub
• Alibaba Cloud OSS

Describe alternatives you've considered
NA

Additional context
NA

ImagePush function | capture name of the built image name, tag and push the resultant image according

Describe the bug

At present, users need to provide the image name as a combination of registry, user and image-name in ocibuilder.yaml. Ideally users should provide just the image-name and ImagePush() function from pkg/docker.go should take care of tagging the image appropriately and pushing it to the registry/registries mentioned in push specs of ocibuilder.yaml file

Current Spec

build:
  templates:
    - name: template-1
      cmd:
        - docker:
            inline:
              - RUN mkdir /dirname
              - WORKDIR /dirname
              - COPY * /dirname/
              - EXPOSE 8888
  steps:
    - metadata:
        name: <registry>/<namespace>/<image-name>
        labels:
          type: build-1
          overlay: overlay-1
      stages:
        - metadata:
            name: build-env
            labels:
              stage: stage-1
              type: build-from-base
          base:
            image: python
            platform: alpine
          template: template-1
      tag: v0.0.1
      distroless: false
      cache: false
      purge: false

Expected Spec

build:
  templates:
    - name: template-1
      cmd:
        - docker:
            inline:
              - RUN mkdir /dirname
              - WORKDIR /dirname
              - COPY * /dirname/
              - EXPOSE 8888
  steps:
    - metadata:
        name: <image-name>
        labels:
          type: build-1
          overlay: overlay-1
      stages:
        - metadata:
            name: build-env
            labels:
              stage: stage-1
              type: build-from-base
          base:
            image: python
            platform: alpine
          template: template-1
      tag: v0.0.1
      distroless: false
      cache: false
      purge: false

buildah auth-string in push and pull

Describe the Enhancement
Auth with push and pull is not currently consistent with that in docker. Update buildah to include auth within the push command itself.

Expected behavior
Push using buildah should push the image successfully with authentication.

ansible role parsing issue

Describe the bug
When you specify ansible role(s) under build template, we get ansible template parsing error.

Expected Syntax for specifying ansible specs in Build stage:

- ansible:
     local:
       # optional
        requirements: path-to-requirements.yaml
        playbook: path-to-playbook.yaml
     galaxy:
        name: name-of-the_role

Expected behavior
should parse ansible roles correctly

Screenshot

Generated Dockerfile:

Step 5/13 : EXPOSE 8888ENV PLAYBOOK_DIR /etc/ansible/
ERRO[2019-11-07 20:02:55] failed to get JSON stream                     error="No port specified: /etc/ansible/<empty>"
Error: No port specified: /etc/ansible/<empty>
ERRO[2019-11-07 20:02:55] error in executing ocictl root command...     error="No port specified: /etc/ansible/<empty>"

Rename spec.yaml to alternative

Is your feature request related to a problem? Please describe.
We want to rename the current spec.yaml to something that is specific to ocibuilder and easily identifiable and searchable

Describe the solution you'd like
Come up with a unique alternative

create docker/buildah/go base image

Create a base image with docker, buildah and go installed such that ocibuilder can be built on top of.

bugfix(): fix ImageBuild function in docker_test.go

The return value of ImageBuild function in docker_test.go throws error.

cannot use composite literal (type testClient) as type client.APIClient in field value:
	testClient does not implement client.APIClient (wrong type for ImageBuild method)
		have ImageBuild("context".Context, io.Reader, "github.com/ocibuilder/ocibuilder/vendor/github.com/docker/docker/api/types".ImageBuildOptions) (io.ReadCloser, error)
		want ImageBuild("context".Context, io.Reader, "github.com/ocibuilder/ocibuilder/vendor/github.com/docker/docker/api/types".ImageBuildOptions) ("github.com/ocibuilder/ocibuilder/vendor/github.com/docker/docker/api/types".ImageBuildResponse, error)

create a prototype repository which use `ocibuilder` to build and push the resultant image to image registry(s)

Describe the solution you'd like
Create a sample repository which uses ocibuilder

Additional context
Idea is to create a sample repository which uses ocibuilder to build and push the resultant image to image registry(s).

e2e testing for the ocibuilder

Add and end to end testing harness to the ocibuilder to be run automatically as part of the pipeline.
Alternatively - investigate the use of Ginkgo as documented here

enhance linter to check to run errors on gofmt

Is your feature request related to a problem? Please describe.
Enhance linter to include errors with standard gofmt and goimports

Describe the solution you'd like
Update the plugins to golangci-lint and fix outstanding issues

Implement K8s operator

Is your feature request related to a problem? Please describe.
One of the goals of this framework is that it must be K8s native. In order to make it K8s native, we need to implement an operator/controller

Describe the solution you'd like
Implement K8s operator

Describe alternatives you've considered
NA

Additional context
NA

pkg/apis/ocibuilder/v1alpha1/zz_generated.deepcopy.go:295:21: in.Body.DeepCopyReadCloser undefined (type io.ReadCloser has no field or method DeepCopyReadCloser)

When we run make test, we see an error

# github.com/ocibuilder/ocibuilder/pkg/apis/ocibuilder/v1alpha1
pkg/apis/ocibuilder/v1alpha1/zz_generated.deepcopy.go:295:21: in.Body.DeepCopyReadCloser undefined (type io.ReadCloser has no field or method DeepCopyReadCloser)
FAIL	github.com/ocibuilder/ocibuilder/common [build failed]

Running hack/update-codegen.sh creates a deepcopy:

// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *ImageBuildResponse) DeepCopyInto(out *ImageBuildResponse) {
	*out = *in
	if in.Body != nil {
		out.Body = in.Body.DeepCopyReadCloser()
	}
	return
}

and in.Body.DeepCopyReadCloser() does not exist.

pull request slack integration

Describe the solution you'd like

Repo integration with slack bot to notify the chat on a new PR.

enhancing CI/CD and the build pipeline

We want to update the build pipeline to include the following:

• dependency vulnerability check
• licensing check
• display code coverage
• ocictl to build and push image - this includes adding a spec.yaml file or generating one in the pipeline
• long term dependency caching - we shouldn't have to run dep every time a pipeline is triggered

init spec from dockerfile

Is your feature request related to a problem? Please describe.
As a user, I want to be able to easy initialise an ocibuilder.yaml specification file from a Dockerfile, which I can then add to and modify

Describe the solution you'd like
As a user, I would like to be able to run ocictl init --from-docker <PATH-TO-DOCKERFILE> which will generate my ocibuilder specification, taking advantage of the templating capabilities of ocibuilder.

Describe alternatives you've considered
N/A

Additional context
Will enable easier conversion from using standard Docker to ocibuilder instead

release automation

Is your feature request related to a problem? Please describe.
We want to build and release ocibuilder using ocibuilder as part of the circleci pipeline.

Describe the solution you'd like
When a tag is dropped on the repository, it should trigger a release of the binaries and of the container image

failing mid-execution not removing generated files

Describe the bug
When I run a Docker build which fails mid way through - my autogenerated Dockerfile is not being deleted

To Reproduce
Steps to reproduce the behavior:

1. Execute an ocictl build which fails during a command
2. Check whether the generated Dockerfile is present

Expected behavior
Generated files should ALWAYS be cleaned up if ocictl is successful or whether it fails.

remove docs from ocibuilder repo

support for different contexts

Allow using git context from a remote repository as the build context.