Coder Social home page Coder Social logo

dapper's Introduction

Dapper - Docker Build Wrapper

Dapper is a tool to wrap any existing build tool in an consistent environment. This allows people to build your software from source or modify it without worrying about setting up a build environment. The approach is very simple and taken from a common pattern that has adopted by many open source projects. Create a file called Dockerfile.dapper in the root of your repository. Dapper will build that Dockerfile and then execute a container based off of the resulting image. Dapper will also copy in source files and copy out resulting artifacts or will use bind mounting if you choose.

Installation

curl -sL https://releases.rancher.com/dapper/latest/dapper-$(uname -s)-$(uname -m) > /usr/local/bin/dapper
chmod +x /usr/local/bin/dapper

From source

go get github.com/rancher/dapper

Example

Dapper is built using dapper so the following is a decent example

go get github.com/rancher/dapper
git clone https://github.com/rancher/dapper.git
cd dapper
dapper

This is the Dockerfile.dapper used

FROM golang:1.15
RUN go get github.com/tools/godep
ENV DAPPER_SOURCE /go/src/github.com/rancher/dapper
ENV DAPPER_OUTPUT bin
WORKDIR ${DAPPER_SOURCE}
ENTRYPOINT ["./script/build"]

Using

Dockerfile.dapper

The Dockerfile.dapper is intended to create a build environment but not really build your code. For example if you need build tools such as make or bundler or language environments for Ruby, Python, Java, etc.

The ENTRYPOINT, CMD, and WORKDIR defined in Dockerfile.dapper are what are used to initiate your build. When running dapper foo bar, foo bar will be passed as the docker CMD. For example, running dapper make install will do the basic equivalent of docker run -it --rm build-image make install. If you want you can set the ENTRYPOINT to make and then dapper install will be the same as make install. Either approach is fine.

You can also customize your build container image with build arguments (via ARG Dockerfile instructions), which are populated from environment variables on dapper image build. That is useful if you want to parameterize your build for different platforms and you're using essentially the same build environment, only on different platforms. For example, if you have ARG ARCH in Dockerfile.dapper, you can have ARCH=arm in your environment variables, and when you run dapper -s your dapper image is built with --build-arg ARCH=arm and $ARCH is effectively replaced with arm in the resulting dapper image.

Dapper Modes: Bind mount or CP

Dapper runs in two modes bind or cp, meaning bind mount in the source or cp in the source. Depending on your environment one or the other could be preferred. If your host is Linux bind mounting is typically preferred because it is very fast. If you are running on Mac, Windows, or with a remote Docker daemon, CP is usually your only option. You can force a specific mode with

dapper --mode|m MODE

For example dapper -m cp or dapper -m bind.

Interactive Shell

If you just want a shell in the build environment run dapper -s.

Configuring

Configuring the behavior of Dapper is done through ENV variables in the Dockerfile.dapper.

DAPPER_SOURCE

DAPPER_SOURCE is the location in the container of where your source should be. For go applications this might look like ENV DAPPER_SOURCE /go/src/github.com/rancher/dapper

In bind mode DAPPER_SOURCE is used in the Docker run command as follows

docker run -v .:${DAPPER_SOURCE} build-image

In CP mode DAPPER_SOURCE is used in a Docker cp command as follows

docker cp . build-container:${DAPPER_SOURCE}

The default value of DAPPER_SOURCE is /source.

DAPPER_CP

DAPPER_CP is the location in the host that should be copied to DAPPER_SOURCE in the container.

In bind mode DAPPER_CP is used in the Docker run command as follows

docker run -v ${DAPPER_CP}:/source/ build-image

In CP mode DAPPER_CP is used in a Docker cp command as follows

docker cp ${DAPPER_CP} build-container:/source/

The default value of DAPPER_CP is ..

DAPPER_OUTPUT

DAPPER_OUTPUT is used after the build is done to copy the build artifacts back to the host. The setting is only used in CP mode. After the build is done equivalent Docker cp command is ran

docker cp ${DAPPER_SOURCE}/${DAPPER_OUTPUT} .

If you don't want the DAPPER_OUTPUT to be relative to the DAPPER_SOURCE then set DAPPER_OUTPUT to a strings that starts with /.

DAPPER_DOCKER_SOCKET

Setting DAPPER_DOCKER_SOCKET will cause the Docker socket to be bind mounted into your build. This is so that your build can use Docker without requiring Docker-in-Docker. The equivalent parameter will be added to Docker.

docker run -v /var/run/docker.sock:/var/run/docker.sock build-image

DAPPER_RUN_ARGS

DAPPER_RUN_ARGS is used to add any parameters to the Docker run command for the build container. For example you may want to set --privileged if you need to do advanced operations as root.

DAPPER_ENV

DAPPER_ENV is a list of ENV variables that should be copied for the host context. Setting DAPPER_ENV=A B C is the equivalent of adding to the Docker run command the following

docker run -e A -e B -e C build-image

License

Copyright (c) 2015-2018 Rancher Labs, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

dapper's People

Contributors

andrelucas avatar brandond avatar cloudnautique avatar dramich avatar drpebcak avatar ibuildthecloud avatar imikushin avatar joakimr-axis avatar kevinjoiner avatar luthermonson avatar oxr463 avatar patst avatar phillipsj avatar rmoriz avatar rohitsakala avatar rosskirkpat avatar svendowideit avatar vincent99 avatar zambon avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dapper's Issues

Build failed if build directory name contains some char, eg: "@"

tag = re.ReplaceAllLiteralString(tag, "-")

DEBU[0000] Building XXX@2:release-develop using Dockerfile.dapper
DEBU[0000] Created tempfile Dockerfile.dapper003042636
DEBU[0000] Running /bin/docker [build -t XXX@2:release-develop --build-arg DAPPER_HOST_ARCH=amd64 -f Dockerfile.dapper003042636 .]
invalid argument "XXX@2:release-develop" for "-t, --tag" flag: invalid reference format

`make generate` fails due to unset `$HOME` (with podman)

Invoking make generate (or, equivalently, .dapper generate) fails at go generate:

Unmodified tree build log with podman

(make-generate.log)

$ make generate
./.dapper generate
<...>
COMMIT k3s:master
--> c5d9ede7f540
Successfully tagged localhost/k3s:master
c5d9ede7f5401ce8ec923b746df13e3510c573a975a9b650c367e1980a3ce6c5
Running: go generate
go: downloading github.com/sirupsen/logrus v1.9.3
<...>
go: downloading github.com/gofrs/uuid v4.4.0+incompatible
build cache is required, but could not be located: GOCACHE is not defined and neither $XDG_CACHE_HOME nor $HOME are defined
main.go:1: running "go": exit status 1
FATA[0074] exit status 1
make: *** [Makefile:13: generate] Error 1

This presumably happens because Dockerfile.dapper sets HOME=${DAPPER_SOURCE} in the same statement as DAPPER_SOURCE=..., which is then not substituted correctly. The problem is not immediately visible on docker only because docker resets an unset $HOME to /root:

Build log with docker, with a patch to make the problem visible

(make-generate-docker.log)

$ git diff
diff --git a/Dockerfile.dapper b/Dockerfile.dapper
index af99d3d35e..81b5af3184 100644
--- a/Dockerfile.dapper
+++ b/Dockerfile.dapper
@@ -56,7 +56,7 @@ ENV DAPPER_RUN_ARGS="--privileged -v k3s-cache:/go/src/github.com/k3s-io/k3s/.ca
     DAPPER_SOURCE="/go/src/github.com/k3s-io/k3s/" \
     DAPPER_OUTPUT="./bin ./dist ./build/out ./build/static ./pkg/static ./pkg/deploy" \
     DAPPER_DOCKER_SOCKET=true \
-    HOME=${DAPPER_SOURCE} \
+    HOME= \
     CROSS=true \
     STATIC_BUILD=true

diff --git a/scripts/generate b/scripts/generate
index ebdaf31cc5..142592f36f 100755
--- a/scripts/generate
+++ b/scripts/generate
@@ -8,4 +8,6 @@ mkdir -p build/data
 GO=${GO-go}

 echo Running: "${GO}" generate
+env | grep HOME=
+exit 1
 "${GO}" generate

$ make generate
./.dapper generate
<...>
Successfully tagged k3s:master

Running: go generate
HOME=/root
FATA[0002] exit status 1
make: *** [Makefile:13: generate] Error 1

Still, this causes the Go cache (mounted at /go/src/github.com/k3s-io/k3s/.cache) to be effectively broken and unused because $HOME doesn't point to it.

http_proxy issue running dapper behind firewall

Step 8 : RUN apt-get update && apt-get install -y python-pip
---> Running in 42a7d9450794
Get:1 http://httpredir.debian.org jessie InRelease [8127 B]
Get:2 http://httpredir.debian.org jessie-updates InRelease [8127 B]
Get:3 http://security.debian.org jessie/updates InRelease [8127 B]
Splitting up /var/lib/apt/lists/partial/httpredir.debian.org_debian_dists_jessie_InRelease into data and signature failedIgn http://httpredir.debian.org jessie InRelease
E: GPG error: http://httpredir.debian.org jessie InRelease: Clearsigned file isn't valid, got 'NODATA' (does the network require authentication?)
The command '/bin/sh -c apt-get update && apt-get install -y python-pip' returned a non-zero code: 100

Reading Env did not work on v0.4.0

Just take RancherOS as an example: https://github.com/rancher/os

Try v0.3.4:

$ ./.dapper -d -m bind -s
...
DEBU[0001] Reading Env: DAPPER_DOCKER_SOCKET=true
DEBU[0001] Reading Env: DAPPER_SOURCE=/go/src/github.com/rancher/os
...
DEBU[0001] Running shell in os:master
DEBU[0001] Exec /usr/bin/docker run [--rm -i --name os-awbwhED -t -v /var/run/docker.sock:/var/run/docker.sock -v /opt/official/os/.:/go/src/github.com/rancher/os/ -e DAPPER_UID=0 -e DAPPER_GID=0 -e VERSION -e DEV_BUILD -e RUNTEST -e DEBUG -e APTPROXY -e ENGINE_REGISTRY_MIRROR -e KERNEL_CHECK -e APPEND_SYSTEM_IMAGES --entrypoint /bin/bash -e TERM --privileged os:master -]

Try v0.4.0:

$ ./.dapper -d -m bind -s
...
DEBU[0005] Running shell in os:master
DEBU[0005] Exec /usr/bin/docker run [--rm -i --name os-NzKWvTZ -t -v /opt/official/os/.:/source/ -e DAPPER_UID=0 -e DAPPER_GID=0 --entrypoint /bin/bash -e TERM os:master -]

v0.3.4 can work well, but v0.4.0 can not.
It seems that the latest version does not execute readEnv, this break the ROS CI.

https://releases.rancher.com/dapper/latest/dapper-Darwin-x86_64 fails with Segmentation Fault

RancherOS Version: (ros os version)
Tested with 1.1.0 and 1.0.3

Where are you running RancherOS? (docker-machine, AWS, GCE, baremetal, etc.)
The problem is building the ISO, not running it

On the latest recent release of Mac OSX High Sierra v10.13.1, the dapper executable located at

https://releases.rancher.com/dapper/latest/dapper-Darwin-x86_64

fails with a segmentation fault 11.

If you build dapper yourself using go version go1.9.2 darwin/amd64 and modify Makefile to use the local version the build seems to work OK!

So a rebuild of the executable at https://releases.rancher.com/dapper/latest/ would be appreciated!

`dapper --build -- -t xxx` can not work on v0.4.0

Just take RancherOS as an example: https://github.com/rancher/os/blob/master/scripts/build-images#L22

It can work on v0.3.4, but not v0.4.0.

Try v0.4.0:

$ dapper -d --build -f images/00-rootfs/Dockerfile -- -t rancher/os-rootfs images/00-rootfs
DEBU[0000] Building os:master using images/00-rootfs/Dockerfile
DEBU[0000] Created tempfile images/00-rootfs/Dockerfile109593587
DEBU[0000] Running /usr/bin/docker [build -t os:master -f images/00-rootfs/Dockerfile109593587 .]
Sending build context to Docker daemon 35.55 MB
Step 1/2 : FROM scratch
 --->
Step 2/2 : ADD build/rootfs.tar /
ADD failed: stat /var/lib/docker/tmp/docker-builder070439420/build/rootfs.tar: no such file or directory
DEBU[0006] Failed running /usr/bin/docker [build -t os:master -f images/00-rootfs/Dockerfile109593587 .]: exit status 1

Try 0.3.4:

$ dapper -d --build -f images/00-rootfs/Dockerfile -- -t rancher/os-rootfs images/00-rootfs
DEBU[0000] Running /usr/bin/docker [build -f images/00-rootfs/Dockerfile -t rancher/os-rootfs images/00-rootfs]
Sending build context to Docker daemon 47.37 MB
Step 1/2 : FROM scratch
 --->
Step 2/2 : ADD build/rootfs.tar /
 ---> Using cache
 ---> a70d5a7eb118
Successfully built a70d5a7eb118
Successfully tagged rancher/os-rootfs:latest

It was introduced by ca29bae

Support newer Docker API versions

Could we please update the shipped Docker binaries? This tool is unusable on modern Docker releases.

Error response from daemon: client version 1.21 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version

Server:
 Version:      17.03.2-ee-5
 API version:  1.27 (minimum version 1.24)
 Go version:   go1.7.5
 Git commit:   fa09039
 Built:        Wed Jul 19 23:56:45 2017
 OS/Arch:      windows/amd64
 Experimental: false

How to pass `--add-host` command to dapper

when performing docker build, sometimes you need service container running such as mariadb, when doing magento build

e.g.

docker build --add-host mariadb:`docker inspect -f '{{.NetworkSettings.IPAddress}}' mariadb` -t magento_app.

is there a way to pass --add-host to dapper

Build no longer possible due to golint changes

Building the project with dapper ci throws an error:

package github.com/golang/lint/golint: code in directory /go/src/github.com/golang/lint/golint expects import "golang.org/x/lint/golint"
The command '/bin/sh -c wget -O - https://storage.googleapis.com/golang/go1.9.3.linux-${!GOLANG_ARCH}.tar.gz | tar -xzf - -C /usr/local &&     go get github.com/rancher/trash && go get github.com/golang/lint/golint' returned a non-zero code: 1

Related to golang/lint#415

DAPPER_DOCKER_SOCKET failing on docker desktop

Hello,
i try to run dapper with docker-desktop as docker-daemon. I need the docker-daemon inside the container, so i set the DAPPER_DOCKER_SOCKET true in the dapper-file like described. This leads to the following call, which fails:

[INFO]  Building docker container
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
time="2021-06-17T12:28:29+02:00" level=debug msg="Failed running C:\\Program Files\\Docker\\Docker\\resources\\bin\\docker.exe [run -i --name app_test-fCUVeDE -v //./pipe/docker_engine://./pipe/docker_engine -e DAPPER_UID=-1 -e DAPPER_GID=-1 -e 
VERSION --privileged app_test:develop]: exit status 1"
time="2021-06-17T12:28:29+02:00" level=debug msg="Deleting temp container app_test-fCUVeDE"
time="2021-06-17T12:28:30+02:00" level=fatal msg="exit status 1"

The problem seems to be the argument -v //./pipe/docker_engine://./pipe/docker_engine

When i manually perform the docker run with the argument -v /var/run/docker.sock:/var/run/docker.sock it works.

Is this a bug, parsing the runtime-args or am i missing something?
Tried many thing but dit not get it working yet.

thanks for your help!

Feature Request: Option to call docker build with Dapper-Dockerfile contents via stdin instead of using `-f`

Dapper currently calls docker build -f <filename> which has a big downside in large monorepo roots (think of multiple GB in size), because this will send everything as build context to the dockerd each time.

Docker itself only allows one "global" .dockeringore file, this would apply for all Dockerfiles including all dapper-related Dockerfiles. Excluding everything there is not an option:
In our case we have existing Dockerfiles that require everything as build-context, and we have several Dockerfile.dapper.xxx which should have an empty build context (data is bind-mounted later).

I suggest to add a flag to dapper so that docker can be called somehow like this:

docker build --tag $TAG - <<EOF
$(cat $DOCKERFILE)
EOF

How did you get loop mounting to work?

When I ran a build using dapper (e.g. RancherOS), I am able to do loop mounts. Yet those capabilities normally are missing from Docker containers. How does Dapper allow loopback mounts? I see nothing about added capabilities anywhere, so how does it do it?

dapper v0.5.5 breaks rancher/os builds

Steps to reproduce:

git clone https://github.com/rancher/os.git rancher-os
cd rancher-os
run make release

Output:

...snip...
+ tag=rancher/os-rootfs:628e6ab0
+ echo 'build-image: Building rancher/os-rootfs:628e6ab0'
build-image: Building rancher/os-rootfs:628e6ab0
+ '[' -x images/00-rootfs/prebuild.sh ']'
+ images/00-rootfs/prebuild.sh
+ dapper -d --build -f images/00-rootfs/Dockerfile -- -t rancher/os-rootfs images/00-rootfs
DEBU[0000] Building os:master using images/00-rootfs/Dockerfile
FATA[0000] pattern contains path separator
+ '[' 1 '!=' 42 ']'
+ exit 1
FATA[0353] exit status 1
make: *** [release-build] Error 1

It look like the following go commit is responsible: golang/go@12cd55c
I assume this was pulled in when updating the go version for v0.5.5.
Pinning the dapper version to v0.5.4 instead of latest in rancher/os allows it to build.

How to build rpi64 image?

Hi,

How actually build rpi64 image? or arm? Is it broken?
I've tried target rpi64.

Getting:

`
make rpi64
./scripts/release-rpi64
++ dirname ./scripts/release-rpi64

  • cd ./scripts/..
  • source ./scripts/version
    +++ git status --porcelain --untracked-files=no
    ++ '[' -n '' ']'
    +++ git rev-parse --short HEAD
    ++ COMMIT=0dc7a00e
    +++ git tag -l --contains HEAD
    +++ head -n 1
    ++ GIT_TAG=
    ++ '[' -z '' ']'
    ++ [[ -z '' ]]
    ++ [[ -n '' ]]
    ++ VERSION=0dc7a00e
    ++ INITRD=initrd-0dc7a00e
    ++ export VERSION COMMIT INITRD GIT_TAG DIRTY
    ++ export SUFFIX=
    ++ SUFFIX=
    ++ '[' -n '' ']'
    ++ echo ' Building 0dc7a00e from 0dc7a00e on '
    Building 0dc7a00e from 0dc7a00e on
  • '[' '!' -f ./dist/artifacts/rootfs_arm64.tar.gz ']'
  • echo 'Please run make first'
    Please run make first
  • exit 0
    `

If I run just 'make' only x86 artefacts are build.

I could not find rpi64 binaries on the release page: https://github.com/burmilla/os/releases
Which is a target from another doc: https://burmillaos.org/docs/installation/server/raspberry-pi/

docker build fails when runc is not the default runtime

I tried to build an rancher/os-kernel, and it dies with

Sending build context to Docker daemon  5.303MB
Step 1/31 : FROM ubuntu:16.04
 ---> 00fd29ccc6f1
Step 2/31 : RUN apt-get update     && apt-get install -y build-essential wget libncurses5-dev unzip bc curl python rsync ccache git vim libssl-dev kmod gnupg2 locales
 ---> Running in 0ab903400076
shim error: json: cannot unmarshal array into Go struct field processState.capabilities of type specs.LinuxCapabilities
time="2017-12-19T20:18:10+10:00" level=fatal msg="exit status 1" 
sven@t440s ~/go/src/github.com/rancher/os-kernel $ 

Invalid tag when repo/directory contains uppercase characters

Error:

โžœ  ExampleKit git:(rmoriz/core-test-docker-dapper) dapper -f  Dockerfile.core.stable        
invalid argument "ExampleKit:rmoriz-core-test-docker-dapper" for "-t, --tag" flag: invalid reference format: repository name must be lowercase
See 'docker build --help'.
FATA[0000] exit status 125    

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.