earthly / earthly Goto Github PK

I was running:

/build/linux/amd64/earth -build-arg RELEASE_TAG \
>     --secret GITHUB_TOKEN \
>     --push -P ./release+release

but unfortunately had misspelled the environment variable as GITHUB_TOKE=asdf.

It would be nice if earth had warned me that it couldn't find the env variable.

There is not output when a build requires the -P flag and it is not provided. earth simply exits with non-zero status without outputting anything.

earth should give clear indication as to what the problem is.

I'm using earth to create a Docker image in 2 recipes.
The first stage compiles the source code, the second sets up a run environment and copies over the artifacts.

I boiled it down to this simple example:

FROM alpine

build:
  RUN touch hello.txt
  SAVE ARTIFACT hello.txt

run:
  FROM ubuntu
  COPY +build/hello.txt .
  EXPOSE 8080
  SAVE IMAGE hello-image

Running earth +run fails with the following error:

Error: parser failure: assignment to entry in nil map

If i comment either the EXPOSE 8080 or FROM ubuntu commands, the command works again, so there seems to be an issue when using both commands in a recipe.

When we have something like

SAVE ARTIFACT *.txt AS LOCAL build/

... and say that there are no *.txt files found, the system detects this as an error only after it has deemed the build as success. It should detect the error before that and not declare it as a success.

Ideas:

• Tie artifact independent states to final side effects with fake deps? This may have performance implications.

A new user of Earthly reported the following:

I wrote up the earth.file trying to follow the tutorial and then ran the earth command and did not really get much further. I see in the help

COMMANDS:
  prune    Prune earthly build cache
  help, h  Shows a list of commands or help for one command

But I am not sure that I wanted to run either of those commands.

It should be more obvious how to run the build, from both

1. the tutorial
2. the earth usage text

Currently git auth is passed automatically to the buildkit daemon via some mounts and env vars. This requires meddling with the buildkit daemon in a manner that is difficult to translate to a build-in-the-cloud setup. A better approach is to pass in this auth via the client-daemon connection:

• For username-password, use buildkit's newly supported token-based auth
• For ssh-based auth, buildkit requires a contribution

the progress statuses can take up a lot of output; perhaps we can change to an ncurses-like progress bar (similar to buildkit's), or change the logic to only print out values when they changes (either by 1, or 5 percentage points, or even once a second).

here's the code:
https://github.com/earthly/earthly/blob/master/builder/solver.go#L298

and here's an example of too much output:

./examples/tests+from-expose-test | +base./examples/tests+from-expose-test |  | ./examples/tests+from-expose-test | sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c 44%
./examples/tests+from-expose-test | +base./examples/tests+from-expose-test |  | ./examples/tests+from-expose-test | sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c 47%
./examples/tests+git-clone-test | +base | sha256:d838eb6031fe0bb4f1f203ec26c293afccce4db86abde5d3b1c3a923cdca5120 32%
./examples/tests+git-clone-test | +base | sha256:d838eb6031fe0bb4f1f203ec26c293afccce4db86abde5d3b1c3a923cdca5120 33%
./examples/tests+excludes-test | +base | sha256:cbdbe7a5bc2a134ca8ec91be58565ec07d037386d1f1d8385412d224deafca08 0%
./examples/tests+excludes-test | +base./examples/tests+excludes-test |  | sha256:cbdbe7a5bc2a134ca8ec91be58565ec07d037386d1f1d8385412d224deafca08 0%
./examples/tests+excludes-test | +base./examples/tests+excludes-test |  | ./examples/tests+excludes-test | sha256:cbdbe7a5bc2a134ca8ec91be58565ec07d037386d1f1d8385412d224deafca08 0%
./examples/tests+git-clone-test | +base./examples/tests+git-clone-test |  | ./examples/tests+git-clone-test | sha256:d838eb6031fe0bb4f1f203ec26c293afccce4db86abde5d3b1c3a923cdca5120 34%
./examples/tests+git-clone-test | +base./examples/tests+excludes-test | +base./examples/tests+excludes-test |  | ./examples/tests+excludes-test | sha256:cbdbe7a5bc2a134ca8ec91be58565ec07d037386d1f1d8385412d224deafca08 0%
./examples/tests+excludes-test | +base./examples/tests+excludes-test |  | sha256:cbdbe7a5bc2a134ca8ec91be58565ec07d037386d1f1d8385412d224deafca08 0%

When a build fails due to a target, all on-going parallel targets are cancelled. The output then lists several targets as having failed - some with "context canceled" reason and one with the original reason that everything was canceled for. The original reason should be more obvious to spot.

Some ideas:

• At the end, print the original reason why everything was stopped.
• Reserve the colors red and orange and use them for this use-case: the original failure gets an extra FAIL label marked with red, while the canceled targets get an extra CANCELED lebel marked with organge.
• Have a way to retrieve the output of the failed target isolated from everything else. This could be a separate command that would be run subsequently by the user.

Issuing

earth +non-existent-target

Currently succeeds (the system succeeds in executing all commands associated with that target - none exist, so all succeed). It should fail.

Buildkit has functionality to store cache in a docker registry. Initial testing showed that it's not just as easy as turning the setting on. Need to explore why that is and address it.

This would make CI integration even better as the system would be much more performant with a cache designed for the purpose.

earthly errors with: Error: parse: invalid number of arguments for BUILD: [--build-arg FOO='$(date +%s)' +build]

when using single quotes for spaces; e.g.

BUILD --build-arg FOO='$(date +%s)' +build

However when doublequotes are used, it works:

BUILD --build-arg FOO="$(date +%s)" +build

https://github.com/earthly/earthly/blob/master/earthfile2llb/parser/EarthLexer.g4#L78 likely needs to be adjusted

in makefiles one can do pattern matching rules like:

%.o: %.c
	$(CC) -I./src/client -I./src/common -I./src/test_common -c $(CCFLAGS) $< -o $@

earthly is missing a similar feature. It would be nice to be able to run BUILD with multiple args:

I'm not sure what the syntax would look like, but something similar to this idea would be nice:

all:
  FROM +code
  foreach file in *.c:
     BUILD --build-arg FOO=$file +buildtarget

Background

Earthfiles are meant to be the entire build definition for a project and not be wrapped in anything else. The reason is that we want no additional layers between the CI and Earthly which could create complexity / confusion / difficulty reproducing CI failures.

However, sometimes developers need to perform operations outside of containers, for development purposes:

• Running the app they are working on
• Running docker containers with host volumes mounted (eg for live reloads)
• Starting a docker-compose stack
• Resetting the dev database state
• Hooking up a debugger
• Etc etc etc

This kind of operations need to be executed on the host environment (or using the host docker daemon).

Traditionally people have used a combination of Makefile recipes or bash scripts to store long commands that they use regularly. This is a proposal for features in Earthly to satisfy this use case.

Proposal

The proposal is to have special Earthly targets that execute the commands on the host system directly, rather than in containers.

Wrap common dev commands

Here is an example use-case: wrapping common dev commands.

unsafe host compose:
  ENV COMPOSE_FILES="-f file1.yml -f file2.yml -f file3.yml -f file4.yml"

unsafe host up:
  FROM +compose
  RUN docker-compose $COMPOSE_FILES up -d "$@"

unsafe host down:
  FROM +compose
  RUN docker-compose $COMPOSE_FILES down "$@"

In this example, the Earthly is used to wrap a complex docker-compose command into simpler forms that can be invoked with just earth +up, earth +down.

Build + start app in one go

Another example is using Earthly to build a series of containers (or making sure that they are up-to-date) and also starting (or restarting) them in a docker compose stack.

all-images:
  BUILD +image1
  BUILD +image2
  BUILD +image3

unsafe host up:
  BUILD +all-images
  RUN docker-compose up -d

How does it work

Notice first of all, the use of unsafe host in the declaration of a target. This is what informs Earthly to run the instructions on the host.

What commands do:

• BUILD +some-target - if the target is of type unsafe host, then simply execute the instructions within (calling targets from other dirs is supported - system cd's into the right dir automatically - but calling a remote target is not supported here). If the target is not unsafe host, run earth +some-target.
• RUN ... - executes the command on the host, in its own shell (eg exporting env vars or cd to different dirs is not propagated from one RUN to another).
• ARG ... - declares an arg, as usual. Variable args are allowed.
• ENV ... - declares an env, as usual.
• FROM ... - executes the commands in the mentioned target before continuing. The referenced target has to be of type local. The key difference from BUILD is that any ENV is propagated in the case of FROM, whereas for BUILD it is not.
• No other commands are supported.

Referencing a local target from a regular target would not be allowed.

Possible extension

Being able to reuse a set of host commands in the context of a containerized build. Example use-case: the same commands to bring up a docker-compose stack could be used both as a local dev environment and also as a setup step for an integration test that runs within Earthly.

In this case, we could use the word "snippet" to declare such special targets. These snippets can then be exectued within regular Earthly targets. The word unsafe can be used in the execution of the earth command itself, as syntactic salt: earth --unsafe +my-snippet.

Possible issues

Executing commands on the host is exactly the kind of thing Earthly stands against :-) . If misused, the user goes back to builds that cannot be easily reproduced. The different commands may do wildly different things on different systems (linux vs mac vs windows) and also caching and idempotency are lost. This proposal could be mistaken for some kind of April fools joke :-)

Also, the language becomes a little more complicated and possibly confusing for new users. It's easy to miss the local keyword and be confused as to how the build actually works.

However, there may be very valid use-cases for this.

Ask

Please upvote / downvote this issue to signify interest. Also, if you have additional example use-cases for such a feature, please leave a comment.

Hi,

so I configured Earthly to use a cache size of 100gb, which should take me a long way, but unfortunately only lasts me for a week or so before I have to clear the cache again.

I'd like to figure out what is actually filling up my cache, but I can't seem to find any way to gain more insight.

I suggest adding commands to help people optimize their builds or decide when a cache clean is necessary:

1. earth cache du/df could list how much space is used/left in the build cache, making it possible to automatically clear the cache when a threshold is reached.*
2. earth cache ls could list the individual docker layers and mount posts command that lists the cached layers and mount points with their respective sizes, giving even more insight about where exactly the disk usage is coming from.

(* I understand that one could also use du -s /tmp/earthly, however this is difficult when earth is called from within a container that mounts the docker.sock of the host system. This is a scenario which often happens when e.g. Jenkins is running in a container and using Earthly to run builds. In this constellation, the cache files will be written to the host file system, not the container file system, making it inaccessible to the container).

Again, good work with this project, I do see a great future ahead for it :)

When earth starts, it checks that the earthly-buildkit container is Running; if not it attempts to start it.

If an existing container exists but is in an exited state the Start() command fails. This should be fixed.

Additionally we will remove the --rm flag so when it does fail, we can look at the logs.

Hello, I have set up earthly to run on Windows with WSL, and I am trying to use a specific image for compiling Unity projects, but I get this error:

+base | ERROR: (FROM gableroux/unity3d:2019.2.11f1-windows) failed to copy: write /tmp/earthly/buildkit/runc-overlayfs/content/ingest/7e1cdf26cfc0f911031b2bd7d6ad7596fe77a38778389fe3f282b2354d40abc6/data: read-only file system
Error: solve side effects: build error group: solve: failed to solve: rpc error: code = Unknown desc = failed to copy: write /tmp/earthly/buildkit/runc-overlayfs/content/ingest/7e1cdf26cfc0f911031b2bd7d6ad7596fe77a38778389fe3f282b2354d40abc6/data: read-only file system

My build.earth simply looks like this:

FROM gableroux/unity3d:2019.2.11f1-windows
WORKDIR /app

build:
        RUN echo "Hello world"`

Currently, as part of a single build, there are multiple Solve commands issued to buildkit and they share nothing other than some copied state. The copied state allows buildkit to reuse a lot of cache.

It's possible that between these Solve's, the cache is purged and then the system unexpectedly has to build the same thing twice.

Is there a better way to do this? Is there a way for us to reuse the session? Would that help with telling buildkit to hold onto the cache for a while?

https://docs.earthly.dev/guides/cache#specifying-cache-size

Cache size is now specified in the config file.

Adding an issue to track this.

First of all, thanks for this interesting new tool!

I have a Makefile that I'm converting to Earth and am running into a weird issue. Consider the following snipet:

.PHONY: _gengo
_gengo:
	protoc \
		-I=. \
		-I=/src \
		-I=/src/github.com/gogo/protobuf/protobuf \
		--gogofaster_out=\
plugins=grpc,\
Mgoogle/protobuf/any.proto=github.com/gogo/protobuf/types,\
Mgoogle/protobuf/duration.proto=github.com/gogo/protobuf/types,\
Mgoogle/protobuf/struct.proto=github.com/gogo/protobuf/types,\
Mgoogle/protobuf/timestamp.proto=github.com/gogo/protobuf/types,\
Mgoogle/protobuf/wrappers.proto=github.com/gogo/protobuf/types,\
Mgoogle/protobuf/empty.proto=github.com/gogo/protobuf/types:\
. \
		./dex.proto

It is important to protoc that the value for --gogofaster_out is a continuous string without spaces. But I'm having trouble getting Earthly to accept the \ escape if it's not preceded by a space.
So when I use the following in my earth.build:

generate-go:
	FROM golang:1-stretch
	RUN go get github.com/gogo/protobuf/protoc-gen-gogofaster
	COPY +protoc/out/bin/protoc /usr/local/bin/protoc
	COPY +gogo-protobuf/out/src /gogo-proto-src
	COPY dex.proto .
	RUN protoc \
		-I=. \
		-I=/gogo-proto-src \
		-I=/gogo-proto-src/github.com/gogo/protobuf/protobuf \
		--gogofaster_out=\
plugins=grpc,\
Mgoogle/protobuf/any.proto=github.com/gogo/protobuf/types,\
Mgoogle/protobuf/duration.proto=github.com/gogo/protobuf/types,\
Mgoogle/protobuf/struct.proto=github.com/gogo/protobuf/types,\
Mgoogle/protobuf/timestamp.proto=github.com/gogo/protobuf/types,\
Mgoogle/protobuf/wrappers.proto=github.com/gogo/protobuf/types,\
Mgoogle/protobuf/empty.proto=github.com/gogo/protobuf/types:\
. \
		dex.proto

I get these errors:

line 41:0 token recognition error at: 'plugins='
line 41:8 token recognition error at: 'grpc,'
line 43:1 token recognition error at: 'google/'
line 43:8 token recognition error at: 'protobuf/'
line 43:17 token recognition error at: 'duration.proto='
line 43:32 token recognition error at: 'github.com/'
line 43:43 token recognition error at: 'gogo/'
line 43:48 token recognition error at: 'protobuf/'
line 43:57 token recognition error at: 'types,'

When I add a space before the \'s, earth can parse the file and starts properly, but protoc doesn't accept the arguments as they are now separate arguments instead of a single value to --gogofaster_out.

I can remove the newlines and have a single long line, but that's not very readable/maintanable. I don't understand why \ need a space before it to escape the newline.

I have an earthly build file that works until I add the folllowing line:

target:
    ...
    EXPOSE 8080
    ...

Adding this line causes Earthly to exit with the following message:

buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)
Error: parser failure: assignment to entry in nil map

Hi, this project is very interesting!

Still, I probably missed something important but from the README it seems pretty close to Docker multi-stage builds. Could somebody elaborate on the differences/advantages?

Thank you

The method proposed to install this software is both not flexible and potentially dangerous.

Currently, there is no user-friendly configuration to keep the system from creating a loop device for the build cache.

This seems to happen now and then, especially if invoking larger builds.

It seems that buildkit GC is not aggressive enough to stay within the confines of the allocated space on the loop device.

Simple workaround is to earth prune --reset and try again. If this happen regularly, increase cache space on disk via export EARTHLY_BUILDKIT_CACHE_SIZE_MB=... (default is 10000).

Sometimes, when the output has some sort of progress bar, earthly does not display it correctly due to the prefix (target name) that it needs to print for every line of the output. This needs to be handled better by Earthly.

Relevant TODO in the code.

Example

Earthly seems to always try pulling images and it won't allow me to build from scratch.

This Dockerfile works:

FROM scratch
WORKDIR /

This earthly build file fails:

FROM scratch
t:
  WORKDIR /

With error:

buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)
Error: parse: apply implicit FROM +base: apply build +base: earthfile2llb for +base: parse: apply FROM scratch: resolve image config for scratch: docker.io/library/scratch:latest: not found

This will also fail if FROM scratch is listed under a target.

How can I build an image from scratch?

Running the Go example, earth floods the terminal with lines like this:

+base | sha256:61b11419476186415e86da00c53c938fd32dc5904a889e1c3d40e238d69d168e 11%
+base | sha256:61b11419476186415e86da00c53c938fd32dc5904a889e1c3d40e238d69d168e 11%
+base | sha256:61b11419476186415e86da00c53c938fd32dc5904a889e1c3d40e238d69d168e 12%
+base | sha256:61b11419476186415e86da00c53c938fd32dc5904a889e1c3d40e238d69d168e 12%
+base | sha256:61b11419476186415e86da00c53c938fd32dc5904a889e1c3d40e238d69d168e 12%
+base | sha256:61b11419476186415e86da00c53c938fd32dc5904a889e1c3d40e238d69d168e 12%
+base | sha256:61b11419476186415e86da00c53c938fd32dc5904a889e1c3d40e238d69d168e 12%
+base | sha256:61b11419476186415e86da00c53c938fd32dc5904a889e1c3d40e238d69d168e 12%

Though it is a lot more of that than I've included here. I would expect this as a progress bar replacing the line on update rather than drawing a new line.

go build is silent on success so this is a lot messier.

Target references and possibly other features (GIT CLONE?) don't support GitLab URLs yet.

CC @rarrais

Docker use is generally not necessary for regular builds. User can always just

FROM <some-image>
RUN --entrypoint

Which is very close to docker run-ing anyway.

However, for complex integration testing, where multiple containers are necessary to run in parallel (eg potentially a whole docker-compose stack), it's necessary that a docker daemon is made available to the build (and also that the docker daemon is isolated well enough with earthly's principles). To allow the use of this docker daemon, the following commands are available, but only as experimental features:

• DOCKER LOAD - loads a target's image output as a docker image within the docker daemon available within the build
• DOCKER PULL - pulls an image from a remote repository - it takes into account proper earthly caching
• RUN --with-docker - allows running docker commands with a docker daemon present

Current issue is that these only work via Docker's vfs storage driver due to docker-in-docker issues. Reason is that overlayfs cannot be used on top of overlayfs. The storage driver vfs is the only driver which works on top of overlayfs. The issue with vfs is that it does not have copy-on-write capabilities, so it ends up copying entire contents for each build layer. This is extremely costly to the disk space and ends up being unusable for anything serious.

Some ideas to work around this:

• Use a host bind-mounted dir for docker-in-docker uses. Within it, create sub-dirs for each docker daemon used within the build. Pass this bind-mount all the way within the build - and within it, mount the docker dir (/var/lib/docker) of the docker-in-docker daemon. Thus, the daemon can now use a native partition (eg ext4), which allows use of overlayfs on top.
• The loop device could be formatted as btrfs, rather than ext4. Btrfs has built-in copy-on-write capabilities and it is not necessary to use overlayfs on top. Buildkit and docker could use btrfs storage driver directly. This would allow chaining docker-in-docker without any overlayfs involved. The current issue is that buildkit does not yet support btrfs. I suspect adding support for it is not too difficult, as it's built on top of containerd & docker which have support for btrfs.
• Use the host docker.sock - however, naming of containers, networks and sharing host resources (eg open ports) would have to be controlled carefully to prevent different parallel builds from interacting with the same resources. This might turn out to be impossible to pull off with good enough isolation.

Support for in-cluster building would be essential to speed up builds and make use of caching.

I'm trying to make an example for .NET Core/C#

I've created sample code based on Java/Scala, but I'm getting extremely confusing error.

radek@radek-ThinkPad-T490s:~/develop/earthly/examples$ earth --no-cache ./dotnet+docker
buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)
context | --> local context ./dotnet
./dotnet+base | --> FROM mcr.microsoft.com/dotnet/core/sdk:3.1.302-alpine3.12
./dotnet+base | resolve mcr.microsoft.com/dotnet/core/sdk:3.1.302-alpine3.12@sha256:a6977770fb5ad9b7535ef867362bb056fa92439a161c2353702440249c2d46dd 0%
./dotnet+base | resolve mcr.microsoft.com/dotnet/core/sdk:3.1.302-alpine3.12@sha256:a6977770fb5ad9b7535ef867362bb056fa92439a161c2353702440249c2d46dd 100%
context | transferring ./dotnet: 0%
context | transferring ./dotnet: 0%
context | transferring ./dotnet: 100%
./dotnet+base | sha256:c991f9f4493de37a1051238d7a1758d68955d4e0b1261fb3e2a87abe94b1a839 100%
./dotnet+base | sha256:a3a43191d31d4956f69774ee89eeea8ceb47b259f7fe0325a6472774cdb8595b 100%
./dotnet+base | sha256:ac47820464eb6c4cde48d5c77047b25ed5d1c629fea54e58ac6604b37bf37bf3 100%
./dotnet+base | sha256:1d4fe617e70734bb22a0d97301c2b87d36cdd66050d3855f74b69f875ba423a3 100%
./dotnet+base | sha256:a6977770fb5ad9b7535ef867362bb056fa92439a161c2353702440249c2d46dd 100%
./dotnet+base | sha256:b0d5a04473425774ee4803fd9c3a89087a60f332c7da8c137ae29a0a48cfe64d 100%
./dotnet+base | sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c 100%
./dotnet+base | sha256:eff32e99990ceb80c49cb2a8037af3f9b53d2f69763f419018bcc02abc930b23 100%
./dotnet+base | sha256:bb9fc6048a9dd25ab6a26f64809be519e91cca2cf15d4e0cdddd0a8f99a3cd94 100%
./dotnet+base | unpacking mcr.microsoft.com/dotnet/core/sdk:3.1.302-alpine3.12@sha256:a6977770fb5ad9b7535ef867362bb056fa92439a161c2353702440249c2d46dd 0%
./dotnet+base | unpacking mcr.microsoft.com/dotnet/core/sdk:3.1.302-alpine3.12@sha256:a6977770fb5ad9b7535ef867362bb056fa92439a161c2353702440249c2d46dd 100%
./dotnet+base | --> WORKDIR /dotnet-example
./dotnet+build | --> COPY [src] src
./dotnet+build | --> RUN [dotnet publish src/HelloEarthly -o publish]
./dotnet+build | Microsoft (R) Build Engine version 16.6.0+5ff7b0c9e for .NET Core
./dotnet+build | Copyright (C) Microsoft Corporation. All rights reserved.
./dotnet+build | 
./dotnet+build |   Determining projects to restore...
./dotnet+build |   Restored /dotnet-example/src/HelloEarthly/HelloEarthly.csproj (in 146 ms).
./dotnet+build |   HelloEarthly -> /dotnet-example/src/HelloEarthly/bin/Debug/netcoreapp3.1/HelloEarthly.dll
./dotnet+build |   HelloEarthly -> /dotnet-example/publish/
./dotnet+build | --> SAVE ARTIFACT publish ./dotnet+build/publish
./dotnet+docker | --> COPY ([]) ./dotnet+build/publish .
Error: solve side effects: build error group: solve: failed to solve: rpc error: code = Unknown desc = failed to compute cache key: failed to create hash for /tmp/PiDh3VmOno5IZWwRHV_oNjZ+jcby9Rt5wuWKeDyQUJY: archive/tar: sockets not supported

My OS is Ubuntu 18
Earthly version 0.1.3
When running the Java and Scala examples, nothing is wrong and the build succeeds.

My code is here : Euphoric@da9d9cc

Currently Earthly builds rely on multiple solve submissions to buildkit. Basically one for the side effects, one for each artifact, one for each image and one for each target with RUN --push instructions.

Although this works reasonably well, there are some possible issues and limitations due to this approach:

• Performance is impacted
• Race conditions are possible as the builds are not associated with each other via a single session. It is possible for a GC to kick in between such related solves and end up needing to rebuild parts that have already been built. This is very rare due to the cache typically not evicting very recent builds - but maybe the GC strategy of buildkit may change.
• Other undiscovered side-effects.

In order to address this, we would need to switch to using the Build API of buildkit, rather than Solve. This would allow to associate multiple solve's within the same session. However, this approach has the following limitations / possible challenges:

• Local dirs need to be specified at the beginning of the build. Possible ways to address this:
  • Add a way in buildkit to incrementally add local dirs in the middle of the session
  • Use two passes when processing earthfiles: one pass to collect all dirs, and another pass to actually take the commands and convert them to LLB (which may also include intermediate solve's). However, this approach prevents us from using variable args for expanding external targets, should we want to do this in the future.
• Exports can become gnarly to track and match back to the originating target as they are returned as a single big blob (?). There may be unexpected hoops we'd need to jump through to make it work for the Earthly use-case.
• The experimental docker-in-docker related features rely on invoking a build in the middle of a build (in order to output a docker image which can be used within a docker-in-docker). This can't be merged into a single build, because the Build API doesn't have the ability to turn a state into a docker .tar in the middle of a build (without exporting to the client).

There's no way to use TLS certs for authenticating the client. Looking at these lines, the likely causes is that the newBuildkitdClient() is called without opts.

Currently, /tmp/earthly is hard-coded.

I suspect workaround via symlink would not work as docker would mount the symlink, not the dir it links to, but not sure.

After performing a successful earth +earth-darwin build, the earthly-buildkitd container is unable to start when attempting to update due to the reuse of the container name.

Error: buildkitd new client: buildkitd new client (own): maybe start buildkitd: start: docker run earthly/buildkitd:v0.2.2: docker: Error response from daemon: Conflict. The container name "/earthly-buildkitd" is already in use by container "48b57c80b6bd5778bf438a54de0e1e79db09e3cf8d629d9270bd953ee291c481". You have to remove (or rename) that container to be able to reuse that name.

Steps should be taken to make this as fast as possible. As it is now, I really like this project but something is adding a lot of latency here.

Changing the string printed in the Go Hello World example.
Local build is 0.7 seconds whereas earth build is 14.6 seconds.

brandon@dev:~/go-earth$ vim main.go # making edit
brandon@dev:~/go-earth$ time go build

real	0m0.732s
user	0m0.413s
sys	0m0.398s
brandon@dev:~/go-earth$ time earth +build
buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)
context | --> local context .
+base | --> FROM golang:1.13-alpine3.11
+base | resolve docker.io/library/golang:1.13-alpine3.11@sha256:5237fe9c1c891aee9ffb22258a412cc7449f310f8f2f65d4389c516218f90443 0%
+base | resolve docker.io/library/golang:1.13-alpine3.11@sha256:5237fe9c1c891aee9ffb22258a412cc7449f310f8f2f65d4389c516218f90443 100%
context | transferring .: 0%
context | transferring .: 0%
context | transferring .: 100%
+base | *cached* --> WORKDIR /go-example
+build | --> COPY [main.go] .
+build | --> RUN [go build -o build/go-example main.go]
+build | Target <redacted>/go-earth:master+build built successfully
=========================== SUCCESS ===========================
+build | Artifact <redacted>/go-earth:master+build/go-example as local build/go-example

real	0m14.589s
user	0m0.824s
sys	0m0.754s

Even with no change and everything cached, the delay that earth adds is several seconds:

brandon@dev:~/go-earth$ time earth +build
buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)
context | --> local context .
+base | --> FROM golang:1.13-alpine3.11
+base | resolve docker.io/library/golang:1.13-alpine3.11@sha256:5237fe9c1c891aee9ffb22258a412cc7449f310f8f2f65d4389c516218f90443 0%
+base | resolve docker.io/library/golang:1.13-alpine3.11@sha256:5237fe9c1c891aee9ffb22258a412cc7449f310f8f2f65d4389c516218f90443 100%
context | transferring .: 0%
context | transferring .: 0%
context | transferring .: 100%
+base | *cached* --> WORKDIR /go-example
+build | *cached* --> COPY [main.go] .
+build | *cached* --> RUN [go build -o build/go-example main.go]
+build | Target <redacted>/go-earth:master+build built successfully
=========================== SUCCESS ===========================
+build | Artifact <redacted>/go-earth:master+build/go-example as local build/go-example

real	0m4.390s
user	0m0.693s
sys	0m0.451s
brandon@dev:~/go-earth$ time go build

real	0m0.179s
user	0m0.240s
sys	0m0.138s

A feature request is the support for .env files (similar to how compose uses them https://docs.docker.com/compose/env-file/).

It might work like this: if an .env file is present in the current dir, or any dir above the current dir (dirs closer to current dir have higher precendence), load the vars and pass them to the build as both build-args AND secrets. EDIT: Using the .env in the current dir only seems to be the the behavior most similar to how other tools work.

CC @zenyui

Thank you for making earthly. It's the smartest thing I've seen in a long while.

Is this possible with buildkit? This would be a nice way to control image layers in the final image. Ref: makisu
Specifically: https://github.com/uber/makisu#explicit-commit-and-cache

Currently the following is not interpreted as a comment. It should.

RUN echo abc # Print abc

On some systems, /tmp is a small tmpfs partition. This creates 2 problems for putting Earthly's cache there:

• not enough space available for large builds

• contents are lost on reboot

Consider checking for the /var/cache directory and putting Eartlhy's cache there if it is available.

related issue: Explore remote cache · Issue #11 · earthly/earthly

reference: Features/tmp-on-tmpfs - Fedora Project Wiki

debugger-docker:
	FROM busybox:latest # cant be FROM scratch because the args require sh to exist

produces the error:

buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)
Error: parse: invalid number of arguments for FROM: [busybox:latest # cant be FROM scratch because the args require sh to exist]

This does not work:

RUN echo something ;\
    echo something else

However this does:

RUN echo something ; \
    echo something else

They should both work.

When building the same target twice, but with different args, as part of the same build, currently it is difficult to tell apart the output of the two.

We should use different colors for the console output, at a minimum. Perhaps when the target starts being built, also list the values of the args being used.

If I do the following in my build.earth file:

ENV ${CODE_SRC}=/go/src/<project>

protobuf:
    COPY --dir api ${CODE_SRC}/backend

Then running earth +protobuf results in the files being copied into /backend:

+protobuf | --> COPY [api] /backend

I have worked around this by specifying the full path in the copy command:

ENV ${CODE_SRC}=/go/src/<project>

protobuf:
    COPY --dir api /go/src/<project>/backend

Hi, I am trying to download a go module from a private github repository within an earthly build and I cannot figure out how this is supposed to work. In the earthly documentation I read about SSH agent socket passing, but I think this is only intended for the GIT CLONE command and won't work for go mod download?

So I tried to do it like described in the buildkit doc, but then docker complains about the missing mount point:

RUN git config --global url."ssh://[email protected]/some-org/".insteadOf https://github.com/some-org/ 
RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
RUN --mount=type=ssh GOPRIVATE=github.com/some-org go mod download

Could you please point me in the right direction?

When I use earth in my default shell/terminal it always results in this error:

$ earth ./dexterity-api/dex/+generate
buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)
buildkitd | Settings do not match. Restarting buildkit daemon with updated settings...
buildkitd | Is docker installed and running? Are you part of the docker group?
Error: buildkitd new client: buildkitd new client (own): maybe start buildkitd: maybe restart: docker run earthly/buildkitd:v0.1.1: 80d0ea1166ef829ef485f42ae43760ba86c1ac13bd00da4d30efeb08bec2c203
docker: Error response from daemon: error while creating mount source path '/tmp/ssh-agent.sock.1000': mkdir /tmp/ssh-agent.sock.1000: file exists.
: exit status 125

From my env vars:

SSH_AUTH_SOCK=/tmp/ssh-agent.sock.1000

I found a quick fix is to unset SSH_AUTH_SOCK and then run earth ... But this will probably cause trouble down the road when I want to let Earthly checkout a private git repo.

Experienced this after adding the build.earth to one of my projects:

$ earth +build
buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)
Error: resolve build context for target +build: Unable to parse git URL ssh://[email protected]/ShiftLeftSecurity/check-string.git

Removing .git/ makes it work.

@zenyui reports issues with using Earthly in combination with Quay as a private registry.

It seems that Earthly is able to resolve to the right sha, but pulling doesn't work. It randomly works for some repos, but not for others. Unknown what causes it to work / no work.

System is a Mac. Connection is over VPN.

Currently the RUN arguments are not expanded for ARG's. They are expanded, however, via the shell. This is different to how Dockerfiles work, however, and may be surprising to the user in some cases.