Comments (10)
Today I got into same issue in one of our build pipelines..
Simplest pipeline that fails..
VERSION 0.8
ARG --global EARTHLY_GIT_HASH
KBT_PREPARE_IMAGE:
FUNCTION
kbt-prepare:
FROM alpine
DO --pass-args +KBT_PREPARE_IMAGE
kbt-image:
FROM alpine
BUILD --pass-args +kbt-prepare
from earthly.
Same issue here. It seems if a target uses a built-in arg and uses --pass-args to invoke other targets, the issue is raised.
from earthly.
I wrote a simple patch to ignore variable names that matches any built-in args instead of erroring out, I can create the PR but I am not sure if this behavior is acceptable by the Earthly team.
I wasn't too sure how to propagate the offending variables up to a "console writer" or how I can refer to the console within the parseArg
function. I believe this can be better polished by the core devs but this works for my issue right now.
diff --git a/variables/parse.go b/variables/parse.go
index 9355bb55..805a5f34 100644
--- a/variables/parse.go
+++ b/variables/parse.go
@@ -1,6 +1,7 @@
package variables
import (
+ "fmt"
"os"
"strings"
@@ -72,13 +73,14 @@ func parseArg(arg string, pncvf ProcessNonConstantVariableFunc, current *Collect
}
if hasValue {
if reserved.IsBuiltIn(name) {
- return "", "", errors.Errorf("value cannot be specified for built-in build arg %s", name)
- }
- v, err := parseArgValue(name, value, pncvf)
- if err != nil {
- return "", "", err
+ fmt.Printf("value cannot be specified for built-in build arg %s and will be ignored\n", name)
+ } else {
+ v, err := parseArgValue(name, value, pncvf)
+ if err != nil {
+ return "", "", err
+ }
+ return name, v, nil
}
- return name, v, nil
}
v, ok := current.Get(name, WithActive())
if !ok {
from earthly.
Thanks for taking a deep dive @dustyhorizon
I think it would be better to tackle it under
earthly/earthfile2llb/converter.go
Line 1708 in 34320d5
earthly/earthfile2llb/converter.go
Line 1901 in 34320d5
However, there's some additional issues though that we need to think about, for example consider:
VERSION 0.8
FROM alpine
a:
RUN env | grep EARTHLY
test:
ARG EARTHLY_GIT_HASH
ARG EARTHLY_TARGET
ARG EARTHLY_NOT_A_RESERVED=hello
BUILD --pass-args +a
when running +test
, I would expect regular non reserved args to be passed in, e.g. the EARTHLY_NOT_A_RESERVED
argument. so when we get to the a
target, we should see:
+a | EARTHLY_NOT_A_RESERVED=hello
But I don't know if we should also see the built ins, e.g.
+a | EARTHLY_TARGET=+test
+a | EARTHLY_GIT_HASH=1d4969b125fb12b010684a1e7fd98a1008086022
I'm leaning towards yes, since pass args should pass all the args in it's scope (and it's designed to prevent users from having to always re-declare ARGs)
What I don't fully know is what we should expect the value of EARTHLY_TARGET
to be -- it was defined under the test
target, but it was passed to a
-- should its value change to be EARTHLY_TARGET=a
even though it wasn't defined under a
? I think so, but am not entirely sure -- it's an edge case we didn't think about.
edit: let's not pass them along -- after combining the scopes, we could introduce a new function like overriding = RemoveReserved(overriding)
from earthly.
In second thoughts (after chatting with @mikejholly), it seems easier to avoid these edge cases of what the values should be by simply not passing built in args.
The --pass-args
feature was created to make it easier to pass args between different targets that are chained between multiple Earthfiles (without having to write tons of glue code); builtin arguments don't have this problem because they are builtin and can simply be declared in the final target and they'll be correctly populated.
from earthly.
@alexcb Your last comment sees reasonable..
How hard it will be to release that change?
Cause it started to pop up in many places in our pipelines (cause I upgraded to 0.8 by default)..
from earthly.
How hard it will be to release that change?
I moved it up in our todo priority; however we'll gladly accept a PR if anyone is able to get to it sooner.
from earthly.
potential fix in #3872
from earthly.
@dustyhorizon can you provide a concrete example for this issue? I went to test it against v0.8.4 and couldn't replicate the issue; here's what I tried:
VERSION 0.8
test:
FROM alpine
ARG EARTHLY_GIT_HASH
RUN echo here1 && env | grep EARTHLY
BUILD --pass-args +subtest
subtest:
FROM alpine
RUN echo here2 && env | grep EARTHLY
and
VERSION 0.8
ARG --global EARTHLY_GIT_HASH
test:
FROM alpine
RUN echo here1 && env | grep EARTHLY
BUILD --pass-args +subtest
subtest:
FROM alpine
RUN echo here2 && env | grep EARTHLY
both output the same values:
/h/a/j/pass-args-builtin+test | here1
/h/a/j/pass-args-builtin+test | EARTHLY_GIT_HASH=434d11e85797c9199b62fbb85fb1e4c03fb89cb5
/h/a/j/pass-args-builtin+subtest | here2
/h/a/j/pass-args-builtin+subtest | EARTHLY_GIT_HASH=434d11e85797c9199b62fbb85fb1e4c03fb89cb5
from earthly.
fixed in v0.8.5
from earthly.
Related Issues (20)
- how can i inspect every stage cost time
- `WITH DOCKER --cache-id` does not expand the `--cache-id` value HOT 7
- Cache with higher precision timestamps HOT 1
- `RUN --raw-output` doesn't work `WITH DOCKER` HOT 1
- less verbose `failed to parse key scan "this-is-a-bad-entry": invalid keyscan` errors
- Create matrix builds from CLI arguments HOT 1
- `RUN --interactive` triggers HTTP 500 when used `WITH DOCKER --load` HOT 1
- Can I `COPY` artifacts from a matrix build? HOT 2
- `--pass-args` busts cache unexpected HOT 8
- DIND networking dont work when using non standard CNI_MTU
- `TRY` / `FINALLY` when used with `WITH DOCKER` produces verbose output HOT 2
- Earthly doesn't respect a host defined in a local `/etc/hosts` file when doing SAVE IMAGE HOT 2
- ticktock cache issue HOT 3
- `TRY` / `FINALLY` doesn't work under `WITH DOCKER`
- error message for WITH DOCKER (unlazy force execution) could be improved
- self-hosted registry reachable from host machine but dns look up failed inside earthly HOT 2
- `+base` target should only be run when necessary
- Running TestContainers in Earthly fails to find Docker Daemon HOT 4
- `docker network create --internal` can fail with a ` network with name ... already exists` error when `WITH DOCKER --cache-id=` is used HOT 2
- BuildKit: Snapshot directory does not exist HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from earthly.