Comments (3)
FWIW, I'm not a Go programmer, but by default I don't think there's a way around this. You have to download the files and calculate the proper hash manually as part of the generation step. Many other tools work this way in some form or another, not just in Buck2-land but elsewhere.
But, to work around this today, instead of using ctx.actions.download_file
you instead could try using ctx.actions.run("curl -L -o ...")
to download the file through a command, then untar it, then check the file hash. I think this won't work with Remote Execution (your RBE runners probably have their network cut off), but it's close to what you want.
I think that to support this Buck would need a feature that the Nix package manager calls "fixed output derivations" or "FODs". These are build rules where you don't give the hash of the downloaded file itself, but instead give a hash of the expected output files that some arbitrary commands produce.
For example, if you download https://example.com/foo-0.1.tar.gz
, you can give the hash of this file directly. But this hash is fragile, because the server admin may recompress the file with different compression settings. But the content of the tarball would remain the same in any case. Therefore, you want to download a file, then run a command to extract it, then check the hash of the output files. This feature also means you can e.g. completely change the download URL to something arbitrary and it will still work and get cache hits/early cut-off, assuming that the contents are in fact the same.
I don't know what FODs would look like in Buck2 at all.
from buck2.
Another way could be not using a GOPROXY and using the default VCS resolving algorithm go uses. This is much more complicated than using the user supplied or default GOPROXY but would probably circumvent unnecessary download.
from buck2.
@thoughtpolice you could imagine doing this in a dynamic_outputs
like way. Imagine essentially the exact same API that dynamic_outputs
has today, except that on the call to dynamic_outputs
you also have to specify the hash all of the output artifact that you're later going to bind within the dynamic_outputs
. In exchange for that, the analysis_context
in the dynamic function could give you access to a version of ctx.actions.download_file
that doesn't require specifying the hash. That's really only part of what I understand FODs to provide, you'd probably also want such a construct to change caching behavior in other ways that is less obvious to really see the benefits.
I agree with Austin though, your best option right now is to have a locally run action that invokes curl
.
from buck2.
Related Issues (20)
- Why can not Buck trigger Buildbarn's g++ using system_cxx_toolchain?
- 'buck2 build //tests:' can not find 'sh' when using case 'buck2/examples/remote_execution/buildbarn/tests'
- License details (for SPDX compliance) HOT 2
- Buck2 performance on single-file update in erlang project HOT 13
- `SRCDIR` is not absolute in genrule script HOT 2
- It should be possible to consume multiple `default_outputs` in an `attrs.list(attrs.source())` HOT 5
- RE: upload cancelled with "stream error: stream no longer needed" HOT 5
- Zig support HOT 2
- Handling environment variables for local and remote builds HOT 2
- How can I check and/or override the path for the linker for rust/cxx? HOT 1
- C++ mixed linking HOT 4
- Read output of `actions.run` from BXL HOT 6
- [Feature request] Support custom caching for non-reproducible actions? HOT 4
- Transitive shared libraries not added to rpath of otherwise static binaries HOT 3
- Errant octal escape strings in prelude docs HOT 1
- RE: Error, message length too large for `BatchReadBlobs` HOT 1
- buck2 fails to build on m3 mac
- Treat
- Test
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from buck2.