numtide / nar-serve Goto Github PK
View Code? Open in Web Editor NEWUnpack and serve NAR file content on the fly
License: Apache License 2.0
Unpack and serve NAR file content on the fly
License: Apache License 2.0
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
Dockerfile
golang 1.23
.github/workflows/go.yml
actions/checkout v4
actions/setup-go v5
.github/workflows/release.yml
actions/checkout v4
actions/setup-go v5
goreleaser/goreleaser-action v6
go.mod
go 1.21
go 1.23.0
cloud.google.com/go/storage v1.43.0
github.com/aws/aws-sdk-go v1.55.3
github.com/go-chi/chi/v5 v5.1.0
github.com/go-chi/hostrouter v0.2.0
github.com/google/go-cmp v0.6.0
github.com/klauspost/compress v1.17.9
github.com/stretchr/testify v1.9.0
github.com/ulikunitz/xz v0.5.12
When multiple clients access the same NAR file at the same time, we could download the file from the S3 bucket only once.
type CachedBinaryCacheReader struct {
Reader BinaryCacheReader
Lock map[string]io.ReaderCloser
Mutex sync.Mutex
}
func (r CachedBinaryCacheReader) GetFile(ctx context.Context, path string) (io.ReadCloser, error) {
}
nar-serve is the only user of that library. This makes it easier to iterate on it.
now.sh doesn't support the http.Flusher interface. aws-lambda-go doesn't
support streaming the content back either:
aws/aws-lambda-go#195
It would be nice if we can define multiple store URLs which will be tried one after another, or in parallel.
My use case here: Query my personal binary cache and fall back to cache.nixos.org afterwards.
Some people's binary cache live in S3.
This will require to extend the go-nix library so that it can parse a store URI and return the right backend for us.
Is your feature request related to a problem? Please describe.
My situation is that I build a system that totals around 5-10GB (currently with debs, but transitioning to Nix) and then run an extensive simulation-based testing framework on that system. Regardless of what I do about build (eg, moving to Hydra), I'd like to be able to continue using Jenkins and Kubernetes workers for the testing aspect of it.
Some "conventional" options could be:
Both of these options potentially incur a lot of unnecessary transfer and archive manipulation and don't really leverage the power of Nix.
Describe the solution you'd like
This may be well out of scope for this project, but it would be super cool if it was able to act as a Kubernetes volume plugin and transparently fill requests for /nix/store
paths from a cache that was shared between all containers on the host:
https://github.com/kubernetes/community/blob/master/sig-storage/volume-plugin-faq.md
This could be set up as an overlay so that writes would always be container-local, but reads would fall through to the volume.
Describe alternatives you've considered
See above.
Instead of downloading the NAR files over and over from the cache, it might be nice to keep a local copy. The LRU cache should be based on a size or percent of the disk space available.
Publish the git tag, and then add it to nixpkgs.
Nix clients who upload with write-nar-listing = true
also upload a .ls
file that contains all the archive metadata.
nix run nixpkgs.{brotli,curl, bash} -c \
curl -v https://cache.nixos.org/$(readlink $(which bash) | cut -d/ -f4 | cut -d- -f1).ls | brotli -d
This could be used to quickly determine whenever a file exists and also create directory listings.
The API currently only works if the storeDir is set to the default /nix/store
. It would be nice if other stores were supported.
I think this would require to change the API shape a bit.
Instead of /nix/store/<hash>...
, load the storeDir and add a prefix: /unpack/<storeDir>/<hash>...
The NAR archive doesn't store any file attributes except the file type, and whenever the file is an executable.
It would be nice to expose whenever a file is executable in a HTTP header.
Eg:
X-Nar-Executable: true
?
Hello,
It seems like HTTP caching headers could score you some very cheap, significant wins.
I saw Cache-Control
already returns immutable
, I think you could probably push that to public, max-age=31536000, immutable
, as well as providing ETag
, Last-Modified
and Expires
.
There are more cases where those can be returned as well, I think that any response for which a NAR was successfully fetched, so directory listing, redirects, but also 404s when a file was not found.
MDN has a great resource about this: https://developer.mozilla.org/en-US/docs/Web/HTTP/Caching
Pretty cool project!
Bests
From #20, the new 0.6.1's api/unpack/index.go
, it looks like hdr.Path
has /-prefixed names like /bin/hello
, whereas the old 0.4.0's api/unpack/index.go
, hdr.Name
names are not /-prefixed, like bin/hello
.
This breaks fetching single files from inside archives (server-side unpacking).
To Reproduce
Steps to reproduce the behavior:
import os
in nixos/tests/nar-serve.nix
nix-build nixos/tests/nar-serve.nix
Expected behavior
Test passes
Actual behavior
Test fails: request returns 404 not found:
server: must succeed: curl -o hello -f http://localhost:8383/nix/store/cv79b81pjmva78whwwpr66l5mfj4pijh/bin/hello
server # [ 29.716220] nar-serve[776]: 3 [cv79b81pjmva78whwwpr66l5mfj4pijh bin hello]
server # [ 29.722451] nar-serve[776]: Fetching the narinfo: cv79b81pjmva78whwwpr66l5mfj4pijh.narinfo from: http://localhost/
server # [ 29.732232] nar-serve[776]: narinfo StorePath: /nix/store/cv79b81pjmva78whwwpr66l5mfj4pijh-hello-2.12
server # [ 29.736556] nar-serve[776]: URL: nar/14dp7chhq8akhgyqvyfkdwgg4d696agd0zga80ib0f382qhc6p8c.nar.xz
server # [ 29.741487] nar-serve[776]: Compression: xz
server # [ 29.743802] nar-serve[776]: FileHash: sha256:14dp7chhq8akhgyqvyfkdwgg4d696agd0zga80ib0f382qhc6p8c
server # [ 29.748350] nar-serve[776]: FileSize: 43624
server # [ 29.751017] nar-serve[776]: NarHash: sha256:0wbkw4v6ykfmm0jjp61z28f9nsf8wzx2cb4mz4g0gnv64sj2vsdk
server # [ 29.754678] nar-serve[776]: NarSize: 181368
server # [ 29.755824] nar-serve[776]: References: cv79b81pjmva78whwwpr66l5mfj4pijh-hello-2.12 f3qlm2873bxlhxns4lrmrinvbzn933pj-glibc-2.34-210
server # [ 29.759341] nar-serve[776]: fetching the NAR: nar/14dp7chhq8akhgyqvyfkdwgg4d696agd0zga80ib0f382qhc6p8c.nar.xz
server # [ 29.771493] nar-serve[776]: newPath bin/hello
server # [ 29.778593] nar-serve[776]: [negroni] 2023-05-26T00:40:05Z | 404 | 63.002979ms | localhost:8383 | GET /nix/store/cv79b81pjmva78whwwpr66l5mfj4pijh/bin/hello
server # curl: (22) The requested URL returned error: 404
...
Exception: command `curl -o hello -f http://localhost:8383/nix/store/cv79b81pjmva78whwwpr66l5mfj4pijh/bin/hello` failed (exit code 22)
System information
NixOS 23.05beta78.04aaf851167
Additional context
Discovered in NixOS/nixpkgs#178851 (comment)
Looks easily fixed: #24
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.