Comments (13)
I think this issue has the same root cause as anchore/syft#2894, which is more obviously a bug in Syft.
from grype.
Hi @JordanFaust,
Would you mind providing more detail in your repro steps? I have never used Nix before, and I don't know how to get into a situation where I can run nix profile install nixpkgs/92d295f588631b0db2da509f381b4fb1e74173c5#grype
. Can you provide a Dockerfile or set of steps that can take me from "this machine has never run nix before at all" to "I can watch this error happen?" I'd prefer a Dockerfile specifically if that's possible.
I tried the following:
- Run
docker run -ti ghcr.io/nixos/nix
- this puts me at a shell in Nix - In the container run
nix profile install nixpkgs/92d295f588631b0db2da509f381b4fb1e74173c5#grype
This fails with so error: experimental Nix feature 'nix-command' is disabled; add '--extra-experimental-features nix-command' to enable it
. I've tried a few things, but I haven't gotten as far as the error you're seeing.
Thanks!
from grype.
Unfortunately it looks like this is a specific issue for NixOS and not nix the package manager. The above runs a container with a different OS and just has the nix CLI. I'm not sure the best way to test that in a container. I'm not sure what the specific issue is but something about these tests are failing when being built within NixOS. If you have any ideas on what could be the source I can try it out myself and try building from there.
from grype.
I wasn't able to reproduce this issue:
FROM ubuntu:22.04
# install prerequisites
RUN apt-get update && apt-get install -y curl sudo xz-utils
# create a non-root user (recommended for running Nix)
RUN useradd -m nixuser && echo 'nixuser ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
# switch to the non-root user
USER nixuser
WORKDIR /home/nixuser
# install Nix
RUN curl -L https://nixos.org/nix/install | sh
# set up Nix environment
ENV USER nixuser
ENV HOME /home/nixuser
ENV PATH /home/nixuser/.nix-profile/bin:/home/nixuser/.nix-profile/sbin:/nix/var/nix/profiles/default/bin:/nix/var/nix/profiles/default/sbin:/run/current-system/sw/bin:/home/nixuser/.local/bin:$PATH
RUN . /home/nixuser/.nix-profile/etc/profile.d/nix.sh
# enable experimental features in Nix configuration
RUN mkdir -p /home/nixuser/.config/nix
RUN echo "experimental-features = nix-command flakes" >> /home/nixuser/.config/nix/nix.conf
Then:
$ docker build -t localhost/nix:latest .
$ docker run --rm -it localhost/nix:latest
And from within the container, everything seems to work:
$ docker run --rm -it localhost/nix:latest
nixuser@78cbeeb35219:~$ nix profile install nixpkgs/92d295f588631b0db2da509f381b4fb1e74173c5#grype
nixuser@78cbeeb35219:~$ grype alpine:latest
✔ Vulnerability DB [updated]
✔ Parsed image sha256:ace17d5d883e9ea5a21138d0608d60aa2376c68f616c55b0b7e73fba6d8556a3
✔ Cataloged contents a0264d60f80df12bc1e6dd98bae6c43debe6667c0ba482711f0d806493467a46
├── ✔ Packages [15 packages]
├── ✔ File digests [80 files]
├── ✔ File metadata [80 locations]
└── ✔ Executables [17 executables]
✔ Scanned for vulnerabilities [14 vulnerability matches]
├── by severity: 0 critical, 0 high, 12 medium, 0 low, 0 negligible (2 unknown)
└── by status: 2 fixed, 12 not-fixed, 0 ignored
NAME INSTALLED FIXED-IN TYPE VULNERABILITY SEVERITY
busybox 1.36.1-r15 apk CVE-2023-42366 Medium
busybox 1.36.1-r15 apk CVE-2023-42365 Medium
busybox 1.36.1-r15 apk CVE-2023-42364 Medium
busybox 1.36.1-r15 apk CVE-2023-42363 Medium
busybox-binsh 1.36.1-r15 apk CVE-2023-42366 Medium
busybox-binsh 1.36.1-r15 apk CVE-2023-42365 Medium
busybox-binsh 1.36.1-r15 apk CVE-2023-42364 Medium
busybox-binsh 1.36.1-r15 apk CVE-2023-42363 Medium
libcrypto3 3.1.4-r5 3.1.4-r6 apk CVE-2024-2511 Unknown
libssl3 3.1.4-r5 3.1.4-r6 apk CVE-2024-2511 Unknown
ssl_client 1.36.1-r15 apk CVE-2023-42366 Medium
ssl_client 1.36.1-r15 apk CVE-2023-42365 Medium
ssl_client 1.36.1-r15 apk CVE-2023-42364 Medium
ssl_client 1.36.1-r15 apk CVE-2023-42363 Medium
A newer version of grype is available for download: 0.77.4 (installed version is 0.76.0)
Are there other specific details that I'm missing from this setup?
from grype.
I don't believe there is an issue for nix the package manager. This specifically is an issue with NixOS and something specific about the nature of the distro tests that are failing within that hermetic build environment. I will need to dig into what the distro tests are actually doing to simulate the different distros. I'm assuming something in the test setup is causing issues within the nixos build environment. I can try and track this down if you can help me understand these tests a little better.
from grype.
Thanks @JordanFaust !
What the "distro" tests are doing is testing Grype's ability to detect what distro the image it's scanning is based on.
The tests are defined here:
grype/grype/distro/distro_test.go
Line 109 in 8347931
My guess is that something in here is broken by the hermetic build environment.
The way the tests essentially work is:
- There's a test fixture that looks the identifying files on the distro, for example https://github.com/anchore/grype/blob/main/grype/distro/test-fixtures/os/alpine/etc/os-release looks like
/etc/os-release
from an Alpine image. - The test makes a source and file resolver from this directory - the source and file resolver are grype's abstraction over file I/O.
- The tests exercise Grype's distro identification code by passing it the resolver.
They're failing at require.NotNil(t, release, "empty linux release info")
, which means that given the file resolver, Grype couldn't identify the distro. I suspect this is because something about the hermetic build prevents Grype from doing the necessary file I/O or something like that.
Creating the file resolver is happening over in Syft code here: https://github.com/anchore/syft/blob/2d318cffaaca07a8b3e87d68f80f881153a305b8/syft/source/directorysource/directory_source.go#L135
I don't see anything in there that I think NixOS should be blocking - the chroot
-named functions don't seem to be actually making a chroot
system call, just returning a file resolver rooted outside the current directory.
Thanks for taking a look! Let me know if there's any other help I can be.
from grype.
Hello all, I help maintain the syft and grype packages in nixpkgs. I'm looking into this as part of our Zero Hydra (our ci system) Failures process for the next stable release.
https://hydra.nixos.org/build/259554448/nixlog/3
This is the tests running within the build sandbox.
I've tried just running go
as normal on my NixOS machine and also get the issue
I have trimmed the logs as they're nothing new.
$ go test -v -run 'Test_NewDistroFromRelease_Coverage' ./grype/distro/...
=== RUN Test_NewDistroFromRelease_Coverage
=== RUN Test_NewDistroFromRelease_Coverage/test-fixtures/os/alpine
distro_test.go:254:
Error Trace: /home/jk/projects/personal/grype/grype/distro/distro_test.go:254
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/os/alpine
Messages: empty linux release info
#...
=== RUN Test_NewDistroFromRelease_Coverage/test-fixtures/os/chainguard
distro_test.go:254:
Error Trace: /home/jk/projects/personal/grype/grype/distro/distro_test.go:254
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/os/chainguard
Messages: empty linux release info
=== NAME Test_NewDistroFromRelease_Coverage
distro_test.go:272: defined: archlinux
#...
distro_test.go:272: defined: debian
distro_test.go:277: distro coverage incomplete (defined=19, coverage=0)
--- FAIL: Test_NewDistroFromRelease_Coverage (0.01s)
--- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/os/alpine (0.00s)
#...
--- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/os/chainguard (0.00s)
FAIL
FAIL github.com/anchore/grype/grype/distro 0.013s
FAIL
Interestingly in a container I don't have the issue
docker run -it nixos/nix nix develop --experimental-features "nix-command flakes" nixpkgs/92d295f588631b0db2da509f381b4fb1e74173c5#grype
Will put you in a shell with the dependencies used at build time
You can then run phases="unpackPhase patchPhase" genericBuild
to get you into a situation where the source code is available in $PWD
Then you can run go test -v -run 'Test_NewDistroFromRelease_Coverage' ./grype/distro/...
And I'm not getting the issue inside the nixos/nix docker container.
Is there an easy way to make the logs visible so I can check the errors during the linux.IdentifyRelease()
call? Just seeing nil
doesn't narrow it down much.
If the logs aren't easily visible I'll probably make it return the error
from grype.
Thanks for taking a look @06kellyjac!
I haven't found an easy way to enable Grype's normal logging during a test run, so it might make sense to do a quick local change to return the error.
from grype.
Here's what's needed to log within tests (today... we can probably make this better):
import (
"github.com/anchore/go-logger"
"github.com/anchore/go-logger/adapter/discard"
"github.com/anchore/go-logger/adapter/logrus"
"github.com/anchore/grype/internal/log"
"github.com/anchore/syft/syft"
...
)
...
for _, test := range tests {
t.Run(test.fixture, func(t *testing.T) {
l, err := logrus.New(
logrus.Config{
EnableConsole: true,
Level: logger.TraceLevel,
},
)
syft.SetLogger(l)
log.Set(l)
require.NoError(t, err)
require.NotNil(t, l)
// run your test...
syft.SetLogger(discard.New())
log.Set(discard.New())
})
}
Another observation, the link to the logs doesn't seem to be running tests with -v
, which means that stdout will be suppressed. So you'll need to make both changes to see the log output in your CI environment.
from grype.
Running locally with those changes:
I don't see anything super useful here from my understanding. Unless maybe it sees "ignoring /" and that ignores everything? But I wouldn't expect that as the test fixture dir is meant to be the root from that point right?
I don't see any of the error traces so unless the logging isn't set up right I guess it's not running into problems there
=== RUN Test_NewDistroFromRelease_Coverage/test-fixtures/os/chainguard
[0000] WARN no explicit name and version provided for directory source, deriving artifact ID from the given path (which is not ideal)
[0000] DEBUG ignoring system mountpoint mountpoint=/
[0000] DEBUG ignoring system mountpoint mountpoint=/run
[0000] DEBUG ignoring system mountpoint mountpoint=/dev
[0000] DEBUG ignoring system mountpoint mountpoint=/dev/shm
[0000] DEBUG ignoring system mountpoint mountpoint=/proc
[0000] DEBUG ignoring system mountpoint mountpoint=/run/wrappers
[0000] DEBUG ignoring system mountpoint mountpoint=/sys
[0000] DEBUG ignoring system mountpoint mountpoint=/run/user/1000
[0000] TRACE indexing filetree path=test-fixtures/os/chainguard
distro_test.go:268:
Error Trace: /home/jk/projects/personal/grype/grype/distro/distro_test.go:268
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/os/chainguard
Messages: empty linux release info
=== NAME Test_NewDistroFromRelease_Coverage
Just running locally in my case I've got a tmpfs on root. Not 100% sure for the CI.
There's a non-insignificant amount of NixOS users following this pattern to identify where their state is.
https://grahamc.com/blog/erase-your-darlings/
https://github.com/nix-community/impermanence
from grype.
[0000] DEBUG ignoring system mountpoint mountpoint=/
ha! Agreed, that one seems a tad impactful (that will ignore the entire filesystem I believe). Syft doesn't catalog within certain filesystem mounts, so it seems like there is something peculiar with the system mounts (or something that needs to be corrected in syft).
Can you confirm on a system that is exhibiting the bad behavior what the filesystem types for each mount? Would you be willing to post the output of df
from that system? (or a portion of it?)
edit: I do see a hint from the links you posted that we know the answer already...
If you have a lot of RAM, you could skip the erase step and make / a tmpfs.
from grype.
Hi @JordanFaust and @06kellyjac I believe anchore/syft#2918, released in https://github.com/anchore/syft/releases/tag/v1.6.0 (which is now a couple behind - latest is https://github.com/anchore/syft/releases/tag/v1.8.0) fixed this. Would you mind re-testing? Please let us know if you're still facing this issue.
from grype.
I can confirm things are working. I can successfully build grype within nixos now
from grype.
Related Issues (20)
- Merge Configuration Files HOT 6
- False Positive: GHSA-248v-346w-9cwc/(CVE-2024-39689) reported for certifi library in python HOT 2
- False negatives on Java org.webjars/bootstrap and org.webjars/jquery HOT 3
- Fail if explicitly specified config file is absent HOT 1
- Update operations are non atmoic across processes HOT 1
- Grype should expand `~` in paths in config file HOT 1
- Top level `output` config should only affect grype root command HOT 1
- Different results scanning PHP SBOMs generated by cdxgen and Syft HOT 1
- Ignoring search results when CPE is not set in the SBOM HOT 1
- Scan specific file
- Noisy INFO logs on scanning composer.lock SBOM generated by Syft
- Failed to parse constraint of CVE-2024-6345 which fails the scan HOT 6
- Check for stale DB on quality gate runs
- Grype only supports SKOPEO when using 'docker-archive' format. HOT 3
- Possible FP GHSA-6757-jp84-gxfx \ CVE-2020-1747 HOT 1
- Posiible FP on pillow package inside amazon image HOT 4
- False Positive: GHSA-jfmj-5v4g-7637 (CVE-2024-5569) python3-zipp due to Syft noise and mismatch of package name HOT 1
- Grype panics with a nil pointer dereference error when given an empty string argument
- no grype db was published since 17.8 HOT 1
- False Positive: GHSA-j225-cvw7-qrx7 (CVE-2023-52323) python3-pycryptodome
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from grype.