Comments (5)
I've just run into this myself - was using
img.Cleanup
per the basic example, only to find ~32gb of tmp files from stereoscope😅Switching to
stereoscope.Cleanup()
looks like it's resolved the problem - it would be good to know the backstory on the comment @slimdevl you've indicated, since it does seem like that should be called instead (or maybe in addition to?)img.Cleanup
.
I did a little digging this weekend and discovered the following:
- One cannot call
stereoscope.Cleanup
if the process uses the library for subsequent scan calls. This is becauserootTempDirGenerator
is the foundation of all allocated temp dirs, and the base directory attached to it is gone after first cleanup. This cascades into subsequent scan failures. So callingstereoscope.Cleanup
is not a good option for long running processes. selectImageProvider()
allocates a childtempDirGenerator
. It passes that to the Provider (in my case DaemonProvider), which in turn passes only the directory to theimage.NewImage()
call. This passes the responsibility to theImage.Cleanup()
method to cleanup directories. However this cannot work because *only the passed directory is cleaned up by the image, and theDaemonProvider
directory is leaked at this point.
This feels like an incomplete chain of responsibility. Because the DaemonProvider
allocates a separate directory for the image.tar
, which must exist until after the client.go
calls image.Read()
. This is partially because of the way v1.Image
lazy load feature behaves. Thus the relationships are broken during the Provide sequence.
IMHO - A better solution is to complete the chain of responsibility but having selectImageProvider()
allocated a rootTempDirGenerator
and pass it to the provider; which in turn passes the tempDirGenerator
to the image.NewImage()
, this completes the chain of responsibility.
If providers make a new generators, or directories, they are all passed to the image for cleanup after the image is no longer needed.
from stereoscope.
+1 on this - we noticed abnormal disk usage when utilizing syft and narrowed down the issue to this root cause as well.
The code to work around it is a bit clunky as users of our wrapper APIs expect that image.Cleanup()
is enough - but that leaves .tar files hanging around.
from stereoscope.
I think this is an interesting problem. There are a few options here:
- option 1: have the file.TempFileGenerator remember generators and call their cleanup method when the top level cleanup is called.... https://github.com/anchore/stereoscope/blob/main/pkg/file/temp_dir_generator.go#L16
- option 2: provide a generic cleanup array on the image which can have methods added by any provider.
- option 3: provide a Cleanup interface method for a provider - and have the provider call the cleanup. (this feels off since the client.go file is the one allocating the temp generator)
Personally I am a fan of option 1... but would like a recommendations from the team - I am happy to open a PR with a little guidance
from stereoscope.
I think this is an interesting problem. There are a few options here:
- option 1: have the file.TempFileGenerator remember generators and call their cleanup method when the top level cleanup is called.... https://github.com/anchore/stereoscope/blob/main/pkg/file/temp_dir_generator.go#L16
- option 2: provide a generic cleanup array on the image which can have methods added by any provider.
- option 3: provide a Cleanup interface method for a provider - and have the provider call the cleanup. (this feels off since the client.go file is the one allocating the temp generator)
Personally I am a fan of option 1... but would like a recommendations from the team - I am happy to open a PR with a little guidance
☺️
nm... I think I was wrong.
Based on a further research - only thing really needed is to call client.go Cleanup()
. which releases the root generator and all children generators....effectively cleaning up the directories.
The comment above cleanup is misleading and what leads to the leak of files I believe.
from stereoscope.
I've just run into this myself - was using img.Cleanup
per the basic example, only to find ~32gb of tmp files from stereoscope😅
Switching to stereoscope.Cleanup()
looks like it's resolved the problem - it would be good to know the backstory on the comment @slimdevl you've indicated, since it does seem like that should be called instead of (or maybe in addition to?) img.Cleanup
.
from stereoscope.
Related Issues (20)
- It should be possible to specify image platform when pulling from the registry
- Wrong source when ":" character in path
- SquashFS Iteration Fails when FIFO Present
- SquashFS iteration fails with "unexpected EOF" HOT 1
- Unable to read image using Docker daemon provider when architecture has variant
- Stereoscope pulls different images when using ``docker:`` vs ``registry:`` from multi-platform images
- Symlink not being recognized HOT 4
- Embed fs.FileInfo in Metadata Struct?
- Support for image indexes with multiple manifests HOT 4
- Odd error message when docker.socket in home dir is used
- Stereoscope cannot inspect images in Docker Desktop HOT 15
- Goroutine Leak in Long-Running Services with Stereoscope HOT 3
- Data Race in Image.Read
- Platform selection in docker client has unnecessary error HOT 2
- NewTarIndex race condition when run in a goroutine HOT 6
- Add support for containers-storage backend for container images
- Cycle during symlink resolution prevents syft/grype scans from enumerating vulnerabilities
- Add release process
- Concurrently analyzing containers can lead to race condition causing error HOT 9
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from stereoscope.