tonistiigi / fsutil

Incremental file directory sync tools in golang.

BENCH_FILE_SIZE=10000 ./bench.test --test.bench .
BenchmarkCopyWithTar10-4                	    2000	    995242 ns/op
BenchmarkCopyWithTar50-4                	     300	   4710021 ns/op
BenchmarkCopyWithTar200-4               	     100	  16627260 ns/op
BenchmarkCopyWithTar1000-4              	      20	  60031459 ns/op
BenchmarkCPA10-4                        	    1000	   1678367 ns/op
BenchmarkCPA50-4                        	     500	   3690306 ns/op
BenchmarkCPA200-4                       	     200	   9495066 ns/op
BenchmarkCPA1000-4                      	      50	  29769289 ns/op
BenchmarkDiffCopy10-4                   	    2000	    943889 ns/op
BenchmarkDiffCopy50-4                   	     500	   3285950 ns/op
BenchmarkDiffCopy200-4                  	     200	   8563792 ns/op
BenchmarkDiffCopy1000-4                 	      50	  29511340 ns/op
BenchmarkDiffCopyProto10-4              	    2000	    944615 ns/op
BenchmarkDiffCopyProto50-4              	     500	   3334940 ns/op
BenchmarkDiffCopyProto200-4             	     200	   9420038 ns/op
BenchmarkDiffCopyProto1000-4            	      50	  30632429 ns/op
BenchmarkIncrementalDiffCopy10-4        	    2000	    691993 ns/op
BenchmarkIncrementalDiffCopy50-4        	    1000	   1304253 ns/op
BenchmarkIncrementalDiffCopy200-4       	     500	   3306519 ns/op
BenchmarkIncrementalDiffCopy1000-4      	     200	  10211343 ns/op
BenchmarkIncrementalDiffCopy5000-4      	      20	  55194427 ns/op
BenchmarkIncrementalDiffCopy10000-4     	      20	  91759289 ns/op
BenchmarkIncrementalCopyWithTar10-4     	    2000	   1020258 ns/op
BenchmarkIncrementalCopyWithTar50-4     	     300	   5348786 ns/op
BenchmarkIncrementalCopyWithTar200-4    	     100	  19495000 ns/op
BenchmarkIncrementalCopyWithTar1000-4   	      20	  70338507 ns/op
BenchmarkIncrementalRsync10-4           	      30	  45215754 ns/op
BenchmarkIncrementalRsync50-4           	      30	  45837260 ns/op
BenchmarkIncrementalRsync200-4          	      30	  48780614 ns/op
BenchmarkIncrementalRsync1000-4         	      20	  54801892 ns/op
BenchmarkIncrementalRsync5000-4         	      20	  84782542 ns/op
BenchmarkIncrementalRsync10000-4        	      10	 103355108 ns/op
BenchmarkRsync10-4                      	      30	  46776470 ns/op
BenchmarkRsync50-4                      	      30	  48601555 ns/op
BenchmarkRsync200-4                     	      20	  59642691 ns/op
BenchmarkRsync1000-4                    	      20	 101343010 ns/op
BenchmarkGnuTar10-4                     	     500	   3171448 ns/op
BenchmarkGnuTar50-4                     	     300	   5030296 ns/op
BenchmarkGnuTar200-4                    	     100	  10464313 ns/op
BenchmarkGnuTar1000-4                   	      50	  30375257 ns/op

fsutil's Issues

`go get -u github.com/tonistiigi/fsutil` fails

When I run go get -u github.com/tonistiigi/fsutil, I get the following error:

# github.com/tonistiigi/fsutil
.../pkg/mod/github.com/tonistiigi/[email protected]/walker.go:120:20: undefined: fileutils.MatchInfo
.../pkg/mod/github.com/tonistiigi/[email protected]/walker.go:121:20: undefined: fileutils.MatchInfo
.../pkg/mod/github.com/tonistiigi/[email protected]/walker.go:176:31: undefined: fileutils.MatchInfo
.../pkg/mod/github.com/tonistiigi/[email protected]/walker.go:180:39: includeMatcher.MatchesUsingParentResults undefined (type *fileutils.PatternMatcher has no field or method MatchesUsingParentResults)
.../pkg/mod/github.com/tonistiigi/[email protected]/walker.go:211:31: undefined: fileutils.MatchInfo
.../pkg/mod/github.com/tonistiigi/[email protected]/walker.go:215:39: excludeMatcher.`MatchesUsingParentResults` undefined (type *fileutils.PatternMatcher has no field or method MatchesUsingParentResults)

It seems to be due to the dependency docker/docker. Even though the commit (87521affb077) of docker/docker in the current go.mod seems to include the code changes related to MatchInfo and MatchesUsingParentResults, it is somehow failing.

However, when I tried using the latest version of docker/docker (v20.10.3-0.20220524152814-18529568d3fb), it seems to work without issues.

This is causing problems when I am trying to build something else which has this as a dependency.

Debian packaging

Following up with #43

I'm packaging fsutil in preparing for the upcoming docker cli packaging.
However, I'm facing difficulties doing that due the test needs to be done as root.
There is a lengthy discussion in debian-go dlist at
https://lists.debian.org/debian-go/2020/03/msg00106.html

Do you have any suggestions how I can proceed?
If I need to change the test files to incorporate Debian packaging, would you accept my PRs?

thanks

missing pkg: github.com/tonistiigi/fsutil/util

github.com/tonistiigi/fsutil/util is referenced from CLIs but not commited to git

copyFileContent broken on zfs (in some cases)

See moby/moby#41866

Basically, need to add EINVAL on this line:

fsutil/copy/copy_linux.go

Line 92 in 0834f99

if (err != unix.ENOSYS && err != unix.EXDEV && err != unix.EPERM) || !first {

Also worth considering if any error should cause the fallback.

I haven't isolated the kernel version/zfs version that causes the issue. My kernel & zfs version are posted in the moby ticket linked above.

There's also an identical issue in containerd/continuity: containerd/continuity#176

undefined: proto.GoGoProtoPackageIsVersion3

Hi,

I'm trying to build this tools as a Debian package but am meeting with building problems. I have no idea, so here is the full build log:

dpkg-buildpackage: info: source package golang-github-tonistiigi-fsutil
dpkg-buildpackage: info: source version 0.0~git20200225.013a9fe-1
dpkg-buildpackage: info: source distribution unstable
dpkg-buildpackage: info: source changed by Tong Sun <[email protected]>
 dpkg-source -i --before-build .
dpkg-buildpackage: info: host architecture amd64
 debian/rules clean
dh clean --builddirectory=_build --buildsystem=golang --with=golang
   dh_auto_clean -O--builddirectory=_build -O--buildsystem=golang
   dh_autoreconf_clean -O--builddirectory=_build -O--buildsystem=golang
   dh_clean -O--builddirectory=_build -O--buildsystem=golang
 debian/rules binary
dh binary --builddirectory=_build --buildsystem=golang --with=golang
   dh_update_autotools_config -O--builddirectory=_build -O--buildsystem=golang
   dh_autoreconf -O--builddirectory=_build -O--buildsystem=golang
   dh_auto_configure -O--builddirectory=_build -O--buildsystem=golang
   dh_auto_build -O--builddirectory=_build -O--buildsystem=golang
        cd _build && go install -trimpath -v -p 2 github.com/tonistiigi/fsutil github.com/tonistiigi/fsutil/bench github.com/tonistiigi/fsutil/cmd/receive github.com/tonistiigi/fsutil/cmd/send github.com/tonistiigi/fsutil/cmd/walk github.com/tonistiigi/fsutil/copy github.com/tonistiigi/fsutil/types github.com/tonistiigi/fsutil/util
internal/cpu
runtime/internal/atomic
runtime/internal/sys
internal/bytealg
runtime/internal/math
internal/race
sync/atomic
runtime
unicode
unicode/utf8
math/bits
math
internal/testlog
runtime/cgo
encoding
unicode/utf16
crypto/internal/subtle
crypto/subtle
internal/reflectlite
sync
math/rand
errors
sort
io
strconv
bytes
reflect
internal/oserror
syscall
time
internal/poll
internal/fmtsort
internal/syscall/unix
strings
os
path
context
github.com/containerd/continuity/syscallx
fmt
path/filepath
io/ioutil
encoding/binary
golang.org/x/sys/unix
os/user
github.com/containerd/continuity/sysx
bufio
encoding/base64
archive/tar
encoding/json
log
regexp/syntax
github.com/sirupsen/logrus
text/scanner
regexp
hash
crypto
github.com/pkg/errors
github.com/gogo/protobuf/proto
github.com/docker/docker/pkg/fileutils
github.com/opencontainers/go-digest
github.com/gogo/protobuf/sortkeys
golang.org/x/sync/errgroup
crypto/cipher
crypto/aes
math/big
github.com/tonistiigi/fsutil/types
# github.com/tonistiigi/fsutil/types
src/github.com/tonistiigi/fsutil/types/stat.pb.go:27:11: undefined: proto.GoGoProtoPackageIsVersion3
src/github.com/tonistiigi/fsutil/types/wire.pb.go:27:11: undefined: proto.GoGoProtoPackageIsVersion3
encoding/hex
flag
crypto/rand
github.com/containerd/continuity/fs
github.com/tonistiigi/fsutil/copy
dh_auto_build: error: cd _build && go install -trimpath -v -p 2 github.com/tonistiigi/fsutil github.com/tonistiigi/fsutil/bench github.com/tonistiigi/fsutil/cmd/receive github.com/tonistiigi/fsutil/cmd/send github.com/tonistiigi/fsutil/cmd/walk github.com/tonistiigi/fsutil/copy github.com/tonistiigi/fsutil/types github.com/tonistiigi/fsutil/util returned exit code 2
make: *** [debian/rules:4: binary] Error 25
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2
debuild: fatal error at line 1182:

I.e., the build failed because of the undefined: proto.GoGoProtoPackageIsVersion3 error. I assume that there are some steps I'm missing in my build setup?
The Debian gbp repo can be found at
https://salsa.debian.org/suntong-guest/golang-github-tonistiigi-fsutil

Any help appreciated. thx.

freebsd ci broken

Looks like something happened to the vagrant setup. Maybe github removed it from the runners. https://github.com/tonistiigi/fsutil/runs/4461198951?check_suite_focus=true

@akhramov @crazy-max

Random Panic

I don't have a lot of info about reproducing this, but I just randomly get this panic when running the buildkit client. I'll try to look at this eventually but just throwing it here just in case this was super obvious what is failing.

github.com/tonistiigi/fsutil.Walk.func1({0xc0035a8990, 0x17}, {0x0, 0x0}, {0x2ae64e0, 0xc00791e5a0})
	/home/ubuntu/go/pkg/mod/github.com/tonistiigi/[email protected]/walker.go:137 +0xf92
path/filepath.walk({0xc0035bff24, 0x4}, {0x2b11468, 0xc007913450}, 0xc003ee1d48)
	/opt/hostedtoolcache/go/1.18.3/x64/src/path/filepath/path.go:450 +0x209
path/filepath.Walk({0xc0035bff24, 0x4}, 0xc002257d48)
	/opt/hostedtoolcache/go/1.18.3/x64/src/path/filepath/path.go:517 +0x6c
github.com/tonistiigi/fsutil.Walk({0x2b0a1b0, 0xc0035d5780}, {0xc001abee40, 0x4}, 0xc0035fef00, 0xc0035e0d38)
	/home/ubuntu/go/pkg/mod/github.com/tonistiigi/[email protected]/walker.go:109 +0x998
github.com/tonistiigi/fsutil.(*fs).Walk(0x1333be2?, {0x2b0a1b0?, 0xc0035d5780?}, 0x14?)
	/home/ubuntu/go/pkg/mod/github.com/tonistiigi/[email protected]/fs.go:36 +0x3d
github.com/tonistiigi/fsutil.(*sender).walk(0xc0012ca180, {0x2b0a1b0, 0xc0035d5780})
	/home/ubuntu/go/pkg/mod/github.com/tonistiigi/[email protected]/send.go:147 +0xa6
github.com/tonistiigi/fsutil.(*sender).run.func1()
	/home/ubuntu/go/pkg/mod/github.com/tonistiigi/[email protected]/send.go:60 +0x2e
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/ubuntu/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:74 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/ubuntu/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:71 +0xa5

Question: is it intended to be a standalone library, or is it part of Docker?

Hi,

while attempting to package Docker for Debian, I came across this library. I first tried to package it as a standalone library, the way it's usually done in Debian, and bump into two problems, mainly:

• the library has no release yet, so I can only package a random snapshot. Not a blocker, but of course a library is better packaged when it has releases.
• the library depends on Docker, so that makes for a circular dependency with Docker (or Moby if you prefer).

So I'm just wondering what's the intent here. Does this library aims to become standalone, or is it just a part of Docker codebase that lives somewhere else for some reasons?

EDIT: I've seen that this library is also used in moby/buildkit, which is also difficult to package for the same reasons.

Thanks!

Add TestCopyExistingDirDest

To guard against bug fixed by #33

clarify why copy.Copy is unsafe

The documentation currently states this:

fsutil/copy/copy.go

Lines 65 to 66 in 9e7a6df

	// Copy copies files using `cp -a` semantics.
	// Copy is likely unsafe to be used in non-containerized environments.

It's not clear what is the safety concern exactly.

TestCopySimple fails at head

is this expected? Running on Linux:

--- FAIL: TestCopySimple (0.01s)
    receive_test.go:251: 
        	Error Trace:	receive_test.go:251
        	Error:      	Not equal: 
        	            	expected: "sha256:99b6ef96ee0572b5b3a4eb28f00b715d820bfd73836e59cc1565e241f4d1bb2f"
        	            	actual  : "sha256:741b190155bf4387b43922020b6c60e809acdb4f36d08d277d816abed51b94c0"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,2 +1,2 @@
        	            	-(digest.Digest) (len=71) "sha256:99b6ef96ee0572b5b3a4eb28f00b715d820bfd73836e59cc1565e241f4d1bb2f"
        	            	+(digest.Digest) (len=71) "sha256:741b190155bf4387b43922020b6c60e809acdb4f36d08d277d816abed51b94c0"
        	            	 
        	Test:       	TestCopySimple
    receive_test.go:255: 
        	Error Trace:	receive_test.go:255
        	Error:      	Not equal: 
        	            	expected: "sha256:dd2529f7749ba45ea55de3b2e10086d6494cc45a94e57650c2882a6a14b4ff32"
        	            	actual  : "sha256:104b73735e1bd9ee5beab668d595f48630bf34e9ec3d201c28a8b820e6e52aff"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,2 +1,2 @@
        	            	-(digest.Digest) (len=71) "sha256:dd2529f7749ba45ea55de3b2e10086d6494cc45a94e57650c2882a6a14b4ff32"
        	            	+(digest.Digest) (len=71) "sha256:104b73735e1bd9ee5beab668d595f48630bf34e9ec3d201c28a8b820e6e52aff"
        	            	 
        	Test:       	TestCopySimple

Suggestion: move to moby/buildkit repo

Revendoring tonistiigi/fsutil into moby/buildkit is sometimes a mess: moby/buildkit#1379 (comment)

Any chance to move the repo into moby/buildkit ?

Add Windows CI workflow

The tests on Windows for a number of packages seem to fail. A lot of them are a simple fix of just doing a filepath.FromSlash() on expected paths, but some are legitimately failing. Looking at the tested code that fails. Looks like most of it is in regards to following symlinks or walking paths.

As this code is being used in buildkit in https://github.com/moby/buildkit/blob/master/cache/util/fsutil.go#L99, this will have to be fixed before we can safely add Windows support to builkit.

Looking into it now, and will post back an update as soon as I have something.

optimize memory allocation in receiver

https://github.com/tonistiigi/fsutil/pull/139/files#diff-d1445ba8bd61e8e9f66950832df5b1d76f222e7f1f38700a30b728c214da04ccR41 added map where all the directory paths+timestamps are written in. Because all the items are sorted, this full map is not needed only timestamps for the parent dirs of current path are needed. So instead of a full map, this can be achieved with a stack that adds item when moving to subdirectory and pops when the current directory has been processed.

TestCopySimple fails

# go get github.com/tonistiigi/fsutil
# go test github.com/tonistiigi/fsutil -v -run ''


=== RUN   TestCopySimple
--- FAIL: TestCopySimple (0.01s)
	assertions.go:238: 
			Error Trace:	receive_test.go:80
			Error:      	Not equal: 
			            	expected: "sha256:2c6454be696ff984b27888e4757cfb5eb65c1a8f0d49c017596e3feefaa0c84e"
			            	actual  : "sha256:99b6ef96ee0572b5b3a4eb28f00b715d820bfd73836e59cc1565e241f4d1bb2f"
	assertions.go:238: 
			Error Trace:	receive_test.go:84
			Error:      	Not equal: 
			            	expected: "sha256:2fcd2d8fb162dbdfb2ecfe93724dc0f4e683ebff2d9f15485136958ef99ef8b7"
			            	actual  : "sha256:dd2529f7749ba45ea55de3b2e10086d6494cc45a94e57650c2882a6a14b4ff32"
	assertions.go:238: 
			Error Trace:	receive_test.go:88
			Error:      	Not equal: 
			            	expected: "sha256:51086b730b712a6c3a11651083e4eb00a095b3ef60174fb8934542e8cebcecca"
			            	actual  : "sha256:eca07e8f2d09bd574ea2496312e6de1685ef15b8e6a49a534ed9e722bcac8adc"
	assertions.go:238: 
			Error Trace:	receive_test.go:137
			Error:      	Not equal: 
			            	expected: "sha256:51086b730b712a6c3a11651083e4eb00a095b3ef60174fb8934542e8cebcecca"
			            	actual  : "sha256:eca07e8f2d09bd574ea2496312e6de1685ef15b8e6a49a534ed9e722bcac8adc"
	assertions.go:238: 
			Error Trace:	receive_test.go:141
			Error:      	Not equal: 
			            	expected: "sha256:f2b015030e257b821d76681d822e23af06de1621e3b1a9168fbc9d22e83ed345"
			            	actual  : "sha256:cd14a931fc2e123ded338093f2864b173eecdee578bba6ec24d0724272326c3a"

Test with both go 1.7 and go 1.10, in case it matters.

flaky test: TestInvalidExcludePatterns

#30 (comment)

--- FAIL: TestInvalidExcludePatterns (0.00s)
        Error Trace:    receive_test.go:58
	Error:		"invalid excludepaths [!]: illegal exclusion pattern: "!"" does not contain "error from sender:"

cc @tiborvass

validator catches false positives in windows for subdirs

$ docker build --stream -t build -f .\Dockerfile.windows .
Error response from daemon: failed to copy to C:\ProgramData\docker\builder\53c9791595c750741e0cb3deff16abe74feb5d1da8932bc3ea03d1d03c385e17: invalid unclean path .github/CODEOWNERS

seems the issue is not using path.Clean in https://github.com/tonistiigi/fsutil/blob/master/validator.go#L29 but should check all other places expect normalized paths as well.

tests: add test cases to cover Windows/Unix path conversions

Follow up from #197 (comment)

./assign @profnandaa