Coder Social home page Coder Social logo

otiai10 / copy Goto Github PK

View Code? Open in Web Editor NEW
705.0 9.0 114.0 224 KB

Go copy directory recursively

Home Page: https://pkg.go.dev/github.com/otiai10/copy

License: MIT License

Go 87.68% Dockerfile 2.82% Shell 9.50%
golang copy files directory recursive go folder folders directories

copy's Introduction

copy

Go Reference Actions Status codecov License: MIT FOSSA Status CodeQL Go Report Card GitHub tag (latest SemVer) Docker Test Vagrant Test GopherJS Go WASM

copy copies directories recursively.

Example Usage

package main

import (
	"fmt"
	cp "github.com/otiai10/copy"
)

func main() {
	err := cp.Copy("your/src", "your/dest")
	fmt.Println(err) // nil
}

Advanced Usage

// Options specifies optional actions on copying.
type Options struct {

	// OnSymlink can specify what to do on symlink
	OnSymlink func(src string) SymlinkAction

	// OnDirExists can specify what to do when there is a directory already existing in destination.
	OnDirExists func(src, dest string) DirExistsAction

	// OnError can let users decide how to handle errors (e.g., you can suppress specific error).
	OnError func(src, dest, string, err error) error

	// Skip can specify which files should be skipped
	Skip func(srcinfo os.FileInfo, src, dest string) (bool, error)

	// RenameDestination can rename destination.
	// If not set, nil, it does nothing.
	RenameDestination func(src, dest string) (string, error)

	// PermissionControl can control permission of
	// every entry.
	// When you want to add permission 0222, do like
	//
	//		PermissionControl = AddPermission(0222)
	//
	// or if you even don't want to touch permission,
	//
	//		PermissionControl = DoNothing
	//
	// By default, PermissionControl = PreservePermission
	PermissionControl PermissionControlFunc

	// Sync file after copy.
	// Useful in case when file must be on the disk
	// (in case crash happens, for example),
	// at the expense of some performance penalty
	Sync bool

	// Preserve the atime and the mtime of the entries
	// On linux we can preserve only up to 1 millisecond accuracy
	PreserveTimes bool

	// Preserve the uid and the gid of all entries.
	PreserveOwner bool

	// The byte size of the buffer to use for copying files.
	// If zero, the internal default buffer of 32KB is used.
	// See https://golang.org/pkg/io/#CopyBuffer for more information.
	CopyBufferSize uint

	// If you want to add some limitation on reading src file,
	// you can wrap the src and provide new reader,
	// such as `RateLimitReader` in the test case.
	WrapReader func(src io.Reader) io.Reader

	// If given, copy.Copy refers to this fs.FS instead of the OS filesystem.
	// e.g., You can use embed.FS to copy files from embedded filesystem.
	FS fs.FS

	// NumOfWorkers represents the number of workers used for
	// concurrent copying contents of directories.
	// If 0 or 1, it does not use goroutine for copying directories.
	// Please refer to https://pkg.go.dev/golang.org/x/sync/semaphore for more details.
	NumOfWorkers int64

	// PreferConcurrent is a function to determine whether or not
	// to use goroutine for copying contents of directories.
	// If PreferConcurrent is nil, which is default, it does concurrent
	// copying for all directories.
	// If NumOfWorkers is 0 or 1, this function will be ignored.
	PreferConcurrent func(srcdir, destdir string) (bool, error)
}
// For example...
opt := Options{
	Skip: func(info os.FileInfo, src, dest string) (bool, error) {
		return strings.HasSuffix(src, ".git"), nil
	},
}
err := Copy("your/directory", "your/directory.copy", opt)

Issues

License

FOSSA Status

copy's People

Contributors

alexandear avatar andrew-farries avatar antonboom avatar apprehensions avatar calebq42 avatar dependabot[bot] avatar drakkan avatar dreamjz avatar dvaumoron avatar eclipseo avatar edigaryev avatar fako1024 avatar fkorotkov avatar flimzy avatar fossabot avatar julplee avatar michalpristas avatar mudler avatar ncopa avatar otiai10 avatar puengel avatar samuel-phan avatar sergeyhush avatar siscia avatar vikramdurai avatar villaquiranm avatar xhit avatar yasushi-saito avatar ycombinator avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

copy's Issues

copy.Copy hangs with named pipes

It seems that copy.Copy is not supporting named pipes, at the moment it tries to copy it, resulting in the process to hang.

Would you be interested in accepting a PR to reproduce the named pipe in the given destination ?

Adding Replace option?

Dear @otiai10,

What do you think of adding copy with replace option?

I noticed that if we copy one directory to another, it preserves old files as the way they are.

I think, it would be a nice feature to have, if you need to replace the entire destination directory by the source one.

By the way, nice package!

add a license

to help folks use this it would be good if you could add a license to the repo please.

Handling owner permissions

First of all, thanks for the great lib ;)

It would be really nice if copy would handle (with an option, maybe) the owner permission of the resulting file - especially when it's recreating the target (symlink, or namedpipe, see also #47)

Cheers

Can we support Update Opt, like --update in `cp`

HiHi, copy is every easy to use, but have a feature to request.

I have a lot of files need to copy from one folder to anther folder, the source folder have several files changed between dest folder.

It will cause poor performance when we just want update several files to dest folder.

So I want copy to support Update Opt like cp in Linux:

-u, --update
              copy only when the SOURCE file is newer than the destination file or when the destination file is missing

opt.Untouchable + opt.PreserveTimes error copying subdirectory

This is very minor issue. But interesting.

Here is error log from my code, which I provide for clarity:

Final: main.go#DoitRunApps(): error App.RunApp(), error = AppLogicCustomStartAsOfDt.RunApp(): (15) Normal Daily Run utility.go#Otiai10CopyMerge() error, in=data, out=bkp/d20210305, error = chtimes bkp/d20210305/downloads: no such file or directory

I decided to use OnDirExists.Untouchable for first test of certain logic in my code, as opposed to
Merge. Also using opt.PreserveTimes = true. Copying directory data=/ which contains data/downloads, data/in and data/out. Destination is bkp/d20210305 where subdirectory
d20210305 must be created as part of the copy -- which it was. The error occurred in
subdirectory downloads/ which was not created.

So...it appears to me that switchboard() code is not liking the fact that destination
bkp/d20210305/downloads failed to exist:

if opt.PreserveTimes { spec := getTimeSpec(info) if err := os.Chtimes(dest, spec.Atime, spec.Mtime); err != nil { return err } }

Thus, the apparent logical difficulty: if directory does not exist then why preserve times?

I will workaround by simply not using PreserveTimes=true in this instance. I suppose
just adding a documentation note about the incompatiblity of these options would
suffice. Or, perhaps, add a line of code or two.

add an option to deep-copy symlinks

Can we have an option to copy symlinks? I'm imagining something like:

Opts struct {
  // Called if a symlink is found. If this function returns false, 
  // Copy copies the symlink itself. Else, Copy() follows the link.
  // If FollowSymlink=nil, Copy never follows symlinks.
  FollowSymlink func(path string) bool
}

func Copy(src, dest string, opt... Opts) error;

I can create a pull request if it looks good.

Copies recursively if source is current directory.

If the source is current directory and we specify a destination to be inside current working directory. it goes on a loop copying the destination folder recursively

cp.Copy(".", "dest")

Error:

open dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/dest/.idea/inspectionProfiles/Project_Default.xml: file name too long

Does not build on solaris.

Hi.

I receive the following in goreleaser on solaris:

⨯ release failed after 186.87s error=failed to build for solaris_amd64: exit status 2: # github.com/otiai10/copy
vendor/github.com/otiai10/copy/copy_namedpipes.go:16:9: undefined: syscall.Mkfifo
vendor/github.com/otiai10/copy/test_setup.go:16:2: undefined: syscall.Mkfifo

copy.Deep does not follow multiple levels of symbolic links

Run the following bash script:

#!/bin/bash

rm -rf src dst
mkdir -p src dst
cd src
ln -s adir/afile a
ln -s bdir adir
mkdir bdir
touch bdir/a

to create the following structure:

❯ tree src
src
├── a -> adir/afile
├── adir -> bdir
└── bdir
    └── a

2 directories, 2 files

Then use the following program:

❯ cat main.go
package main

import (
        "log"

        "github.com/otiai10/copy"
)

func main() {
        opt := copy.Options{OnSymlink: func(string) copy.SymlinkAction { return copy.Deep }}
        if err := copy.Copy("src/a", "dst/b", opt); err != nil {
                log.Fatal(err)
        }

}

finally, run it:

❯ go run main.go
2020/12/07 18:29:30 lstat adir/afile: no such file or directory
exit status 1

another recursive copy

Copy("test/data.copy", "test/data.copy/alltestdata")

Note: when I found the alltestdata and deleted it, MacOS refused to empty the trash
because "pathname too long". I tried sudo rm etc but "invalid operation". So the
solution if cannot empty trash: drag items out of trash to, say, Downloads and
then just "rm" them.

Symlinks don't resolve relative paths properly

When copying a directory to another directory that is at a different path level, symlinks that resolve to relative paths fail to be copied as the source path of the symlink is not resolved to the absolute path.

Example:

Source folder:
src/
a/
someFile
b/
someFile (symlink to '../a/someFile')

Copy("src", "dest")

will work fine, however:

`Copy("src", "../dest") fails with:

readlink "dest/a/someFile": no such file or directory

What is happening as when using os.ReadLink, e.g.:

src, _ := os.Readlink("src/b/someFile")

src is return as "../a/someFile"

but it should really be expanded to the full path of "src/a/someFile", e.g. using filepath.Abs(src)

Add support for embed directories

It would be super useful if this package could copy embed directories.

For example:

Given the following fs tree:

/
  /assets
    /foo
       foo.txt
       bar.txt
    hello.txt
    goodbye.txt
  main.go

It could use the embed FS to use the copy function.

//go:embed assets
var assets embed.FS

func foo() error {
  file, err := assets.Open("assets")
  if err != nil {
    return err
  }

  err := Copy(file, "destination")
  if err != nil {
    return err
  }
  return nil
}

Or maybe use the Option structure

opt := Options{
	FS: assets,
}
err := Copy("assets/foo", "new/destination", opt)

parsing go.mod: unexpected module path "copy"

When attempting to use go modules with this package I get:

go: github.com/otiai10/[email protected]: parsing go.mod: unexpected module path "copy"

I suspect it has to do with the go.mod module being declared as copy rather than the full package name:

Ex.

module copy

require (
	bou.ke/monkey v1.0.1 // indirect
	github.com/otiai10/mint v1.2.1
)

rather than:

module github.com/otiai10/copy

require (
	bou.ke/monkey v1.0.1 // indirect
	github.com/otiai10/mint v1.2.1
)

v2 wishlist

We want

  • Skip to have os.FileInfo #40
  • multiple symlink to be followed #32
  • owner permission to be able to be set #48

Deep symlink copy doesn't work with relative `../foo` target

Deep symlink copy will fail in this situation:

test
├── data
│   ├── case00
│   │   └── README.md
│   ├── case01
│   │   └── README.md
│   ├── case02
│   │   └── README.md
│   ├── case03
│   │   ├── README.md
│   │   ├── case01 -> test/data/case01
│   │   └── relative_case01 -> ../case01

Though it's entirely acceptable to have a symlink working only at a certain current working dir, most of the time, we have symlinks:

  • either to an absolute path
  • or relative to itself (eg ../something). 🐛 This case doesn't work properly.

Copy between 2 different file systems

I see the option to specify a file system (i.e. embed) for copying. But that seems to assume you are copying within the same file system. Is there a way to copy between two different file systems? As in copy from an embedded file system to the host file system?

Use of bou.ke/monkey -- unneeded?

I've seen that there's a reference to bou.ke/monkey in the go.mod but I'm unsure where it's being used.

The license for monkey pretty much disallows anyone from using it, which in term means that no one can use your library either. See here and here for more information.

Is it needed, or is can we just remove it and keep this library as MIT? 😄

[Bug] callback func onDirExist does not effect

[occurrence]

when testing the conflict with the same directory, the onDirExists callback function doest not effect, code as following:

func TestRename(t *testing.T) {
	targetDir := "./test/dir1"
	newDir := "./test/dir2/dir1"
	err := cp.Copy(targetDir, newDir, cp.Options{
		OnDirExists: func(src, dest string) cp.DirExistsAction {
			fmt.Printf("Directory Exists, do nothing.\n")
			return cp.Untouchable
		},
		PreserveTimes: true,
		PreserveOwner: true,
	})
	if err != nil {
		fmt.Printf("Failed to rename: %s \n", err.Error())
		return
	}
	fmt.Printf("Success to rename \n")
}

working directory tree is as following:
image

[personal opinio]

I 'm confused about the conditions under which the onDirExist callback function is executed.

func onDirExists(opt Options, srcdir, destdir string) (bool, error) {
	_, err := os.Stat(destdir)
	if err == nil && opt.OnDirExists != nil && destdir != opt.intent.dest {
		switch opt.OnDirExists(srcdir, destdir) {
		case Replace:
			if err := os.RemoveAll(destdir); err != nil {
				return false, err
			}
		case Untouchable:
			return true, nil
		} // case "Merge" is default behaviour. Go through.
	} else if err != nil && !os.IsNotExist(err) {
		return true, err // Unwelcome error type...!
	}
	return false, nil
}

we know that variable destdir is always equal to opt.intent.dest, if we use default options as following:

// part of default options
func getDefaultOptions(src, dest string) Options {
	return Options{
		// ignore the extra code .....
		intent:            intent{src, dest, nil, nil},
	}
}


// part of merge options 
func assureOptions(src, dest string, opts ...Options) Options {
	defopt := getDefaultOptions(src, dest)
	// ignore the extra code .....
	opts[0].intent.src = defopt.intent.src
	opts[0].intent.dest = defopt.intent.dest
	return opts[0]
}

so, how is it possible that the switch-case code of onDirExists callback func executes.

Symbolic links are not handled correctly

Symbolic links need to be handled separately from directories and regular files.

Example Failure

  • myDir
  • mySymlink → myDir

Calling Copy("mySymlink", "mySymlink2") fails because the library tries to read the file like a regular file, which fails because the symbolic link is pointing to a directory.

Fix

The fix is to check whether a file is a symbolic link and if so, call a new function that simply copies the link to the destination.

Add config to rename file

I have several files embedded (waiting for the PR for the support)
When copying them it would be nice if I can edit the name.
Would it be possible to add a function in the configuration to do this?
What do you think? @otiai10

Adding Replace option

Dear @otiai10,

What do you think of adding copy with replace option (WithReplace, true or false)?

I noticed that if we copy one directory to another, it preserves old files as the way they are.

I think, it would be a nice feature to have, if you need to replace the entire destination directory by the source one.

By the way, nice package!

Option to preserve special files

Similar to rsync --specials an option to handle special files as special is required: Currently the code tries to always copy the content of such files, e.g. /dev/console, pipes, sockets, … instead of copying it as a special files.
This is causing kaniko to stall when building base images: GoogleContainerTools/kaniko#960

Question: backwards compatibility 1.7.0 to 1.9.0+?

Version 1.9.0 broke backwards compatibility of the library, meaning other projects using it stopped compiling if someone upgraded the github.com/otiai10/copy package, either upgrading "by accident", or if they include a different package that has a minimum version dependency on github.com/otiai10/copy that is v1.9.0 or newer.

One tool that relies on github.com/otiai10/copy is go-licenses, which stopped compiling after I inadvertently upgraded the copy dependency (I am only a user of the tool).

I am mostly posting this for posterity, and to bring the issue up since I couldn't see anything about it in the issue tracker.

One way to restore backwards compatibility I think, could be to restore the copy function's previous skip type, and add a new function that uses the Options-pattern (option 3) to be able to extend the function with new options in the future without breaking backwards compatibility. It is also possible to mark the old copy function as deprecated

This will of course break users of v1.9.0 and v1.11.0, but assuming those users are fewer than users of the older versions might be reasonable, and their software is probably earlier in the development/in active development meaning they would be relatively quick in fixing the issue.

"no such file or directory" when copying large amounts of data

I'm using this package in a tool for backing up Docker volumes: https://github.com/offen/docker-volume-backup

Users that do not want to stop their containers while taking a backup can opt in to copying their data to a temporary location before creating the tar archive so that creating the archive does not fail in case data is being written to a file while it's being backed up. To perform this copy, package copy is used (thanks for making it public, much appreciated).

This seemed to work well in tests as well as the real world, however recently an issue was raised where copy would fail with the following error when backing up the data volume for a Prometheus container:

open /backup/prometheus_data/01FSM8TPFEXQ0QC28H11PMQZ0R: no such file or directory

The dataset that is being copied seems to be a. very large and b. pretty volatile which has me thinking this file might actually have been deleted/moved before copy finds the resources to actually copy it. This is the downstream issue: offen/docker-volume-backup#49

Is this issue somehow known? Is there a way to fix it by configuring copy differently?

This is the part where I use copy in code and also where the above error is being returned:

if err := copy.Copy(s.c.BackupSources, backupSources, copy.Options{
	PreserveTimes: true,
	PreserveOwner: true,
}); err != nil {
	return fmt.Errorf("takeBackup: error creating snapshot: %w", err)
}

Remove symbolic link from test case 03

Hi,

There is a symbolic link from testdata/case03/case01 to ./testdata/case01. When vendoring the package, the dependency manager vendors the link but not the case01 directory, breaking the link.

I understand that you need the link for your tests, but would it be possible to generate it during your tests instead? The broken link causes problems if one tries to copy it or in Windows OS.

Thanks

cannot copy to kernel dir

fmt.Println(src,dst)
	opt := copy.Options{
		Skip: func(src string) (bool, error) {
			return strings.HasSuffix(src, ".ext"), nil
		},
	}
	if err := copy.Copy(
		src,
		dst,
		opt,
		);err != nil {
		fmt.Println("[-] Cannot copy files to kernel dir:",err)
		return err
	}

output:

/path/to/src/ /lib/modules/4.11.0-generic/build/drivers/tc/ [-] Cannot copy files to kernel dir: stat /lib/modules/4.11.0-generic/build/drivers/tc/: invalid argument

dir symlink recursive loop

With Deep symlink copying in use, when a directory containing a symlink
is copied and the symlink points directly to the directory or to
another directory -- which is subsequently copied and contains
a symlink to the original directory, a loop ensues which is terminated in
os.Create with a "file name too long error" ...

I understand that there may be other issues copying symlinks,
but

https://github.com/enthor/copy/tree/enthor-patch-1

fixes this issue. (I believe...)

I included a test case -- search "case03b" in all_test.go.
To see the error output remove the "Not()." in
the following lines (temporarily):

// stop the recursive loop: opt = Options{OnSymlink: func(string) SymlinkAction { return Deep }} err = Copy("test/data/case03b", "test/data.copy/case03b.deep", opt) Expect(t, err).Not().ToBe(nil)

Upstream change in GopherJS (Go 1.17->1.18) breaks pipeline

It seems that a recent update on github.com/gopherjs/gopherjs (Switch to Go 1.18: gopherjs/gopherjs@f754ef8) breaks the GopherJS build pipeline (seemingly because it uses Go 1.17), e.g. on this branch (PR #83):
https://github.com/otiai10/copy/runs/7760149468?check_suite_focus=true

# github.com/gopherjs/gopherjs/compiler
Error: compiler/package.go:403:14: fun.Type.TypeParams undefined (type *ast.FuncType has no field or method TypeParams)
Error: compiler/package.go:405:35: fun.Type.TypeParams undefined (type *ast.FuncType has no field or method TypeParams)
Error: compiler/package.go:492:55: named.TypeParams undefined (type *types.Named has no field or method TypeParams)
note: module requires Go 1.18
Error: Process completed with exit code 2.

I assume this is caused by the pipeline using master while at the same time specifying Go 1.17 for the deployment:

# This package only works fully with GopherJS as of 2c06401a, which is not yet
# released, thus we're pulling in `master`. Once GopherJS 1.18 is released with
# this support, we can depend on a proper tag.
name: GopherJS
...
  build:
    name: Build
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
	os: [ubuntu-latest]
        go: [1.17]

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.