mistifyio / go-zfs Goto Github PK
View Code? Open in Web Editor NEWGo wrappers for ZFS commands
License: Apache License 2.0
Go wrappers for ZFS commands
License: Apache License 2.0
Hi,
This issue is in conjunction with #39.
In an effort to come up with a PR for Solaris support I investigated performance on using zfs list instead of zfs get and came up with the following numbers on Linux.
In short zfs list is quicker, especially when retrieving a specific set of properties.
I wrote a small timer function using the go "time" package that would time zfs Init().
On a Ubuntu system that had 40 containers on it: (As the number of containers on the system increases the time difference also increases drastically)
Without my changes daemon startup showed ZFS init took:
root:~/docker# ./docker-1.11.0-dev-baseline daemon --storage-driver=zfs
ZFS driver init took 716.156429ms time
INFO[0000] Graph migration to content-addressability took 0.00 seconds
INFO[0000] Firewalld running: false
INFO[0001] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address
WARN[0001] Your kernel does not support swap memory limit.
INFO[0001] Loading containers: start.
INFO[0002] Loading containers: done.
INFO[0002] Daemon has completed initialization
INFO[0002] Docker daemon commit=d687023-unsupported execdriver=native-0.2 graphdriver=zfs version=1.11.0-dev
INFO[0002] API listen on /var/run/docker.sock
With my changes:
root@:~/docker# bundles/1.11.0-dev/binary/docker-1.11.0-dev daemon --storage-driver=zfs
ZFS driver init took 282.353825ms time
INFO[0000] Graph migration to content-addressability took 0.00 seconds
INFO[0000] Firewalld running: false
INFO[0000] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address
WARN[0001] Your kernel does not support swap memory limit.
INFO[0001] Loading containers: start.
INFO[0001] Loading containers: done.
INFO[0001] Daemon has completed initialization
INFO[0001] Docker daemon commit=d687023-unsupported execdriver=native-0.2 graphdriver=zfs version=1.11.0-dev
INFO[0001] API listen on /var/run/docker.sock
Also zfs_test,go does a good job of running all the ZFS commands supported and in all we saved almost 6s on the test run with the changes:
root@:~/vendor/src/github.com/mistifyio/go-zfs# go test
PASS
ok github.com/mistifyio/go-zfs 20.738s
root@:~/vendor/src/github.com/amitkris/go-zfs# go test
PASS
ok github.com/amitkris/go-zfs 14.677s
Container startup time is still comparable:
root@:~/docker# time bundles/1.11.0-dev/binary/docker-1.11.0-dev run -i ubuntu /bin/true
real 0m2.368s
user 0m0.104s
sys 0m0.008s
root@:~/docker# time ./docker-1.11.0-dev-baseline run -ti ubuntu /bin/true
real 0m2.039s
user 0m0.076s
sys 0m0.020s
CHANGES made:
Hi,
When I try to test your porject it fails with
--- FAIL: TestDatasets (0.00s)
zfs_test.go:74: unexpected error: exec: "zpool": executable file not found in $PATH: "zpool zpool create test /tmp/zfs-911153758 /tmp/zfs-579910181 /tmp/zfs-122527040" =>
--- FAIL: TestDatasetGetProperty (0.00s)
zfs_test.go:74: unexpected error: exec: "zpool": executable file not found in $PATH: "zpool zpool create test /tmp/zfs-352642975 /tmp/zfs-305956978 /tmp/zfs-140087081" =>
--- FAIL: TestSnapshots (0.00s)
zfs_test.go:74: unexpected error: exec: "zpool": executable file not found in $PATH: "zpool zpool create test /tmp/zfs-392938612 /tmp/zfs-191840067 /tmp/zfs-120972230" =>
--- FAIL: TestFilesystems (0.00s)
zfs_test.go:74: unexpected error: exec: "zpool": executable file not found in $PATH: "zpool zpool create test /tmp/zfs-363306605 /tmp/zfs-084848360 /tmp/zfs-820738855" =>
--- FAIL: TestCreateFilesystemWithProperties (0.00s)
zfs_test.go:74: unexpected error: exec: "zpool": executable file not found in $PATH: "zpool zpool create test /tmp/zfs-288479322 /tmp/zfs-461731313 /tmp/zfs-613427356" =>
--- FAIL: TestVolumes (0.00s)
zfs_test.go:74: unexpected error: exec: "zpool": executable file not found in $PATH: "zpool zpool create test /tmp/zfs-507569483 /tmp/zfs-925299246 /tmp/zfs-713572789" =>
--- FAIL: TestSnapshot (0.00s)
zfs_test.go:74: unexpected error: exec: "zpool": executable file not found in $PATH: "zpool zpool create test /tmp/zfs-108708752 /tmp/zfs-225214383 /tmp/zfs-188771138" =>
--- FAIL: TestClone (0.00s)
zfs_test.go:74: unexpected error: exec: "zpool": executable file not found in $PATH: "zpool zpool create test /tmp/zfs-074942905 /tmp/zfs-070772164 /tmp/zfs-521692243" =>
--- FAIL: TestSendSnapshot (0.00s)
zfs_test.go:74: unexpected error: exec: "zpool": executable file not found in $PATH: "zpool zpool create test /tmp/zfs-795910550 /tmp/zfs-294920701 /tmp/zfs-851205944" =>
--- FAIL: TestChildren (0.00s)
zfs_test.go:74: unexpected error: exec: "zpool": executable file not found in $PATH: "zpool zpool create test /tmp/zfs-919999799 /tmp/zfs-514381098 /tmp/zfs-689129601" =>
--- FAIL: TestListZpool (0.00s)
zfs_test.go:74: unexpected error: exec: "zpool": executable file not found in $PATH: "zpool zpool create test /tmp/zfs-367396844 /tmp/zfs-327222363 /tmp/zfs-453601790" =>
--- FAIL: TestRollback (0.00s)
zfs_test.go:74: unexpected error: exec: "zpool": executable file not found in $PATH: "zpool zpool create test /tmp/zfs-592104261 /tmp/zfs-918564832 /tmp/zfs-517285311" =>
--- FAIL: TestDiff (0.00s)
FAIL
exit status 1
FAIL github.com/mistifyio/go-zfs 0.005s
Thank you for your help
zfsonlinux changed its format and illumos/omnios is broken also
vagrant@omnios-vagrant:/export/home/vagrant/src/github.com/mistifyio/go-zfs$ sudo ./go-zfs.test -test.v
=== RUN TestError
--- PASS: TestError (0.00s)
=== RUN TestDatasets
--- PASS: TestDatasets (0.13s)
=== RUN TestSnapshots
--- PASS: TestSnapshots (0.13s)
=== RUN TestFilesystems
--- PASS: TestFilesystems (0.18s)
=== RUN TestCreateFilesystemWithProperties
--- PASS: TestCreateFilesystemWithProperties (0.15s)
=== RUN TestVolumes
--- PASS: TestVolumes (1.15s)
=== RUN TestSnapshot
--- PASS: TestSnapshot (0.17s)
=== RUN TestClone
--- PASS: TestClone (0.19s)
=== RUN TestSendSnapshot
--- PASS: TestSendSnapshot (0.17s)
=== RUN TestChildren
--- PASS: TestChildren (0.20s)
=== RUN TestListZpool
--- PASS: TestListZpool (0.12s)
=== RUN TestRollback
--- PASS: TestRollback (0.24s)
=== RUN TestDiff
zfs_test.go:361:
exp: "/test/origin/i โค unicode"
got: "/test/origin/i \xff77777742\xff77777635\xff77777644 unicode"
--- FAIL: TestDiff (0.10s)
FAIL
In some cases zpool reterns '-' in 'fragmentation' field
$ sudo zpool get -p name,health,dedupratio,fragmentation rpool
NAME PROPERTY VALUE SOURCE
rpool name rpool -
rpool health ONLINE -
rpool dedupratio 1.00x -
rpool fragmentation - -
parseLine doesn't check if there is trainling '%' in the string before removing it, and this leads to empty string passed to setUint
.
This leads to broken output of docker info
$ docker info
Storage Driver: zfs
Zpool: error while getting pool information strconv.ParseUint: parsing "": invalid syntax
Zpool Health: not available
#45 added fields, we should have some tests in place
Similar to zpool states, string constants would be less fragile and error-prone for various dataset types.
http://play.golang.org/p/tAHoCFRN5F
In addition, these constants could be used at several places within the existing library:
https://github.com/mistifyio/go-zfs/blob/master/zfs.go#L64-L66
https://github.com/mistifyio/go-zfs/blob/master/zfs.go#L93-L95
https://github.com/mistifyio/go-zfs/blob/master/zfs.go#L189-L191
Hello,
I'd like to be able to create sparse volumes using this library. To do that with zfs
command one need to specify -s
flag so a function like this would work (it's just a copy of CreateVolume
with additional flag in args[2]
):
// CreateVolume creates a new ZFS volume with the specified name, size, and
// properties.
// A full list of available ZFS properties may be found here:
// https://www.freebsd.org/cgi/man.cgi?zfs(8).
func CreateSparseVolume(name string, size uint64, properties map[string]string) (*Dataset, error) {
args := make([]string, 4, 5)
args[0] = "create"
args[1] = "-p"
args[2] = "-sV"
args[3] = strconv.FormatUint(size, 10)
if properties != nil {
args = append(args, propsSlice(properties)...)
}
args = append(args, name)
_, err := zfs(args...)
if err != nil {
return nil, err
}
return GetDataset(name)
}
At time the zfs commands stuck for a long time. Is there a way to kill the commands after a specified time.
ps -ef | grep 'zfs list' gives me a long list of commands.
This could allow for easier checks of common zpool states, and for more specific logic in the case of a pool which is degraded, offline, etc.
http://play.golang.org/p/_bSlCjlTWZ
Perhaps the zpool.Healthy()
method is not entirely necessary, but it's a lot nicer than doing if z.Health == "ONLINE"
.
Hello all,
I have done a function SendSnapshotIncremental that add the options "-i" et "-I". As I don't know if this project is till active, before to do a pull request, I open this issue.
// SendFlag is the options flags passed to SendSnapshot
type SendFlag int
// Valid send options
const (
SendDefault SendFlag = 1 << iota
IncrementalStream = 1 << iota
IncrementalPackage = 1 << iota
)
func SendSnapshotIncremental(output io.Writer, d1 *Dataset, d2 *Dataset, flags SendFlag) error {
if d1.Type != DatasetSnapshot || d2.Type != DatasetSnapshot {
return errors.New("can only send snapshots")
}
// Flags for SendSnapshot
option := ""
if flags&IncrementalStream !=0 {
option = "-i"
}
if flags&IncrementalPackage !=0 {
option = "-I"
}
c := command{Command: "zfs", Stdout: output}
_, err := c.Run("send", option, d1.Name, d2.Name)
return err
}
Anyway, If there are anybody who can improve it, this will be perfect !
Hi,
This is part of a larger effort to develop a native port of Docker on Solaris.
go-zfs is great but has the following issues on Solaris:
As a part of coming up with a PR for this fix I investigated whether this should be resolved by creating a separate invocation for Solaris which would call zfs list but I propose a generic change due to performance improvements that it will bring to go-zfs.
Please see the issue titled "switch from zfs get to zfs list calls for better performance" (#40)
I propose to submit a commom PR to fix both these issues.
While installing docker I had a serious problem with my ZFS-Installation. My installation had no fragmentation determination enabled and therefore the command
root@atomic:~# zpool get -p name,health,allocated,size,free,readonly,dedupratio,fragmentation,freeing,leaked tank
NAME PROPERTY VALUE SOURCE
tank name tank -
tank health ONLINE -
tank allocated 2098558267392 -
tank size 2989297238016 -
tank free 890738970624 -
tank readonly off -
tank dedupratio 1.00x -
tank fragmentation - -
tank freeing 0 default
tank leaked 0 default
can not be parsed. the lib struggles with the "-" in fragmentation. This results in the following error message: "Zpool: error while getting pool information strconv.ParseUint: parsing "": invalid syntax"
Original issue in Docker issue trakcer: moby/moby#30551
code:
go fmt.Println(go-zfs.Snapshots("")
output:
[] Output does not match what is expected on this platform
When getting a size related attribute via dataset.GetProperty
because is not a field of the dataset
struct (e.g. refquota
), it is returned with K
, M
, `G suffix. Bytes would probably be preferable in most cases and you could still convert them afterwards if you wanted to.
I can open PR if no one working on this yet.
Opening an issue for adding support for zfs set
command. This would allow setting various properties on zfs filesystems/volumes/snapshots (such as quota
). I'm going to start working on a pull request for this.
we always call create with -p which will cause it to ignore already existing datasets. Should this be a flag?
As we will be iterating on this frequently.
This would remove the need for checking to ensure that a dataset is of the proper type before a certain method is called. (example: can only send snapshot: https://github.com/mistifyio/go-zfs/blob/master/zfs.go#L93-L95)
I think the Go technique of embedding is a good fit here: https://golang.org/doc/effective_go.html#embedding
And here is a quick example I threw together of how it could potentially look:
http://play.golang.org/p/mt4tgTtrk2
Overall, I think this approach could create a cleaner API. Please feel free to share your thoughts or concerns!
Due to fix for openzfs/zfs#3417
Hey, great lib ๐
Is it possible to access the "creation date"?
Its possible to list this information using:
zfs list -o name,creation -S creation
The last part (-S creation
) is not necessary, its just sorting the output already.
If its not yet possible, how could we add this?
Thx!
Compare to https://github.com/bakins/zfsd/blob/2b704f2e27703fc90d2dd6bd865c528e35ecd72f/utils.go#L124
Is "brute-force", but no magic, etc.
Hello all,
I have a question. When I use the function ReceiveSnapshot(), it does not end to read of conn net.Conn if there is not an error. So, if the delivery is good, I don't know how to stop the function and I don't want to use conn.Close because I want to continue sending information...
Thank you.
zfs get -H creation disk4T/code
returns: disk4T/code creation mer. mai 4 15:59 2016 -
but
func main() {
ds, _ := zfs.GetDataset("disk4T/code")
prop, _ := ds.GetProperty("creation")
fmt.Printf("%#v\n", prop)
}
returns:
"mer."
This is because in utils.go:Run() use:
for i, l := range lines {
output[i] = strings.Fields(l)
}
to split fields. But stings.Fields() doesn't make difference between space and tabulation.
If an error occurs when running zfs
or zpool
, and a non-zero exit code is returned, the following fmt.Errorf()
statement is used to create an error message:
https://github.com/mistifyio/go-zfs/blob/master/utils.go#L43
To enable better error handling in client code, I propose that this be replaced with a new struct type, and an Error()
method on that struct which can generate a string in the same format as the existing one.
This struct could look something like the following:
http://play.golang.org/p/ctPnZgy2UG
I would be happy to submit a PR if you'd like to move in this direction! Thanks for providing this excellent package!
Hi,
I'm going to use the mystifyio/go-zfs package in a cobra based command line application for filesystem and snapshot management. Thanks for your work so far!
I have implemented a straight-forward solution to support remote execution of the ZFS commands via SSH. I've tried to do this in a way, which has minimum impact on the core system and all your tests are running in a local environment as well as remotely against the Vagrant Ubuntu box. Pls. see ssh-remote-command branch of my fork for details. If you think, the implementation is of general interest for the project, I will be happy to review the code and create a pull request.
Additional features from my backlog are:
Thanks for your reply and comments,
Bernd
dataset.GetProperty uses the first line of zfs get
without any switches.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.