Migrations for the filesystem repository of ipfs clients
License: MIT License
Check out our website at ipfs.tech.
For papers on IPFS, please see the Academic Papers section of the IPFS Docs.
MIT.
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
![ipfs-mgmt-read-write[bot] avatar](https://avatars.githubusercontent.com/in/195163?v=4 "ipfs-mgmt-read-write[bot]")
If you install ipfs from snap and upgrade it, repo migration fails and you are stuck with an unusable IPFS data.
go test ./... fails for various reasons
One of them is dependency problems.
ipfs-1-to-2/go-datastore/lru/datastore.go:6:2: cannot find package "github.com/ipfs/fs-repo-migrations/ipfs-1-to-2/golang-lru" ...
ipfs-1-to-2/goprocess/context/context.go:5:2: cannot find package "golang.org/x/net/context" ..
ipfs-3-to-4/flatfs/flatfs_test.go:12:2: cannot find package "github.com/ipfs/go-datastore/flatfs" ...
...
Note, that one of the ideas (I think) behind fs-repo-migrations was that all dependences should be checked in (@whyrusleeping is this correct?)
Also note that one of the failed dependence is with a test of a dependency.
Another are broken tests:
ipfs-4-to-5/go-ds-flatfs/convert_test.go:18: undefined: tempdir
ipfs-4-to-5/go-ds-flatfs/convert_test.go:60: undefined: tempdir
ipfs-4-to-5/go-ds-flatfs/convert_test.go:133: undefined: tempdir
ipfs-4-to-5/go-ds-flatfs/convert_test.go:168: undefined: tempdir
? github.com/ipfs/fs-repo-migrations/ipfs-4-to-5 [no test files]
ok github.com/ipfs/fs-repo-migrations/ipfs-4-to-5/go-datastore 0.007s
ok github.com/ipfs/fs-repo-migrations/ipfs-4-to-5/go-datastore/query 0.003s
FAIL github.com/ipfs/fs-repo-migrations/ipfs-4-to-5/go-ds-flatfs [build failed]
? github.com/ipfs/fs-repo-migrations/ipfs-4-to-5/migration [no test files]
Note that the broken test is inside a dependency.
Note the README has this (under Developing Migrations):
The process should be:
- Frozen. After the tool is written, all code must be frozen and vendored.
So it may not be kosher to fix old code.
fs-repo-migrations is a binary that knows how to migrate repository versions since version 1 to soon-to-be version 9.
Each individual migration is a go-module (and a go executable) but they are all wired together in fs-repo-migrations, which is published on dist.ipfs.io and consumed by ipfs and ipfs-update (in very similar ways) when a migration is necessary (either by http-download or by ipfs-download).
Each migration module needs to be built offline, and they use vendored dependencies. Due to the long history of migrations, these includes library copy-pastes added as extra modules, gxed-vendored dependencies added as submodules and go-mod vendored dependencies.
Migrations are mostly tested with single, non vendored sharness tests, with some deps depending on iptb.
fs-repo-migrations binary is now 54M (to version 7), soon to growinit()ed ONCE. Multiple subpackages may vendor different versions and then fail badly. This needs patching across all migrations to prevent init() from happening or to align the libraries./net/trace (if I remember well) do not allow multiple versions and fail. This requires alignment between all submodules.We should make it easy for anyone to drop in and write a migration. Migration code should be fully isolated and testable in isolation. We should prevent having to touch any of the old migrations, or to having to adjust test things which affect all migrations and end up becoming a time sink on the developer.
Thus:
fs-repo-migrations binary is deprecated and we stop using it. We build migration-specific binaries for all migrations.ipfs-version-to-version. Tests for new migrations (sharness or not) are added in the submodule directly for new migrationsgo-migrations.ipfs.io (using the same mechanism as dist.ipfs.io).This is not much work but, I hope, will facilitate the life of anyone needing to write a migration in the future a lot.
Crash during pin add:
Apr 08 00:56:03 corbett ipfs[574]: panic: counter already signaled done. use a new counter.
Apr 08 00:56:03 corbett ipfs[574]: goroutine 268550671 [running]:
Apr 08 00:56:03 corbett ipfs[574]: gx/ipfs/QmQNQhNmY4STU1MURjH9vYEMpx2ncMS4gbwxXWtrEjzVAq/go-todocounter.(*todoCounter).Increment(0xc0b7a1b5c0, 0xc000000001)
Apr 08 00:56:03 corbett ipfs[574]: /home/steb/projects/go/src/github.com/ipfs/distributions/dists/go-ipfs/gopath/src/gx/ipfs/QmQNQhNmY4STU1MURjH9vYEMpx2ncMS4gbwxXWtrEjzVAq/go-todocounter/counter.go:86 +0xaf
Apr 08 00:56:03 corbett systemd[1]: ipfs.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Apr 08 00:56:03 corbett systemd[1]: ipfs.service: Unit entered failed state.
Apr 08 00:56:03 corbett systemd[1]: ipfs.service: Failed with result 'exit-code'
If related, server was in the process of pinning a directory with 2094 files with a cumulative size of 18Gb, 541 child blocks, happened at about 30% progress.
pin add -r --progress QmW3fQ76LmpCQ3DMDQF21dB2ZGsp9cGuDKMxobrUQpmsuq
go-ipfs version: 0.4.19-
Repo version: 7
System version: amd64/linux
Golang version: go1.11.5
Config is profile badgerds,server
"Datastore": {
"BloomFilterSize": 0,
"GCPeriod": "1h",
"HashOnRead": false,
"Spec": {
"child": {
"path": "badgerds",
"syncWrites": true,
"truncate": true,
"type": "badgerds"
},
"prefix": "badger.datastore",
"type": "measure"
},
"StorageGCWatermark": 90,
"StorageMax": "300GB"
},
I've managed to successfully migrate my repo_v3 to a repo_v4, but I was unable to revert it back. It seems that there is no way to revert at the high level tool
» fs-repo-migrations -h
Usage of fs-repo-migrations:
-to int
specify version to upgrade to (default 4)
-v print highest repo version and exit
-y answer yes to all prompts
And using the dedicated binary doesn't work
⭠ master± ⮀ ~/code/go-projects/src/github.com/ipfs/fs-repo-migrations/ipfs-3-to-4 ⮀
» ./ipfs-3-to-4 --revert
Usage of ./ipfs-3-to-4:
-f whether to force a migration (ignores warnings)
-path string
file path to migrate for fs based migrations
-revert
whether to apply the migration backwards
-verbose
enable verbose logging
⭠ master± ⮀ ~/code/go-projects/src/github.com/ipfs/fs-repo-migrations/ipfs-3-to-4 ⮀
» ./ipfs-3-to-4 -revert
Usage of ./ipfs-3-to-4:
-f whether to force a migration (ignores warnings)
-path string
file path to migrate for fs based migrations
-revert
whether to apply the migration backwards
-verbose
enable verbose logging
I've also found it to be weird to have full word flags behind a - and not two, like --verbose instead of -verbose.
As a go-ipfs user, I was expecting for the installation process to be a bit more verbose or even use gx, currently all that is printed is:
» make install
go install
It would be useful to have a message saying that it was installed and that users can type fs-repo-migrations on their terminal.
Or at least it should have a way to tell which migrations it can handle, without trying the migration.
Otherwise it is difficult for people to know if they have the right version of fs-repo-migrations for their needs.
Currently running:
go-ipfs version: 0.4.8-rc1-
Repo version: 5
System version: amd64/linux
Golang version: go1.7.1
Commit a8b56d3ad7789ca2ec12257a00d8d0408c3c0c9e
Initializing daemon...
Adjusting current ulimit to 2048...
Successfully raised file descriptor limit to 2048.
Found outdated fs-repo, migrations need to be run.
Run migrations now? [y/N] Y
=> Looking for suitable fs-repo-migrations binary.
=> Running: /storage/home/achin/devel/gopath/bin/fs-repo-migrations -to 5 -y
Found fs-repo version 4 at /nas/achin/ipfs
===> Running migration 4 to 5...
applying 4-to-5 repo migration
locking repo at "/nas/achin/ipfs"
- verifying version is '4'
> Upgrading datastore format to have sharding specification file
> creating a new flatfs datastore with new format
> converting current flatfs datastore to new format
Moving Keys...
22790 keys so farERROR: failed to convert flatfs datastore: rename /nas/achin/ipfs/blocks-v4/12205/1220543aa6f6b9a533c8bf80568090cdf24b693aaa2f9b574a33784d8462fdc5579c.data /nas/achin/ipfs/blocks-v5/79/1220543aa6f6b9a533c8bf80568090cdf24b693aaa2f9b574a33784d8462fdc5579c.data: no such file or directory
attempting to revert...
Moving Keys...
22790 keys so far
Cleaning Up...
All Done.
ipfs migration: migration 4 to 5 failed: rename /nas/achin/ipfs/blocks-v4/12205/1220543aa6f6b9a533c8bf80568090cdf24b693aaa2f9b574a33784d8462fdc5579c.data /nas/achin/ipfs/blocks-v5/79/1220543aa6f6b9a533c8bf80568090cdf24b693aaa2f9b574a33784d8462fdc5579c.data: no such file or directory
=> Failed: /storage/home/achin/devel/gopath/bin/fs-repo-migrations -to 5 -y
The migrations of fs-repo failed:
migration failed: exit status 1
If you think this is a bug, please file an issue and include this whole log output.
https://github.com/ipfs/fs-repo-migrations
Error: migration failed: exit status 1
ipfs tells me I need to run a migration.I'm running the the latest version of ipfs from Homebrew, which only supports repo version 5:
> ipfs version --all
go-ipfs version: 0.4.10-4679f80
Repo version: 5
System version: amd64/darwin
Golang version: go1.8.3
And fs-repo-migrations -to 5 doesn't work. I got some suggestions for how to downgrade manually in #ipfs but it sounds like a pain. It would be nice if it had just upgraded to 5, since that's what my ipfs claims to support. Seems like an easy enough check?
I somehow ended up with go-ipfs v0.4.4 installed in my regular gopath, just running tests in a vanilla clone.
I upgraded from v0.3.8 to v0.4.0-dev (ipfs/kubo@e04a31d) and:
$ ipfs daemon
Initializing daemon...
Error: Repo has incorrect version: 2
Program version is: 3
Please run the ipfs migration tool before continuing.
See https://github.com/ipfs/fs-repo-migrations/blob/master/run.md
Sorry for the inconvenience. In the future, these will run automatically.
$ go get -u github.com/ipfs/fs-repo-migrations
$ fs-repo-migrations
ipfs migration: already at or above target version number
$ ipfs daemon
Initializing daemon...
Error: Repo has incorrect version: 2
Program version is: 3
Please run the ipfs migration tool before continuing.
See https://github.com/ipfs/fs-repo-migrations/blob/master/run.md
Sorry for the inconvenience. In the future, these will run automatically.
Debugging info:
$ go version
go version go1.5.1 linux/amd64
$ ipfs version
ipfs version 0.4.0-dev
$ uname -a
Linux thunder 3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Ignoring arguments can lead people to wrongly think that it has some hidden features:
> fs-repo-migrations versions
Found fs-repo version 2 at /home/utilisateur/.ipfs
Do you want to upgrade this to version 3? [y/n] n
> fs-repo-migrations sdff sdfsf sdfsdf
Found fs-repo version 2 at /home/utilisateur/.ipfs
Do you want to upgrade this to version 3? [y/n] n
Although they are advertised as '386', the migration binaries for the, um, x86-32 architecture contain SSE2 instructions, which generate an 'illegal opcode' exception on CPUs older than Pentium 4 (i786). (And possibly some instructions that will crash even that; I haven't checked.) This may leave users unable to actually use the IPFS software after upgrading:
$ ipfs daemon
Initializing daemon...
Found outdated fs-repo, migrations need to be run.
Run migrations now? [y/N] y
=> Looking for suitable fs-repo-migrations binary.
=> None found, downloading.
The migrations of fs-repo failed:
no fs-repo-migration binary found for version 6: failed to check migrations version: signal: illegal instruction (core dumped)
If you think this is a bug, please file an issue and include this whole log output.
https://github.com/ipfs/fs-repo-migrations
Error: no fs-repo-migration binary found for version 6: failed to check migrations version: signal: illegal instruction (core dumped)
Compiling for an older subarchitecture will of course result in some performance degradation, but given that migrations are supposed to happen rarely (they are, right?), I believe sacrificing performance for portability is acceptable. Newer x86 computers are 64-bit anyway.
As for which x86-32 sub-architecture to target instead... I guess either literal i386 (for maximum portability), or i686, which is targeted by many Linux distributions.
During migration to go-ipfs 0.4.5
Found fs-repo version 4 at /home/ipfs/.ipfs
===> Running migration 4 to 5...
applying 4-to-5 repo migration
locking repo at "/home/ipfs/.ipfs"
- verifying version is '4'
> Upgrading datastore format to have sharding specification file
> creating a new flatfs datastore with new format
> converting current flatfs datastore to new format
Moving Keys...
7230 keys so farfailed to decode entry in flatfs
20940 keys so farfailed to decode entry in flatfs
31630 keys so farfailed to decode entry in flatfs
...
I got
ERROR: failed to convert flatfs datastore: remove /home/ipfs/.ipfs/blocks-v4/CIQN3: directory not empty
attempting to revert...
And I wondering about 'failed to decode entry in flatfs' lines during moving.
Details:
https://gist.github.com/vitlav/090c09353be611238574df0115ce9115
While this isn't really an issue now, but it soon may be as datastore plugin support got merged recently.
With how the go plugin system works, we won't be able to load the plugins here, so we need to figure out a way of working with custom datastores:
repo/fsrepo and parts of the go-ipfs plugin subsystemPlease refer here :
ipfs/kubo#7676
Initializing daemon...
go-ipfs version: 0.6.0-53971d664
Repo version: 10
System version: amd64/darwin
Golang version: go1.14.4
Found outdated fs-repo, migrations need to be run.
Run migrations now? [y/N] y
=> Looking for suitable fs-repo-migrations binary.
=> None found, downloading.
=> Running: /var/folders/yl/5sr1t26d27ddbrx5mcfqr48r0000gn/T/go-ipfs-migrate437691780/fs-repo-migrations -to 10 -y
Found fs-repo version 7 at /Users/pandiyarajaramamoorthy/.ipfs
===> Running migration 7 to 8...
applying 7-to-8 repo migration
locking repo at "/Users/pandiyarajaramamoorthy/.ipfs"
Upgrading config to new format
Bootstrap field missing or of the wrong type
Nothing to migrate
updated version file
===> Migration 7 to 8 succeeded!
===> Running migration 8 to 9...
applying 8-to-9 repo migration
updated version file
===> Migration 8 to 9 succeeded!
===> Running migration 9 to 10...
applying 9-to-10 repo migration
locking repo at "/Users/pandiyarajaramamoorthy/.ipfs"
Upgrading config to new format
ipfs migration: migration 9 to 10 failed: unexpected end of JSON input
=> Failed: /var/folders/yl/5sr1t26d27ddbrx5mcfqr48r0000gn/T/go-ipfs-migrate437691780/fs-repo-migrations -to 10 -y
The migrations of fs-repo failed:
migration failed: exit status 1
If you think this is a bug, please file an issue and include this whole log output.
https://github.com/ipfs/fs-repo-migrations
Error: migration failed: exit status 1
I am getting this error now:
> bin/ipfs-update versions
ERROR: fetching resource: 400 Bad Request
ERROR: Failed to query versions: 400 Bad Request: Path Resolve error: could not resolve name.
Any idea about what's going on?
maybe the migrations should be allowed to output a little bit of important text to users. in particular, i'd like to know the fact that ~/.go-ipfs is now at ~/.ipfs
now:
> fs-repo-migrations
Found fs-repo version 0 at /Users/jbenet/.go-ipfs
Do you want to upgrade this to version 2? [y/n]
Please press either 'y' or 'n'
Do you want to upgrade this to version 2? [y/n]
y
Running migration 0 to 1...
Migration 0 to 1 succeeded!
Running migration 1 to 2...
Migration 1 to 2 succeeded!
maybe make it:
> fs-repo-migrations
Found fs-repo version 0 at /Users/jbenet/.go-ipfs
Do you want to upgrade this to version 2? [Y/n]
==> Migration 0-to-1 applying...
added version file
==> Migration 0-to-1 succeeded!
==> Migration 1-to-2 applying...
updated version file
moved ipfs objects from leveldb (datastore/) into flatfs (blocks/)
moved repo from /Users/jbenet/.go-ipfs to /Users/jbenet/.ipfs
==> Migration 1 to 2 succeeded!
Right now there are a large number of vendored gx dependencies in fs-repo-migrations/ipfs-6-to-7/vendor/gx/ipfs/'. Could we move those to fs-repo-migrations/vendor/gx/ipfs/`.
I need to make use of many of the same gx deps. in my migration code and want to avoid checking in the same files in two places.
CC @Stebalien
On windows, if no 'IPFS_PATH' was set, the migration tools complains with "could not determine IPFS_PATH, home dir not set".
The default path is in the user's directory under .ipfs
Setting the variable to this folder solved the issue for me.
Currently not able to upgrade an ~/.ipfs2 instance via fs-repo-migrations as this does not accept location argument for the target to upgrade.
Hit an error after the migration with fs-repo-migration.
Yesterday, I upgraded ipfs version from 0.4.15 to 0.4.18, according to the manual of IPFS- Update, finished migration job before that. Everything seemed fine before I restarted my mac this morning. There was an error msg popped up for couple of times before ipfs-desktop quit by itself. The msg list below（the screenshot picture is attached ）
"Some unexpected error occurred and we couldn't handle it. Please check /Users/maxzong/Library/Application Support/ipfs-desktop/logs/error.log for the latest logs and open an issue on https://github.com/ipfs-shipyard/ipfs-desktop/issues."
Ok, after viewed the error.log what lies in /Users/maxzong/Library/Application Support/ipfs-desktop/logs/ I got the information lists below:
"Your programs version (6) is lower than your repos (7).\nPlease update ipfs to a version that supports the existing repo, or run\na migration in reverse.\n\nSee https://github.com/ipfs/fs-repo-migrations/blob/master/run.md for details"
Does it that the latest ipfs-desktop application isn't adapting to repo's version? Repo is the newest I downloaded and installed yesterday, it's the latest version following the steps in "Running Repo Migrations"
Is there anyone can gimme a hand to solve it? thx a lot
Failed to install 0.4.3-rc3 with ipfs-update:
> ipfs-update install v0.4.3-rc3
fetching go-ipfs version v0.4.3-rc3
binary downloaded, verifying...
success!
stashing old binary
installing new binary to /usr/local/bin/ipfs
checking if repo migration is needed...
check complete, migration required.
fetching fs-repo-migrations version v1.1.0
ERROR: Migration Failed: no known migration supports version 4
install failed, reverting changes...
no known migration supports version 4Looks like fs-repo-migrations latest is not yet in dist.ipfs.io. And https://dist.ipfs.io/fs-repo-migrations shows only two.
The combination of us releasing a version of go-ipfs that almost forbids insure Cids but doesn't go as far as purging them from the datastore, and the release of the 6-to-7 migration created a messy situation.
There are several problems, the main one is that there is no good automatic way to deal with pinned insecure Cids. I would like to abort and tell the user to unpin them manually. To give them a chance to save the contents somehow, however, this means the user will need to install go-ipfs v0.4.14, but in order to use the migration they will need run another migration to downgrade there repo to version '6' (as the 6-to-7 conversion runs before the 7-to-8) and then unpin them with the older version of go-ipfs.
Also injecting insecure hashes and then testing that they are not in the repo after the 7-to-8 migration is difficult to do in the sharness testing. This utility could help.
So I am thinking about creating a one-off utility that works works with repo version 6,7,and 8 and has the following command:
6-7-8-migration-helper pin-rm <hash>...
6-7-8-migration-helper block-put <hash>
# ^ contents in stdin, used for testing, and allows insecure and id. hashes if the repo version < 8
6-7-8-migration-helper block-get <hash>
6-7-8-migration-helper dag-get <hash>
# ^ useful in examine insure cids structure
6-7-8-migration-helper list-insecure|list-identity
If I create this utility the binary should be provided in dist.ipfs.io somehow, probably as part of the migration.
Thoughts? Overkill?
For testing I could settle with unit tests as it far easier to inject insure hashes using code. Or maybe just find a way to hack it in the sharness testing (like manually mess with the version version string in the repo)
For the case when there are pinned insecure hashes, just tell the user what they need to do to clean up. I am hoping there are not many users with this problem but the fact that ipfs/kubo#5067 was filed mean they do exist.
migrations should acquire the repo lock (as it was supposed to be acquired at the particular version) to ensure that the repo is not modified while a daemon is running. the error message to the user should be friendly.
What i got:
> old-ipfs init
> old-ipfs daemon &
> fs-repo-migrations
Found fs-repo version 0 at /Users/jbenet/.go-ipfs
Do you want to upgrade this to version 2? [y/n]
y
Running migration 0 to 1...
Migration 0 to 1 succeeded!
Running migration 1 to 2...
ipfs migration: migration 1 to 2 failed: resource temporarily unavailable
what i want
Found fs-repo version 0 at /Users/jbenet/.go-ipfs
Do you want to upgrade this to version 2? [y/n]
y
error: daemon is currently running.
please close the running daemon and run this command again.
the 2-3pins test checks the pins but does not:
it also does not remove pins objects.
an important goal of these tests is to exercise the pinset across versions, including deletion of pins and GCs that do remove some content and leave other content.
I want to run go-ipfs version: 0.4.20-rc1-5880b3075-dirty, but it mention i should migration to Repo version: 7 .
And then an error occur:
ipfs migration: migration 1 to 2 failed: rename E:\go_work\ipfs E:\go_work\ipfs: Access is denied.
Here is the details of the log:
I don't know this is a bug or not, please help me to make clear to it.
Thanks .
Hey @kevina, I mentioned on IRC that there's another config change I'd like to get into the 5-to-6 migration. Sorry I'm bringing this forward at the 11th hour. At least it only simply adds list values, and doesn't do any structural changes.
What I have in mind is adding /dnsaddr addresses for each of the default QmSoL bootstrappers. The current addresses are very fragile since they are coupled with the DigitalOcean hosts that currently run the bootstrappers. This actually makes the IPFS network quite vulnerable since anyone can just attack these 8 little hosts. The /dnsaddr on the other hand works similarly to dnslinks -- check out the examples in https://github.com/multiformats/go-multiaddr-dns.
I'm fine doing this in another PR on top of yours, but it would be great if you could look into it since you already have all the context present. Otherwise I can give it a try next week.
Migration:
Bootstrap item
QmSoLBootstrap actually contains the item
/dnsaddr/bootstrap.libp2p.io for this item to Bootstrap list/ip4/1.2.3.4/tcp/4001/ipfs/QmSoL, append /dnsaddr/bootstrap.libp2p.io/ipfs/QmSoLRevert:
Bootstrap
/dns
BootstrapUpgraded from version 2 to 4, and migrated repo today, but ipfs daemon is still reporting v.3. I suspect there is a problem with an environment PATH somewhere. I don't really want to downgrade the repo if not really necessary.
$ fs-repo-migrations
Found fs-repo version 2 at /Users/joe/.ipfs
Do you want to upgrade this to version 4? [y/n] y
===> Running migration 2 to 3...
applying 2-to-3 repo migration
locking repo at "/Users/joe/.ipfs"
- verifying version is '2'
beginning pin transfer
- opening datastore at "/Users/joe/.ipfs"
- created version 3 pinner
- loading recursive pins
- transfered recursive pins
- loading direct pins
- transfered direct pins
pinner synced to disk
cleaning old pins
- deleting recursePin root key: "/local/pins/recursive/keys"
- deleting pin key: "/local/pins/recursive/keys/\x12 R\xc6<wu9k?\x82\xc69\x97zr#\xc2\xd9j\x9fp\xb5\xfd\x8b\x1dQ?\x8c[iܮ\xd4"
- deleting pin key: "/local/pins/recursive/keys/\x12 W\xed\xa7\x06\x1b\xa3\xfc_\xc5@\xcaAX\x8b\xdfԫ\xab\xbb\xe0\x8f\x15x\f\xb2Sxdz$i\xa9"
- deleting pin key: "/local/pins/recursive/keys/\x12 \x90\xc0zw\x95\xc1\x195\x10\xa6\x96\xd1\xfd\xfc\x0f\x1eIG\xcf\xf8\xe4\"a\t\x96\xe6\t\xdb˗e\x98"
- deleting pin key: "/local/pins/recursive/keys/\x12 \x92\x9a0<9ڊ\vg\xc0\x96\x97F/hz\x00\xc68\xbc\xb5\x80\xfe\xae\x06E.\f\x1f \xb4"
- deleting pin key: "/local/pins/recursive/keys/\x12 ׁ\x8bI\x8b\x9b\\\x8e\x8b\x84\xd01\xf1\xc83\xab\xfd\x06dC\x10SN\xf5+\xfd\f$3\xc0*b"
- deleting pin key: "/local/pins/recursive/keys/\x12 \xec>\x13U\xd4+\xef\xe0ɂ<T\xdc\xcb\x16\xa5\xfa\v\xbbm\xfeb\xc8@\xad\xf5\x1c\x8e:*\b\x8f"
- deleting pin key: "/local/pins/recursive/keys/\x12 \xfa\xe5\xb4\x1a\x86\xf0,\xed\xef_\al\xc4\u007f\x93S{\xc5fc\xc2oq\xd5\x01\x02\xa7\xed\xc0\xd4՟"
- cleaned up oldstyle recursive pins
- deleting recursePin root key: "/local/pins/recursive/keys"
- cleaned up oldstyle direct pins
- deleting recursePin root key: "/local/pins/recursive/keys"
- cleaned up oldstyle indirect pins
pin transfer completed successfuly
updated version file
===> Migration 2 to 3 succeeded!
===> Running migration 3 to 4...
applying 3-to-4 repo migration
locking repo at "/Users/joe/.ipfs"
- verifying version is '3'
- opening datastore at "/Users/joe/.ipfs"
ipfs migration: migration 3 to 4 failed: resource temporarily unavailable
localhost:~ joe$ fs-repo-migrations
Found fs-repo version 3 at /Users/joe/.ipfs
Do you want to upgrade this to version 4? [y/n] y
===> Running migration 3 to 4...
applying 3-to-4 repo migration
locking repo at "/Users/joe/.ipfs"
- verifying version is '3'
- opening datastore at "/Users/joe/.ipfs"
transfering blocks to new key format
[12464 / 12464] Approx time remaining: 3s
transferring stored public key records
gathering keys...
got 80 keys, beginning transfer. This will take some time.
[80 / 80] Approx time remaining: 90s
transferring stored ipns records
gathering keys...
got 96 keys, beginning transfer. This will take some time.
[96 / 96] Approx time remaining: 540s
updated version file
===> **Migration 3 to 4 succeeded!**
localhost:~ joe$ ipfs daemon
Initializing daemon...
Error: Repo has incorrect version: 4
Program version is: 3
Please run the ipfs migration tool before continuing.
See https://github.com/ipfs/fs-repo-migrations/blob/master/run.md
Sorry for the inconvenience. In the future, these will run automatically.
localhost:~ joe$ ipfs-update version
v0.4.2
localhost:~ joe$ ipfs-update fetch
fetching go-ipfs version v0.4.2
localhost:~ joe$ **ipfs-update version
v0.4.2**
localhost:~ joe$ ipfs daemon
Initializing daemon...
Error: Repo has incorrect version: 4
Program version is: 3
Please run the ipfs migration tool before continuing.
See https://github.com/ipfs/fs-repo-migrations/blob/master/run.md
Sorry for the inconvenience. In the future, these will run automatically.
localhost:~ joe$
this needs a license
if the repo is in a different location, fs-repo-migrations fails to run 1-to-2 because of path conflict:
root@pluto:/ipfs/ipfs_master# IPFS_PATH=repo ./fs-repo-migrations
Found fs-repo version 0 at repo
Do you want to upgrade this to version 2? [y/n] y
===> Running migration 0 to 1...
wrote version file
===> Migration 0 to 1 succeeded!
===> Running migration 1 to 2...
ipfs migration: migration 1 to 2 failed: mkdir repo: file exists
It would be nice if there was a table showing which repo version corresponds to which ipfs versions.
Also if fs-repo-migration gets its own version number (see issue #12), these version numbers could be added to this table too.
cc @Kubuxu, @michaelavila as the current DRIs for these migrations.
fs-repo-migrations (workarounds exist)> ipfs version
ipfs version 0.4.5
> ipfs-update install v0.4.4
fetching go-ipfs version v0.4.4
binary downloaded, verifying...
success!
stashing old binary
installing new binary to /Users/jbenet/go/bin/ipfs
checking if repo migration is needed...
- Migration required
fetching fs-repo-migrations version v1.2.0
running migration: '/var/folders/27/vc54xwmx2q5dvx9y9vcss5l00000gn/T/ipfs-update-migrate541727985/fs-repo-migrations -to 4 -y'
ipfs migration: already at or above target version number
migration succeeded!
installation complete.
> ipfs version
ipfs version 0.4.4
> ipfs daemon
Initializing daemon...
Error: Your programs version (4) is lower than your repos (5).
Please update ipfs to a version that supports the existing repo, or run
a migration in reverse.
See https://github.com/ipfs/fs-repo-migrations/blob/master/run.md for details.
Obviously this is wrong.
fs-repo-migrations -to 4 -y is only forward. and therefore ipfs-update's (normal behavior) failed.It is not clear from the doc and from running fs-repo-migrations -h if it is possible to go back to a previous version or not. And if it is possible it is not clear how.
I'm in the process of updating the ipfs package/port for OpenBSD, and have noticed fs-migrations fail.
The current package/port builds ipfs-0.4.22, and when attempting to run the ipfs-0.5.1 binary (easily built with make build), I see the following error:
$ ipfs daemon
Initializing daemon...
go-ipfs version: 0.5.1-8431e2e87
Repo version: 9
System version: amd64/openbsd
Golang version: go1.13.9
Found outdated fs-repo, migrations need to be run.
Run migrations now? [y/N] y
=> Looking for suitable fs-repo-migrations binary.
=> None found, downloading.
=> Failed to download fs-repo-migrations.
The migrations of fs-repo failed:
failed to download latest fs-repo-migrations: GET https://ipfs.io/ipfs/QmUgfXycSjF9R8F4Tyauaz6LZ4bj5nbksg54G9GdF4fit6/fs-repo-migrations/v1.5.1/fs-repo-migrations_v1.5.1_openbsd-amd64.tar.gz error: 404 Not Found: ipfs resolve -r /ipfs/QmUgfXycSjF9R8F4Tyauaz6LZ4bj5nbksg54G9GdF4fit6/fs-repo-migrations/v1.5.1/fs-repo-migrations_v1.5.1_openbsd-amd64.tar.gz: no link named "fs-repo-migrations_v1.5.1_openbsd-amd64.tar.gz" under QmWgjNoXVSfZXJAG5LaovdDHMn7nT8hJGzXVeVHMq6fxy4
If you think this is a bug, please file an issue and include this whole log output.
https://github.com/ipfs/fs-repo-migrations
Error: failed to download latest fs-repo-migrations: GET https://ipfs.io/ipfs/QmUgfXycSjF9R8F4Tyauaz6LZ4bj5nbksg54G9GdF4fit6/fs-repo-migrations/v1.5.1/fs-repo-migrations_v1.5.1_openbsd-amd64.tar.gz error: 404 Not Found: ipfs resolve -r /ipfs/QmUgfXycSjF9R8F4Tyauaz6LZ4bj5nbksg54G9GdF4fit6/fs-repo-migrations/v1.5.1/fs-repo-migrations_v1.5.1_openbsd-amd64.tar.gz: no link named "fs-repo-migrations_v1.5.1_openbsd-amd64.tar.gz" under QmWgjNoXVSfZXJAG5LaovdDHMn7nT8hJGzXVeVHMq6fxy4
$
I'd like for this to work so updating a the go-ipfs package from the older (0.4.22) version to the newer (0.5.1 and beyond) will "just work".
I suspect some work needs to be done to remove the $GOOS ("openbsd" here) details from the filename...how different can these be between linux and openbsd?
I hit an error trying to upgrade my go-ipfs today.
$ ipfs --version
ipfs version 0.4.14
$ ipfs update install v0.4.17
fetching go-ipfs version v0.4.17
binary downloaded, verifying...
success!
stashing old binary
installing new binary to /usr/local/bin/ipfs
checking if repo migration is needed...
- Migration required
fetching fs-repo-migrations version v1.4.0
running migration: '/var/folders/gm/h2m4xm1d65d1yjbdz2v9kn5r0000gn/T/ipfs-update-migrate325778338/fs-repo-migrations -to 7 -y'
Found fs-repo version 6 at /Users/oli/.ipfs
===> Running migration 6 to 7...
applying 6-to-7 repo migration
ipfs migration: migration 6 to 7 failed: cannot acquire lock: Lock FcntlFlock of /Users/oli/.ipfs/repo.lock failed: resource temporarily unavailable
ERROR: Migration Failed: migration failed: exit status 1
install failed, reverting changes...
ERROR: migration failed: exit status 1
Error: exit status 1I've tried it with ipfs daemon running, not runnning, with sudo, without sudo. Is there anything else I should try or is there a bug I can help diagnose?
$ go get -u github.com/ipfs/fs-repo-migrations
# cd /home/kyrias/src/github.com/ipfs/fs-repo-migrations; git submodule update --init --recursive
fatal: no submodule mapping found in .gitmodules for path 'ipfs-1-to-2/golang-lru'
package github.com/ipfs/fs-repo-migrations: exit status 128There was some kind of upgradation or migration. When I starting my IPFS daemon server. I am getting datastore not found error. Please can one can help me solving this issue.
Rutviks-MacBook-Pro-2:~ rutvikpatel$ ipfs daemon
Initializing daemon...
go-ipfs version: 0.7.0
Repo version: 10
System version: amd64/darwin
Golang version: go1.14.4
Found outdated fs-repo, migrations need to be run.
Run migrations now? [y/N] y
=> Looking for suitable fs-repo-migrations binary.
=> None found, downloading.
=> Running: /var/folders/7y/pyj6j88x6c37113c_lfnsmjc0000gn/T/go-ipfs-migrate054774885/fs-repo-migrations -to 10 -y
Found fs-repo version 2 at /Users/rutvikpatel/.ipfs
===> Running migration 2 to 3...
applying 2-to-3 repo migration
locking repo at "/Users/rutvikpatel/.ipfs"
Error: migration failed: exit status 1
$ fs-repo-migrations
Found fs-repo version 3 at /Users/will/.ipfs
Do you want to upgrade this to version 4? [y/n] y
===> Running migration 3 to 4...
applying 3-to-4 repo migration
locking repo at "/Users/will/.ipfs"
- verifying version is '3'
- opening datastore at "/Users/will/.ipfs"
transfering blocks to new key format
enumerating keysskipping (no .data): 12204961/put-676068147
skipping (no .data): 12205497/put-106496515
skipping (no .data): 1220553f/put-763233442
skipping (no .data): 122063bf/put-044131062
enumerating keys...skipping (no .data): 1220c4f1/put-380945039
enumerating keys....skipping (no .data): 1220f279/put-415760420
skipping (no .data): 1220f2c9/put-732064153
[1 / 11872]failed to decode: /Users/will/.ipfs/blocks/CIQA2/CIQA23O4Q<....>7N2PPGBKBZFBUDYZA.data
ipfs migration: migration 3 to 4 failed: encoding/hex: odd length hex string
go-ipfs master already requires 5
I get this in conjunction with ipfs-update:
% ipfs-update install latest
fetching go-ipfs version v0.4.5
binary downloaded, verifying...
success!
stashing old binary
installing new binary to /usr/local/bin/ipfs
checking if repo migration is needed...
check complete, migration required.
fetching fs-repo-migrations version v1.2.1
ERROR: Migration Failed: no known migration supports version 5
install failed, reverting changes...
no known migration supports version 5
haven't checked on its own.
(see also: ipfs/ipfs-update#55)
Feb 12 20:13:19 ipfs-ink systemd[1]: Started docker: go-ipfs.
Feb 12 20:13:20 ipfs-ink docker[17027]: ipfs version 0.4.5
Feb 12 20:13:20 ipfs-ink docker[17027]: Found IPFS fs-repo at /data/ipfs
Feb 12 20:13:20 ipfs-ink docker[17027]: Initializing daemon...
Feb 12 20:13:20 ipfs-ink docker[17027]: Found outdated fs-repo, migrations need to be run.
Feb 12 20:13:20 ipfs-ink docker[17027]: => Looking for suitable fs-repo-migrations binary.
Feb 12 20:13:20 ipfs-ink docker[17027]: => None found, downloading.
Feb 12 20:13:22 ipfs-ink docker[17027]: The migrations of fs-repo failed:
Feb 12 20:13:22 ipfs-ink docker[17027]: no fs-repo-migration binary found for version 5: failed to check migrations version: fork/exec /tmp/go-ipfs-migrate573118154/fs-repo-migrations: no such file or directory
Feb 12 20:13:22 ipfs-ink docker[17027]: If you think this is a bug, please file an issue and include this whole log output.
Feb 12 20:13:22 ipfs-ink docker[17027]: https://github.com/ipfs/fs-repo-migrations
Feb 12 20:13:22 ipfs-ink docker[17027]: Error: no fs-repo-migration binary found for version 5: failed to check migrations version: fork/exec /tmp/go-ipfs-migrate573118154/fs-repo-migrations: no such file or directory
Feb 12 20:13:23 ipfs-ink systemd[1]: go-ipfs.service: Main process exited, code=exited, status=1/FAILURE
In func (m Migration) Apply(opts migrate.Options) error I was expecting to see lines similar to:
log.VLog("locking repo at %q", opts.Path)
lk, err := lock.Lock2(opts.Path)
if err != nil {
return err
}
defer lk.Close()
repo := mfsr.RepoPath(opts.Path)
log.VLog(" - verifying version is '5'")
if err := repo.CheckVersion("6"); err != nil {
return err
}
I am not sure if the Lock is necessary here, and I suppose the version check is not required because the repo is opened with go-ipfs code. However, if there is a repo mismatch, it will create a confusing error as to run the migration, from within the migration code itself!
I am opening this issue mainly for how to handle a similar situation in my migration which will clean the repo of insure and identity hashes.
CC @Stebalien
Notes on recovery when the automatic migration fails:
https://gist.github.com/kevina/d39e800ddffed217ca149c600f0735e4
While testing some changes to ipfs-update, this was output from fs-repo-migrations.
(Note: I bypassed the checks in ipfs-update to test fs-repo-migrations execution on Windows, but I don't have old data to migrate so I'm not sure yet why it's starting w/ v1 in that case since I should only have the latest IPFS data version)
fetching go-ipfs version v0.4.10
binary downloaded, verifying...
success! tests all passed.
installing new binary to C:\Go\bin\ipfs.exe
checking if repo migration is needed...
check complete, migration required.
fetching fs-repo-migrations version v1.2.2
running migration: 'C:\Users\<USER>\AppData\Local\Temp\ipfs-update-migrate040886245\fs-repo-migrations.exe -to 5 -y'
Found fs-repo version 1 at C:\Users\<USER>/.ipfs
===> Running migration 1 to 2...
performed sanity check
moved blocks from leveldb to flatfs
ipfs migration: migration 1 to 2 failed: rename C:\Users\<USER>/.ipfs C:\Users\<USER>/.ipfs: Access is denied.
ERROR: Migration Failed: migration failed: exit status 1
install failed, reverting changes...
ERROR: migration failed: exit status 1
If you have ideas about some kind of migration tests we could do, please add them as comments in this issue. Thanks!
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.