Comments (8)
From Kubuxu on IRC:
(19:09:15) Kubuxu: chriscool: in general it would be worth checking if running 0.3.x on 0.4.x does not damage the repos. It might not fit repo-migrations tests but I just thought about it.
(19:09:49) Kubuxu: running 0.3.x on 0.4.x repo, but it might already be tested.
from fs-repo-migrations.
(preface: i havent looked at the latest tests-- i will later this week.) i think the big thing for fs-repo-migrations testing is making sure everything works fine going up and down without losing any data, including for large repos or repos with large pinsets.
from fs-repo-migrations.
i wrote up somewhere in an issue or pr (dont recall where) a sample workload (adding lots of stuff, adding pins, removing pins, adding more stuff, gc-ing, applying migrations, etc, all interspersed).
i suspect too that a long-running randomized approach might be helpful in discovering problems.
from fs-repo-migrations.
@jbenet yeah I will search for what you wrote already
from fs-repo-migrations.
@jbenet: it's in PR #7. I will extract your ideas and @whyrusleeping's in comments below.
from fs-repo-migrations.
@whyrusleeping's idea from PR #7:
i'm fine having this tested with or without docker, my main idea for testing this goes something like:
- download ipfs v0.3.10
- init with that binary
- A: generate a set of files that get added and pinned
- B: generate a set of files that get added and unpinned
- C: generate a set of directory trees that get added
- D: generate a set of directory trees that have their subtrees added as well (pin root recursive, and some subtree)
- E: generate some files that have their root directly pinned, and nothing else
- run fs-repo-migration
- download 0.4.0 binary
- verify each of the above
from fs-repo-migrations.
it may be nice to start with the very first version, and ratchet up all the way, and all the way back down. to ensure all the migrations continue working. :) basically:
(first) for each repo version:
- download a version of go-ipfs corresponding to that repo
for each repo version transition (A->B), both going up and going down:
- (1) ipfs init (with binary known to work with A)
- (2) exercise ipfs repo with lots of data + an involved workload:
- add lots of files, some large. exercise pinset + gc, pin + unpin various nodes, directly or recursively. get lots of overlap. finish with significant unpinned content (which would go away with a repo gc) and with various recursive pins that overlap
- https://github.com/jbenet/go-random-files may be useful here
- (3) verify (a) content exists or does not exist in repo as expected, (b) pinset values work as expected (ie the right {recursive, direct, and indirect} pins)
- (4) attempt to use binary known to work with B (should get repo version error)
- (5) migrate repo from A to B (either up, or down)
- (6) verify the same things again. (now with binary known to work with B)
maybe we can start with all that for 2 version transitions:
test transition 2-to-3
test transition 3-to-2
repo version 2 with [email protected]
repo version 3 with [email protected]
from fs-repo-migrations.
So right now we have a number of tests that test transition 2-to-3 and 3-to-2 in:
- t0040-migration-2-3.sh: simple v0.3.10 to v0.4.0-dev and back with a few files
- t0050-migration-2-3-pins.sh: same as previous with pins and gc
- t0060-migration-stress.sh: v0.3.11 to v0.4.0-dev and back with lots of objects and daemon running
- t0070-migration-many.sh: v0.3.7 to v0.3.10 to v0.4.0-dev to v0.3.8 to v0.3.10
Suggestions for more or improved tests?
from fs-repo-migrations.
Related Issues (20)
- 11-to-12: Raw multihash migration: improvements to GC and reverts HOT 1
- Cannot use "go install" for latest release of fs-repo-migrations binary HOT 3
- fs-repo-migrations is difficult to package HOT 3
- ipfs daemon -c PATH --migrate doesn't use supplied path HOT 2
- Build with CGO_ENABLED=0 for dist.ipfs.io binaries HOT 3
- Go 1.16 is EOL HOT 3
- panic: qtls.ClientHelloInfo doesn't match HOT 2
- [Tracker] Kubo 0.16 migration list
- Migration output should provide progress to completion HOT 3
- 11-to-12 migration fails with s3ds plugin HOT 1
- failed to download migration HOT 1
- Migration not working automatically HOT 1
- version is lower but cannot update HOT 2
- [windows] migration on windows 13-to-14 fails with `paramter is incorrect` message HOT 1
- 14 to 15 does not work with empty `HTTPHeader` field
- Document upgrade reversion HOT 5
- Configuration issue on Kubo 0.23dev and migration 14-15 HOT 2
- bug: repo migration errors HOT 12
- bug: cannot `go install github.com/ipfs/fs-repo-migrations@latest` HOT 2
- test: does IPFS fallback work properly for downloading repo migrations? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fs-repo-migrations.