svn-all-fast-export / svn2git Goto Github PK
View Code? Open in Web Editor NEW:octopus: A fast-import based converter for an svn repo to git repos
License: GNU General Public License v3.0
:octopus: A fast-import based converter for an svn repo to git repos
License: GNU General Public License v3.0
I get a segmentation fault while processing a SVN repository. There is no information other than:
fatal: EOF in data (285 bytes remaining)
It creates a dump report but I don't know what the contents means. It prints '+' to the screen over and over and it never appears to move past that point. It's like it has encountered a circular dependency in the svn repository.
Here's the back trace from the core dump:
[New LWP 26037]
[New LWP 26038]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `svn2git/svn-all-fast-export --identity-map ../tools/authors/authors.txt --rules'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007faaad763de1 in QByteArray::append (this=this@entry=0x189ef78, ba=...) at tools/qbytearray.cpp:1616
1616 d->data[d->size] = '\0';
(gdb) bt
#0 0x00007faaad763de1 in QByteArray::append (this=this@entry=0x189ef78, ba=...) at tools/qbytearray.cpp:1616
#1 0x0000000000418c61 in append (s=..., this=0x189ef78) at /usr/include/QtCore/qstring.h:997
#2 FastImportRepository::Transaction::addFile (this=0x189ef30, path=..., mode=33261, length=236) at repository.cpp:899
#3 0x000000000042259e in dumpBlob (txn=txn@entry=0x189ef30, fs_root=fs_root@entry=0xd3f6f0, pathname=<optimized out>, finalPathName=..., pool=pool@entry=0x7faa99e92038) at svn.cpp:292
#4 0x0000000000423fff in recursiveDumpDir (txn=txn@entry=0x189ef30, fs_root=fs_root@entry=0xd3f6f0, pathname=..., finalPathName=..., pool=pool@entry=0xdd2218) at svn.cpp:340
#5 0x0000000000423f53 in recursiveDumpDir (txn=txn@entry=0x189ef30, fs_root=fs_root@entry=0xd3f6f0, pathname=..., finalPathName=..., pool=pool@entry=0xda01d8) at svn.cpp:335
#6 0x0000000000423f53 in recursiveDumpDir (txn=txn@entry=0x189ef30, fs_root=fs_root@entry=0xd3f6f0, pathname=..., finalPathName=..., pool=pool@entry=0xd8f1c8) at svn.cpp:335
#7 0x0000000000423f53 in recursiveDumpDir (txn=txn@entry=0x189ef30, fs_root=fs_root@entry=0xd3f6f0, pathname=..., finalPathName=..., pool=pool@entry=0xdc6208) at svn.cpp:335
#8 0x0000000000423f53 in recursiveDumpDir (txn=txn@entry=0x189ef30, fs_root=fs_root@entry=0xd3f6f0, pathname=..., finalPathName=..., pool=pool@entry=0xd6b188) at svn.cpp:335
#9 0x0000000000423f53 in recursiveDumpDir (txn=txn@entry=0x189ef30, fs_root=fs_root@entry=0xd3f6f0, pathname=..., finalPathName=..., pool=pool@entry=0xdbb1f8) at svn.cpp:335
#10 0x0000000000423f53 in recursiveDumpDir (txn=txn@entry=0x189ef30, fs_root=fs_root@entry=0xd3f6f0, pathname=..., finalPathName=..., pool=pool@entry=0xdb11e8) at svn.cpp:335
#11 0x0000000000423f53 in recursiveDumpDir (txn=txn@entry=0x189ef30, fs_root=fs_root@entry=0xd3f6f0, pathname=..., finalPathName=..., pool=pool@entry=0xd861b8) at svn.cpp:335
#12 0x0000000000423f53 in recursiveDumpDir (txn=txn@entry=0x189ef30, fs_root=0xd3f6f0, pathname=..., finalPathName=..., pool=pool@entry=0xcf81f8) at svn.cpp:335
#13 0x0000000000426e07 in SvnRevision::exportInternal (this=this@entry=0x7fff44b63ca0, key=key@entry=0x189d8d8 "/FusionWorks/branches/FW_5_5_0_226", change=change@entry=0xcf9780,
path_from=<optimized out>, path_from@entry=0xd4ee87 "/branches", rev_from=rev_from@entry=149644, current=..., rule=..., matchRules=...) at svn.cpp:914
#14 0x000000000042b56b in SvnRevision::exportDispatch (this=this@entry=0x7fff44b63ca0, key=key@entry=0x189d8d8 "/branches/FW_5_5_0_226", change=change@entry=0xcf9780,
path_from=0xd4ee87 "/FusionWorks/branches", rev_from=149644, changes=changes@entry=0xcf96b0, current=..., rule=..., matchRules=..., pool=pool@entry=0xd4ee08) at svn.cpp:716
#15 0x000000000042cff0 in SvnRevision::exportEntry (this=this@entry=0x7fff44b63ca0, key=0x189d8d8 "/FusionWorks/branches/FW_5_5_0_226", change=0xcf9780, changes=0xcf96b0) at svn.cpp:666
#16 0x000000000042f091 in SvnRevision::prepareTransactions (this=this@entry=0x7fff44b63ca0) at svn.cpp:536
#17 0x000000000042f660 in SvnPrivate::exportRevision (this=<optimized out>, revnum=revnum@entry=149645) at svn.cpp:444
#18 0x000000000042f7ec in Svn::exportRevision (this=this@entry=0x7fff44b63e40, revnum=revnum@entry=149645) at svn.cpp:154
#19 0x0000000000407754 in main (argc=6, argv=<optimized out>) at main.cpp:276
(gdb)
I can't get the build working, see the following
$ cat .qmake.stash
QMAKE_CXX.QT_COMPILER_STDCXX = 201402L
QMAKE_CXX.QMAKE_GCC_MAJOR_VERSION = 8
QMAKE_CXX.QMAKE_GCC_MINOR_VERSION = 2
QMAKE_CXX.QMAKE_GCC_PATCH_VERSION = 1
QMAKE_CXX.COMPILER_MACROS = \
QT_COMPILER_STDCXX \
QMAKE_GCC_MAJOR_VERSION \
QMAKE_GCC_MINOR_VERSION \
QMAKE_GCC_PATCH_VERSION
QMAKE_CXX.INCDIRS = \
/usr/include/c++/8.2.1 \
/usr/include/c++/8.2.1/x86_64-pc-linux-gnu \
/usr/include/c++/8.2.1/backward \
/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include \
/usr/local/include \
/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include-fixed \
/usr/include
QMAKE_CXX.LIBDIRS = \
/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 \
/usr/lib \
/lib
make
fails with the following:
[...] # snipped
g++ -Wl,-O1 -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -o ../svn-all-fast-export ruleparser.o repository.o svn.o main.o CommandLineParser.o -L -lsvn_fs-1 -lsvn_repos-1 -lapr-1 -lsvn_subr-1 -lQt5Core -lpthread
/usr/bin/ld: svn.o: undefined reference to symbol 'svn_fs_file_contents'
/usr/bin/ld: /usr/lib/libsvn_fs-1.so.0: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:203: ../svn-all-fast-export] Error 1
make[1]: Leaving directory '/home/kartik/Projects/svn2git/src'
make: *** [Makefile:47: sub-src-make_first] Error 2
I am getting the error: Exporting revision 1 "/content/" did not match any rules; cannot continue
/content/ is a directory in trunk and each branch, but why would it be part of the match? I'm really confused.
My SVN rules are:
#
# Declare the repositories we know about:
#
create repository pw_content
end repository
#
# Declare the rules
# Note: rules must end in a slash
#
match /trunk/
repository pw_content
branch master
end match
match (branch|tag)/([^/]+)/
repository pw_content
branch \2
end match
We are facing issues after migration when users add .gitattributes
file to keep LF for shell scripts when working on Windows platform.
"Perfect world" expectation would be that svn-all-fast-export inspected svn:eol-style Subversion property and applies ".gitattributes" for each imported commit.
What is the recommended way to get all text files with LF EOL in repository itself, so that Git attributes can be used seamlessly after migration when committing from different EOL-style platforms?
Seems like the application is ignoring my identity-map file. I'm using the git-svn syntax in the file, and when i run svn-all-fast-export, it seems like it either thinks I'm not passing anything to that part or ignores it completely. any ideas?
Any help getting this to build on osx?
Thanks to brew, I think I have almost everything installed, but I'm getting the error below. I think I'm missing some svn libs, but I'm not certain how to get them. Hoping someone else has done this before and can help.
Thanks!
qmake && make
Info: creating stash file /Users/macfly/dmjohns/svn-all-fast-export/.qmake.stash
cd src/ && ( test -e Makefile || /usr/local/opt/qt/bin/qmake -o Makefile /Users/macfly/dmjohns/svn-all-fast-export/src/src.pro ) && /Library/Developer/CommandLineTools/usr/bin/make -f Makefile
Project WARNING: Qt has only been tested with version 10.13 of the platform SDK, you're using 10.14.
Project WARNING: This is an unsupported configuration. You may experience build issues, and by using
Project WARNING: the 10.14 SDK you are opting in to new features that Qt has not been prepared for.
Project WARNING: E.g., 10.14 enables dark mode and layer-backed views, which Qt 5.11 does not support.
Project WARNING: Please downgrade the SDK you use to build your app to version 10.13, or configure
Project WARNING: with CONFIG+=sdk_no_version_check when running qmake to silence this warning.
/Library/Developer/CommandLineTools/usr/bin/clang++ -c -pipe -stdlib=libc++ -O2 -std=gnu++11 -arch x86_64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -mmacosx-version-min=10.11 -Wall -W -fPIC -DVER=""83d6f8d2b28e7db5c1ac959df96fc74af363ab92"" -DQT_NO_DEBUG -DQT_CORE_LIB -I. -I. -I/usr/include/subversion-1 -I/usr/local/include/subversion-1 -I/usr/include/apr-1.0 -I/usr/include/apr-1 -I/usr/local/include/apr-1 -I/usr/local/Cellar/qt/5.11.2/lib/QtCore.framework/Headers -I. -I/usr/local/Cellar/qt/5.11.2/mkspecs/macx-clang -F/usr/local/Cellar/qt/5.11.2/lib -o ruleparser.o ruleparser.cpp
/Library/Developer/CommandLineTools/usr/bin/clang++ -c -pipe -stdlib=libc++ -O2 -std=gnu++11 -arch x86_64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -mmacosx-version-min=10.11 -Wall -W -fPIC -DVER=""83d6f8d2b28e7db5c1ac959df96fc74af363ab92"" -DQT_NO_DEBUG -DQT_CORE_LIB -I. -I. -I/usr/include/subversion-1 -I/usr/local/include/subversion-1 -I/usr/include/apr-1.0 -I/usr/include/apr-1 -I/usr/local/include/apr-1 -I/usr/local/Cellar/qt/5.11.2/lib/QtCore.framework/Headers -I. -I/usr/local/Cellar/qt/5.11.2/mkspecs/macx-clang -F/usr/local/Cellar/qt/5.11.2/lib -o repository.o repository.cpp
/Library/Developer/CommandLineTools/usr/bin/clang++ -c -pipe -stdlib=libc++ -O2 -std=gnu++11 -arch x86_64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -mmacosx-version-min=10.11 -Wall -W -fPIC -DVER=""83d6f8d2b28e7db5c1ac959df96fc74af363ab92"" -DQT_NO_DEBUG -DQT_CORE_LIB -I. -I. -I/usr/include/subversion-1 -I/usr/local/include/subversion-1 -I/usr/include/apr-1.0 -I/usr/include/apr-1 -I/usr/local/include/apr-1 -I/usr/local/Cellar/qt/5.11.2/lib/QtCore.framework/Headers -I. -I/usr/local/Cellar/qt/5.11.2/mkspecs/macx-clang -F/usr/local/Cellar/qt/5.11.2/lib -o svn.o svn.cpp
svn.cpp:528:13: warning: 'svn_fs_paths_changed2' is deprecated [-Wdeprecated-declarations]
SVN_ERR(svn_fs_paths_changed2(&changes, fs_root, pool));
^
/usr/local/include/subversion-1/svn_fs.h:1754:1: note: 'svn_fs_paths_changed2' has been explicitly marked deprecated here
SVN_DEPRECATED
^
/usr/local/include/subversion-1/svn_types.h:56:42: note: expanded from macro 'SVN_DEPRECATED'
^
svn.cpp:569:13: warning: 'svn_fs_revision_proplist' is deprecated [-Wdeprecated-declarations]
SVN_ERR(svn_fs_revision_proplist(&revprops, fs, revnum, pool));
^
/usr/local/include/subversion-1/svn_fs.h:2854:1: note: 'svn_fs_revision_proplist' has been explicitly marked deprecated here
SVN_DEPRECATED
^
/usr/local/include/subversion-1/svn_types.h:56:42: note: expanded from macro 'SVN_DEPRECATED'
^
2 warnings generated.
/Library/Developer/CommandLineTools/usr/bin/clang++ -c -pipe -stdlib=libc++ -O2 -std=gnu++11 -arch x86_64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -mmacosx-version-min=10.11 -Wall -W -fPIC -DVER=""83d6f8d2b28e7db5c1ac959df96fc74af363ab92"" -DQT_NO_DEBUG -DQT_CORE_LIB -I. -I. -I/usr/include/subversion-1 -I/usr/local/include/subversion-1 -I/usr/include/apr-1.0 -I/usr/include/apr-1 -I/usr/local/include/apr-1 -I/usr/local/Cellar/qt/5.11.2/lib/QtCore.framework/Headers -I. -I/usr/local/Cellar/qt/5.11.2/mkspecs/macx-clang -F/usr/local/Cellar/qt/5.11.2/lib -o main.o main.cpp
/Library/Developer/CommandLineTools/usr/bin/clang++ -c -pipe -stdlib=libc++ -O2 -std=gnu++11 -arch x86_64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -mmacosx-version-min=10.11 -Wall -W -fPIC -DVER=""83d6f8d2b28e7db5c1ac959df96fc74af363ab92"" -DQT_NO_DEBUG -DQT_CORE_LIB -I. -I. -I/usr/include/subversion-1 -I/usr/local/include/subversion-1 -I/usr/include/apr-1.0 -I/usr/include/apr-1 -I/usr/local/include/apr-1 -I/usr/local/Cellar/qt/5.11.2/lib/QtCore.framework/Headers -I. -I/usr/local/Cellar/qt/5.11.2/mkspecs/macx-clang -F/usr/local/Cellar/qt/5.11.2/lib -o CommandLineParser.o CommandLineParser.cpp
/Library/Developer/CommandLineTools/usr/bin/clang++ -stdlib=libc++ -headerpad_max_install_names -arch x86_64 -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -mmacosx-version-min=10.11 -Wl,-rpath,@executable_path/Frameworks -o ../svn-all-fast-export.app/Contents/MacOS/svn-all-fast-export ruleparser.o repository.o svn.o main.o CommandLineParser.o -F/usr/local/Cellar/qt/5.11.2/lib -lsvn_fs-1 -lsvn_repos-1 -lapr-1 -lsvn_subr-1 -framework QtCore -framework DiskArbitration -framework IOKit
ld: library not found for -lsvn_fs-1
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [../svn-all-fast-export.app/Contents/MacOS/svn-all-fast-export] Error 1
make: *** [sub-src-make_first] Error 2
Hello,
we have the following situation:
Applying the svn2git tool with the option --empty-dirs
, the commit on the SVN branch added, as expected, two .gitignore
files.
However, the subsequent merge commits did not carry this information, and I sort-of see the point: only the content of the SVN merge commit got transcribed and reproduced, but it creates this weird situation where the master (ex trunk) does not contain exactly the same content at the other branch, despite having done a merge.
Would there be any work-around on this? The easy solution is to reperform the merge, at the end of the migration, where this time the empty .gitignore will be picked, but if there would be a cleaner way, that would be nice.
Thanks in advance, and thank you for the tool!
Hi there,
I'm currently working on migrating a huge SVN repo into Git using your tool. I found it to do the job quite nicely (thanks for that!), however I have stumbled across one slight issue:
I'm passing the following parameters: --identity-map xyz.authors --rules xyz.rules --add-metadata --stats --svn-ignore --commit-interval 1000 /tmp/svn/
However, in the resulting repo, I see no gitignore files although svn ignore properties are existing in the repo.
Am I doing something wrong or is there some way to further troubleshoot this problem?
Thanks in advance and kind regards,
Philipp
I have to migrate an old svn repository to git.
I tried if my rules are working with --dry-run until all looked good, but when i'm starting the tool without the dry-run parameter it freezes at the first revision with content
Exporting revision 7487 nothing to do
Exporting revision 7488 nothing to do
Exporting revision 7489 .
At that spot it's frozen for hours without any relevant cpu or memory usage
After a while i tried to start it with --debug-rules and could see that it founds a match for first file in the first revision with content and then does nothing for ... some time
Exporting revision 7487 nothing to do
Exporting revision 7488 nothing to do
Exporting revision 7489 rev 7489 /TMS/trunk/tms/.classpath matched rule: "../svn2git.rules:4 /TMS/trunk/" exporting.
.add/change file ( /TMS/trunk/tms/.classpath -> "master" "tms/.classpath"`
I call
~/svn2git/svn-all-fast-export.exe --debug-rules --identity-map ../trunk/authors-transform.txt --rules ../svn2git.rules local
and my current rules are pretty simple
create repository tms
end repository
match /TMS/trunk/
repository tms
branch master
end match
match /TMS/branches/Feature_Branches/([^/]+)/
repository tms
branch feature/\1
end match
match /TMS/branches/([^/]+)/
repository tms
branch \1
end match
match /TMS/tags/([^/]+)/
repository tms
branch ref/tags/\1
annotated true
end match
I ran into this warning in CI logs:
bats warning: duplicate test name(s) in /home/runner/work/svn2git/svn2git/test/empty-dirs.bats: test_branching_with_svn-2dignore-2c_svn-2dbranches_and_empty-2ddirs_parameter_should_not_replace_filled_-2egitignore_files_with_empty_ones
I can confirm that we have two tests named branching with svn-ignore, svn-branches and empty-dirs parameter should not replace filled .gitignore files with empty ones
:
# git --no-pager grep -n 'should not replace'
test/empty-dirs.bats:1031:@test 'branching with svn-ignore, svn-branches and empty-dirs parameter should not replace filled .gitignore files with empty ones' {
test/empty-dirs.bats:1063:@test 'branching with svn-ignore, svn-branches and empty-dirs parameter should not replace filled .gitignore files with empty ones' {
Both tests were added in fcb62af.
CC @Vampire
Hi,
I am running the command below but it appears that there is an assertion failure. I tried with and without a leading / on the URI without any luck.
svn-all-fast-export --identity-map=authors.txt --rules=standard-rules-development.txt --stats --add-metadata https://svn.com/svn/MyRepo Invoked as:' svn-all-fast-export --identity-map=authors.txt --rules=standard-rules-development.txt --stats --add-metadata https://svn.com/svn/MyRepo' Loading rules from: "standard-rules-development.txt" Loading rules from "standard-rules-development.txt" Creating new repository "MyRepo" svn-all-fast-export: subversion/libsvn_subr/dirent_uri.c:972: svn_dirent_join: Assertion 'svn_dirent_is_canonical(base, pool)' failed. Aborted (core dumped)
~/folder$ svn-all-fast-export --identity-map=../../authors.txt --rules=../rules.txt --stats --add-metadata https://svn.repo.com/folder-with-trunk-branches-tags
Invoked as:' svn-all-fast-export --identity-map=../../authors.txt --rules=../rules.txt --stats --add-metadata https://svn.repo.com/folder-with-trunk-branches-tags'
Loading rules from: "../rules.txt"
Loading rules from "../rules.txt"
Creating new repository "my-new-git-repo"
svn-all-fast-export: /build/subversion-yttRtj/subversion-1.9.3/subversion/libsvn_subr/dirent_uri.c:972: svn_dirent_join: Assertion `svn_dirent_is_canonical(base, pool)' failed.
Aborted (core dumped)
../rules.txt < https://gist.github.com/Vampire/f48bd7e3667a0cc0cbe1b5746c492be8
../../authors.txt < list of user = Name <[email protected]>
Step 4/7 : ADD . /usr/local/svn2git
---> Using cache
---> 8b89fdb5ba3b
Step 5/7 : RUN cd /usr/local/svn2git && qmake && make
---> Running in 7b3a74125f6e
Usage: /usr/lib/x86_64-linux-gnu/qt4/bin/qmake [mode] [options] [files]
QMake has two modes, one mode for generating project files based on
some heuristics, and the other for generating makefiles. Normally you
shouldn't need to specify a mode, as makefile generation is the default
mode for qmake, but you may use this to test qmake on an existing project
Mode:
-project Put qmake into project file generation mode
In this mode qmake interprets files as files to
be built,
defaults to *.c; *.ui; *.y; *.l; *.ts; *.xlf; *.qrc; *.h; *.hpp; *.hh; *.hxx; *.H; *.cpp; *.cc; *.cxx; *.C
Note: The created .pro file probably will
need to be edited. For example add the QT variable to
specify what modules are required.
Warnings Options:
-Wnone Turn off all warnings; specific ones may be re-enabled by
later -W options
The command '/bin/sh -c cd /usr/local/svn2git && qmake && make' returned a non-zero code: 1
Hi! I'm getting compile error with 1.0.15 that I'm not getting with 1.0.13. Any ideas what's wrong?
x86_64-pc-linux-gnu-g++ -c -march=core2 -msse4.1 --param l1-cache-line-size=64 --param l1-cache-size=32 --param l2-cache-size=6144 -O2 -pipe -Wall -W -D_REENTRANT -fPIC -DVER="\"1.0.15\"" -DQT_NO_DEBUG -DQT_CORE_LIB -I. -I. -isystem /usr/include/subversion-1 -isystem /usr/include/apr-1 -isystem /usr/include/qt5 -isystem /usr/include/qt5/QtCore -I. -I/usr/lib64/qt5/mkspecs/linux-g++ -o repository.o repository.cpp
repository.cpp: In member function ‘virtual void FastImportRepository::restoreBranchNotes()’:
repository.cpp:471:34: error: variable ‘QDataStream branchNotesStream’ has initializer but incomplete type
QDataStream branchNotesStream(&branchNotesFile);
^
repository.cpp: In member function ‘virtual void FastImportRepository::saveBranchNotes()’:
repository.cpp:798:34: error: variable ‘QDataStream branchNotesStream’ has initializer but incomplete type
QDataStream branchNotesStream(&branchNotesFile);
^
(From trying to update the Gentoo package of it, hence the color and Gentoo bits)
svn2git-1.0.15-build-log.txt
I'm trying to convert a single SVN repo over to a single git repo. After migration of the svn repository to git I have noticed that git does not seem to follow svn copy operations so the resulting history is much briefer than I expect.
To illustrate - The existing history can be viewed in tortoise svn if you uncheck the "Stop on copy/rename" checkbox while viewing a folder's history but it cannot be viewed in Git after migration.
root
a
b
c
parent project
b
c
Projects b and c were recently copied under parent-proj as part of a restructuring effort with the intention of eventually deleting them from their old locations under root. After migration the resulting git repo is missing all of the history that originated from /b and /c before the move.
I have created a SO post for this here - https://stackoverflow.com/questions/53704582/retaining-svn-copy-history-when-converting-to-git-using-kde-svn2git-tool
docker build -t svn2git .
on master seems broken
g++ -m64 -Wl,-O1 -o ../svn-all-fast-export ruleparser.o repository.o svn.o main.o CommandLineParser.o -L/usr/lib/x86_64-linux-gnu -lsvn_fs-1 -lsvn_repos-1 -lapr-1 -lsvn_subr-1 -lQtCore -lpthread
ruleparser.o: In function `QList<Rules::Repository::Branch>::free(QListData::Data*) [clone .isra.0]':
ruleparser.cpp:(.text+0x109): undefined reference to `operator delete(void*, unsigned long)'
ruleparser.o: In function `QList<Rules::Match::Substitution>::free(QListData::Data*) [clone .isra.0]':
ruleparser.cpp:(.text+0x1a1): undefined reference to `operator delete(void*, unsigned long)'
ruleparser.o: In function `QList<Rules::Repository>::free(QListData::Data*) [clone .isra.0]':
ruleparser.cpp:(.text+0x37b): undefined reference to `operator delete(void*, unsigned long)'
ruleparser.o: In function `QList<Rules::Match>::free(QListData::Data*) [clone .isra.0]':
ruleparser.cpp:(.text+0x4b8): undefined reference to `operator delete(void*, unsigned long)'
ruleparser.o: In function `QList<QList<Rules::Match> >::free(QListData::Data*) [clone .isra.0]':
ruleparser.cpp:(.text+0x599): undefined reference to `operator delete(void*, unsigned long)'
ruleparser.o:ruleparser.cpp:(.text+0xbd4): more undefined references to `operator delete(void*, unsigned long)' follow
collect2: error: ld returned 1 exit status
make[1]: *** [../svn-all-fast-export] Error 1
Makefile:110: recipe for target '../svn-all-fast-export' failed
make[1]: Leaving directory '/usr/local/svn2git/src'
Makefile:39: recipe for target 'sub-src-make_default' failed
make: *** [sub-src-make_default] Error 2
The command '/bin/sh -c cd /usr/local/svn2git && qmake && make' returned a non-zero code: 2
The structure of my svn repository looks similar to this:
/Poject/
trunk/
dir1/
dir2/
dir3/
...
dir<n>/
branches/
branch_1/
dir1/
dir2/
dir3/
branch_2/
dir1/
dir2/
dir3/
...
dir<n>/
tags/
tag_1/
dir1/
dir2/
dir3/
tag_2/
dir1/
dir2/
dir3/
...
dir<n>/
In reality there are several branches and several tags. Most of the tags were generated after the branches and the name of these tags is the same as the corresponding branch. The repository has over 26000 revisions.
The svn-repository is splitted into several git repositories (in total 8) where some of the directories (dir1, ..., dir<n>
) were grouped together, for example:
Here is the rules file for this repository:
# Declare the repositories we know about:
#
create repository git_repo_1
end repository
create repository git_repo_2
end repository
create repository git_repo_3
end repository
#
# git_repo_1
#
# git_repo_1 = dir1 + dir4 +dir5
#
match /Project/trunk/(dir1|dir4|dir5)/
repository git_repo_1
prefix \1/
branch master
end match
match /Project/branches/([^/]+)/(dir1|dir4|dir5)/
repository git_repo_1
prefix \2/
branch \1
end match
match /Project/tags/([^/]+)/(dir1|dir4|dir5)/
repository git_repo_1
prefix \2/
branch refs/tags/v\1
annotated true
end match
#
# git_repo_2 = dir2
#
match /Project/trunk/dir2/
repository git_repo_2
branch master
end match
match /Project/branches/([^/]+)/dir2/
repository git_repo_2
branch \1
end match
match /Project/tags/([^/]+)/dir2/
repository git_repo_2
branch refs/tags/v\1
annotated true
end match
#
# git_repo_3
#
# git_repo_3= dir6 + dir10
#
match /Project/trunk/(dir6 |dir10)/
repository git_repo_3
prefix \1/
branch master
end match
match /Project/branches/([^/]+)/(dir6 |dir10)/
repository git_repo_3
prefix \2/
branch \1
end match
match /Project/tags/([^/]+)/(dir6 |dir10)/
repository git_repo_3
prefix \2/
branch refs/tags/v\1
annotated true
end match
match /
end match
Results:
After performing the migration not all the tags were fetched into the corresponding git-repositories.
In the logs there was found a message the following message for some repositories: recursiveDumpDir: "/Project/tags/tag_1/dir2/" skip entry for different/ignored repository
Questions:
Why are the tags not fetched?
Why did the message recursiveDumpDir
appear in the logs even though none of the match definitions contains the parameter action export
and all them have defined the parameter repository
?
I'm trying to import a very large and ugly SVN repo on the order of 22GB. In this commit the following are deleted:
new/
dir1
dir2
dir3
And the following are copied from points in the history
new/
subdir/
dir1
dir2
dir4
After running for almost 4 hours I get the following error:
terminate called after throwing an instance of 'bad_alloc'
what(): std::bad_alloc
the fast_import_crash_xxx says:
fata: EOF in data (xxxx bytes remaining)
On Ubuntu 16.04 LTS:
Error:
$ qmake && make
cd src/ && ( test -e Makefile || /usr/lib/x86_64-linux-gnu/qt5/bin/qmake /home/me/src/svn2git/src/src.pro -o Makefile ) && make -f Makefile
make[1]: Entering directory '/home/me/src/svn2git/src'
g++ -c -m64 -pipe -O2 -Wall -W -D_REENTRANT -DVER=""5caa913b3ac6fea8ea3a92558889a12e7feed21e"" -DQT_NO_DEBUG -DQT_CORE_LIB -I/usr/share/qt4/mkspecs/linux-g++-64 -I. -I/usr/include/qt4/QtCore -I/usr/include/qt4 -I. -I/usr/include/subversion-1 -I/usr/local/include/subversion-1 -I/usr/include/apr-1.0 -I/usr/include/apr-1 -I/usr/local/include/apr-1 -I. -o ruleparser.o ruleparser.cpp
ruleparser.cpp:18:23: fatal error: QTextStream: No such file or directory
compilation terminated.
Makefile:211: recipe for target 'ruleparser.o' failed
make[1]: *** [ruleparser.o] Error 1
make[1]: Leaving directory '/home/me/src/svn2git/src'
Makefile:42: recipe for target 'sub-src-make_first' failed
make: *** [sub-src-make_first] Error 2
Dependencies are installed:
...with the exception that qt-qmake needed to be qt5-qmake (Documentation issue):
`$ sudo apt-get install qt-qmake' #this doesn't work
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package qt-qmake
$ apt-cache policy build-essential subversion git qtchooser qt5-qmake qt5-default libapr1 libapr1-dev libsvn-dev
build-essential:
Installed: 12.1ubuntu2
Candidate: 12.1ubuntu2
Version table:
*** 12.1ubuntu2 500
500 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
100 /var/lib/dpkg/status
subversion:
Installed: 1.9.3-2ubuntu1
Candidate: 1.9.3-2ubuntu1
Version table:
*** 1.9.3-2ubuntu1 500
500 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
100 /var/lib/dpkg/status
git:
Installed: 1:2.7.4-0ubuntu1
Candidate: 1:2.7.4-0ubuntu1
Version table:
*** 1:2.7.4-0ubuntu1 500
500 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
100 /var/lib/dpkg/status
qtchooser:
Installed: 52-gae5eeef-2build1gcc5.2gcc5.2
Candidate: 52-gae5eeef-2build1
Version table:
*** 52-gae5eeef-2build1~gcc5.2 500
500 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
100 /var/lib/dpkg/status
qt5-qmake:
Installed: 5.5.1+dfsg-16ubuntu7.2
Candidate: 5.5.1+dfsg-16ubuntu7.2
Version table:
*** 5.5.1+dfsg-16ubuntu7.2 500
500 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
100 /var/lib/dpkg/status
5.5.1+dfsg-16ubuntu7 500
500 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
qt5-default:
Installed: 5.5.1+dfsg-16ubuntu7.2
Candidate: 5.5.1+dfsg-16ubuntu7.2
Version table:
*** 5.5.1+dfsg-16ubuntu7.2 500
500 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages
100 /var/lib/dpkg/status
5.5.1+dfsg-16ubuntu7 500
500 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
libapr1:
Installed: 1.5.2-3
Candidate: 1.5.2-3
Version table:
*** 1.5.2-3 500
500 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
100 /var/lib/dpkg/status
libapr1-dev:
Installed: 1.5.2-3
Candidate: 1.5.2-3
Version table:
*** 1.5.2-3 500
500 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
100 /var/lib/dpkg/status
libsvn-dev:
Installed: 1.9.3-2ubuntu1
Candidate: 1.9.3-2ubuntu1
Version table:
*** 1.9.3-2ubuntu1 500
500 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
100 /var/lib/dpkg/status
I do see:
$ ll /usr/lib/x86_64-linux-gnu/qt5/bin/qmake
-rwxr-xr-x 1 root root 3071144 Sep 19 2016 /usr/lib/x86_64-linux-gnu/qt5/bin/qmake*
Can you assist?
I've successfully converted a large SVN repo.
Only use-case issue I'm facing is that all the matching SVN folders come together in the Git root folder.
SVN
--Product1/branches/1.0
----file1.txt
--Product2/branches/1.0
----file2.txt
Output to:
--/
--file1.txt
--file2.txt
I would like to specify a subfolder within the output repo, like:
match /([^/]+)/branches/([^/]+)/
repository git-repo
branch \2
directory \1 -- statement example
end match
Outputting as:
--Product1/file1.txt
--Product2/file2.txt
Hi!
Are there any plans to add CI here? Would you welcome PRs with basic GitHub Actions or Travis integration?
Best, Sebastian
[When using dry-run]
As trivial as they come: first run, you sort of expect it to git init the directory for you if its absent, partly because of the amount of text to declare one:
create repository foo
end repository
but the absence of the repos still causes progress to halt, preventing you from doing a dry-run...
The use of the term "name" also sets up a false hope that the above would create a "foo.git" folder for me and use that.
I'm trying to migrate a huge svn repo (76G in size). It crashes in the middle after about 24 hours.
fast-import crash report:
fast-import process: 15424
parent process : 23192
at Tue Mar 20 09:41:13 2018
fatal: core git rejected index objects/pack/pack-2aa286062dcf4aa1d6bed58a2693a6cd00dba5a6.idx
Most Recent Commands Before Crash:
....
Active Branch LRU
-----------------
active_branches = 1 cur, 5 max
pos clock name
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1) 9999 refs/heads/master
Inactive Branches
-----------------
refs/heads/master:
status : active loaded
tip commit : de565fa294b19e927915f4343d192ed36aed7c2d
old tree : 0572758db74396179eb9473ab8e0fa881cf4cc33
cur tree : 0572758db74396179eb9473ab8e0fa881cf4cc33
commit clock: 9999
last pack : 0
Not quite sure why the repo is that big tbh, cause it has only 19000 revisions. and we have other repo that has way more revisions but only a few G.
I'm evaluating how to migrate our svn-repositories to git. At the moment we have multi project repositories. So we need to export subpathes to different git repositories. The problem is that the structure was changed over the time.
At the moment I try the following approach:
run log -xml -v <mySubPath>
Let explain it on a simple example with the following history
I'm unsure if this is possible with svn-all-fast-export. Can someone give me a hint?
It may be just my ignorance, but even after installing the whole QT (which takes longer than Visual Studio!) I still cannot build it on my Windows machine. Anyone who publishes Windows build instructions and/or makes it more user friendly will have my blessing!
I had to migrate a large SVN repo but run into trouble.
After analyzing the problem a bit, it seems that SVN copy operations for directories which replace
older directories are problematic, because svn2git does not delete all the old content in this case
(as SVN does it).
Are there any options to work around (or resolve) the problem? Will it be possible to fix this in the tool?
To illustrate the problem, I created a very small test repository showing the problem. The output of "svn log -v" can be found in attached file svn_log.txt.
Basically a file in trunk is created (folder_a/file1.txt), then a branch is created (branch1) from trunk,
the first file is deleted and another file is created (folder_a/file2.txt) and another branch from trunk is created (branch2).
Finally the directory folder_1 from branch2 is copied to branch1 replacing the older directory folder_a.
In SVN the older file folder_a/file1.txt is then (implicitly) deleted. But after migrating to Git it is
still there :-(
Documentation is missing for: annotate, substitute repository and substitute branch, and decoding what they do is a little hard.
I am currently migrating a large SVN repository to Git which has a monolithic TTB structure (meaning: no subprojects, all branches are copies of the entire ^/trunk
). During the conversion, most of the repository should go into a single large Git repository, except for one directory, which should move to a separate repository. I had some problems doing the conversion which I would like to describe first before asking some questions. To demonstrate my problem, I prepared a simplified toy example.
Attached below the issue you will find a bash script (show-issue
) to reproduce the toy example from scratch.
The SVN repository (example.svn
) has two branches, the trunk (^/trunk
) and a feature branch (^/branches/feature
). Both branches have the same directory structure:
.
├── bar.txt
├── baz.txt
├── foo.txt
├── other
│ ├── bar.py
│ ├── baz.py
│ └── foo.py
└── src
├── bar.c
├── baz.c
└── foo.c
As one can see from the following history, the feature branch was branched off the trunk in r4 and both branches received additional commits.
------------------------------------------------------------------------
r8 | msp | 2020-04-26 20:54:27 +0200 (So, 26 Apr 2020) | 1 line
Changed paths:
M /branches/feature/other/bar.py
Add more stuff to other files
------------------------------------------------------------------------
r7 | msp | 2020-04-26 20:54:26 +0200 (So, 26 Apr 2020) | 1 line
Changed paths:
M /branches/feature/bar.txt
M /branches/feature/src/bar.c
Add more stuff to my files
------------------------------------------------------------------------
r6 | msp | 2020-04-26 20:54:26 +0200 (So, 26 Apr 2020) | 1 line
Changed paths:
M /trunk/other/bar.py
Add some stuff to other files
------------------------------------------------------------------------
r5 | msp | 2020-04-26 20:54:26 +0200 (So, 26 Apr 2020) | 1 line
Changed paths:
M /trunk/bar.txt
M /trunk/src/bar.c
Add some stuff to my files
------------------------------------------------------------------------
r4 | msp | 2020-04-26 20:54:25 +0200 (So, 26 Apr 2020) | 1 line
Changed paths:
A /branches/feature (from /trunk:3)
Create feature branch
------------------------------------------------------------------------
r3 | msp | 2020-04-26 20:54:25 +0200 (So, 26 Apr 2020) | 1 line
Changed paths:
A /trunk/other
A /trunk/other/bar.py
A /trunk/other/baz.py
A /trunk/other/foo.py
Add other files
------------------------------------------------------------------------
r2 | msp | 2020-04-26 20:54:25 +0200 (So, 26 Apr 2020) | 1 line
Changed paths:
A /trunk/bar.txt
A /trunk/baz.txt
A /trunk/foo.txt
A /trunk/src
A /trunk/src/bar.c
A /trunk/src/baz.c
A /trunk/src/foo.c
Add my files
------------------------------------------------------------------------
r1 | msp | 2020-04-26 20:54:24 +0200 (So, 26 Apr 2020) | 1 line
Changed paths:
A /branches
A /tags
A /trunk
Create ttb structure
------------------------------------------------------------------------
All files from the example.svn
repository should go into a single large Git repository main.git
, except for the contents of the other
directory, which go to other.git
. This here was my first attempt:
broken.rules
create repository broken/main.git
end repository
create repository broken/other.git
end repository
match /trunk/other/
repository broken/other.git
branch master
end match
match /trunk/
repository broken/main.git
branch master
end match
match /branches/([^/]+)/other/
repository broken/other.git
branch \1
end match
match /branches/([^/]+)/
repository broken/main.git
branch \1
end match
broken.log
+ svn-all-fast-export --debug-rules --add-metadata --rules broken.rules example.svn
WARNING; no identity-map or -domain specified, all commits will use default @localhost email address
Loading rules from: "broken.rules"
Loading rules from "broken.rules"
marksfile "broken/main.git/marks-broken_main.git"
marksfile "broken/other.git/marks-broken_other.git"
Invoked as:' svn-all-fast-export --debug-rules --add-metadata --rules broken.rules example.svn'
The content of the main.git
repository looks ok:
##
## broken/main.git
##
* commit 1585deab140fde1c26e05caa2b3715c066523430
| Author: msp <msp@localhost>
| Date: Sun Apr 26 18:54:26 2020 +0000
|
| Add more stuff to my files
|
| svn path=/branches/feature/; revision=7
|
| bar.txt | 1 +
| src/bar.c | 1 +
| 2 files changed, 2 insertions(+)
|
| * commit faffe7f8e11b5a5077d79ede381250457e9e5014
|/ Author: msp <msp@localhost>
| Date: Sun Apr 26 18:54:26 2020 +0000
|
| Add some stuff to my files
|
| svn path=/trunk/; revision=5
|
| bar.txt | 1 +
| src/bar.c | 1 +
| 2 files changed, 2 insertions(+)
|
* commit f8b88128f0b96a901ae7ca8abc16b3da36232df2
Author: msp <msp@localhost>
Date: Sun Apr 26 18:54:25 2020 +0000
Add my files
svn path=/trunk/; revision=2
bar.txt | 1 +
baz.txt | 1 +
foo.txt | 1 +
src/bar.c | 1 +
src/baz.c | 1 +
src/foo.c | 1 +
6 files changed, 6 insertions(+)
tree of master branch:
100644 blob ce781c47e1cab657df5ca60339bf2a578defda8a bar.txt
100644 blob 76018072e09c5d31c8c6e3113b8aa0fe625195ca baz.txt
100644 blob 257cc5642cb1a054f08cc83f2d943e56fd3ebe99 foo.txt
100644 blob ce781c47e1cab657df5ca60339bf2a578defda8a src/bar.c
100644 blob 76018072e09c5d31c8c6e3113b8aa0fe625195ca src/baz.c
100644 blob 257cc5642cb1a054f08cc83f2d943e56fd3ebe99 src/foo.c
tree of feature branch:
100644 blob f0f5c7c766cdb85df51efef2b02dff568788e0c0 bar.txt
100644 blob 76018072e09c5d31c8c6e3113b8aa0fe625195ca baz.txt
100644 blob 257cc5642cb1a054f08cc83f2d943e56fd3ebe99 foo.txt
100644 blob f0f5c7c766cdb85df51efef2b02dff568788e0c0 src/bar.c
100644 blob 76018072e09c5d31c8c6e3113b8aa0fe625195ca src/baz.c
100644 blob 257cc5642cb1a054f08cc83f2d943e56fd3ebe99 src/foo.c
But the feature branch of the other.git
repository is 'detached' from the master
branch and its tree is incomplete, i.e., it contains only the single file (bar.py
) which was modified after branching (in r8).
##
## broken/other.git
##
* commit 0d7132382ede82f4df2c1fe0c1120b1c69fce221
Author: msp <msp@localhost>
Date: Sun Apr 26 18:54:27 2020 +0000
Add more stuff to other files
svn path=/branches/feature/other/; revision=8
bar.py | 2 ++
1 file changed, 2 insertions(+)
* commit 87ff8db4f665a7a8637029b3d5cb5f4c777176e9
| Author: msp <msp@localhost>
| Date: Sun Apr 26 18:54:26 2020 +0000
|
| Add some stuff to other files
|
| svn path=/trunk/other/; revision=6
|
| bar.py | 1 +
| 1 file changed, 1 insertion(+)
|
* commit 604ca6dd5b379634d18e14f4d776e8d594a01075
Author: msp <msp@localhost>
Date: Sun Apr 26 18:54:25 2020 +0000
Add other files
svn path=/trunk/other/; revision=3
bar.py | 1 +
baz.py | 1 +
foo.py | 1 +
3 files changed, 3 insertions(+)
tree of master branch:
100644 blob ce781c47e1cab657df5ca60339bf2a578defda8a bar.py
100644 blob 76018072e09c5d31c8c6e3113b8aa0fe625195ca baz.py
100644 blob 257cc5642cb1a054f08cc83f2d943e56fd3ebe99 foo.py
tree of feature branch:
100644 blob f0f5c7c766cdb85df51efef2b02dff568788e0c0 bar.py
I learned from UsingSvn2Git that the rule
match /branches/([^/]+)/other/
repository broken/other.git
branch \1
end match
won't be matched in r4 (A /branches/feature (from /trunk:3)
), unless I define a recurse action
match /branches/([^/]+)/$
action recurse
end match
However, I also need an export action for the main repository
match /branches/([^/]+)/
repository broken/main.git
branch \1
end match
and unfortunately svn-all-fast-export
does not support an 'export-and-recurse' action.
The only solution I could come up with to fix that conflict was to create two separate rule files:
main.rules
create repository fixed/main.git
end repository
match /trunk/other/
action ignore
end match
match /trunk/
repository fixed/main.git
branch master
end match
match /branches/([^/]+)/other/
action ignore
end match
match /branches/([^/]+)/
repository fixed/main.git
branch \1
end match
other.rules
create repository fixed/other.git
end repository
match /trunk/other/
repository fixed/other.git
branch master
end match
match /branches/([^/]+)/$
action recurse
end match
match /branches/([^/]+)/other/
repository fixed/other.git
branch \1
end match
match /
action ignore
end match
Initially, the downside of this approach was that I had to call svn-all-fast-export
twice, which doubled the conversion time. But then I discovered (from the source code) a handy feature, namely that you can specify a comma separated list of rules on the command line.
svn-all-fast-export --debug-rules --add-metadata --rules main.rules,other.rules example.svn
The two rule files are not merged, but instead executed independently in parallel. This feature was added in commit a741bdb by @tnyblom. This is an excellent feature IMO and unfortunately poorly documented.
This time, the result looks good:
fixed.log
+ svn-all-fast-export --debug-rules --add-metadata --rules main.rules,other.rules example.svn
WARNING; no identity-map or -domain specified, all commits will use default @localhost email address
Loading rules from: "main.rules"
Loading rules from "main.rules"
Loading rules from: "other.rules"
Loading rules from "other.rules"
marksfile "fixed/main.git/marks-fixed_main.git"
marksfile "fixed/other.git/marks-fixed_other.git"
Invoked as:' svn-all-fast-export --debug-rules --add-metadata --rules main.rules,other.rules example.svn
##
## fixed/main.git
##
* commit 1585deab140fde1c26e05caa2b3715c066523430
| Author: msp <msp@localhost>
| Date: Sun Apr 26 18:54:26 2020 +0000
|
| Add more stuff to my files
|
| svn path=/branches/feature/; revision=7
|
| bar.txt | 1 +
| src/bar.c | 1 +
| 2 files changed, 2 insertions(+)
|
| * commit faffe7f8e11b5a5077d79ede381250457e9e5014
|/ Author: msp <msp@localhost>
| Date: Sun Apr 26 18:54:26 2020 +0000
|
| Add some stuff to my files
|
| svn path=/trunk/; revision=5
|
| bar.txt | 1 +
| src/bar.c | 1 +
| 2 files changed, 2 insertions(+)
|
* commit f8b88128f0b96a901ae7ca8abc16b3da36232df2
Author: msp <msp@localhost>
Date: Sun Apr 26 18:54:25 2020 +0000
Add my files
svn path=/trunk/; revision=2
bar.txt | 1 +
baz.txt | 1 +
foo.txt | 1 +
src/bar.c | 1 +
src/baz.c | 1 +
src/foo.c | 1 +
6 files changed, 6 insertions(+)
tree of master branch:
100644 blob ce781c47e1cab657df5ca60339bf2a578defda8a bar.txt
100644 blob 76018072e09c5d31c8c6e3113b8aa0fe625195ca baz.txt
100644 blob 257cc5642cb1a054f08cc83f2d943e56fd3ebe99 foo.txt
100644 blob ce781c47e1cab657df5ca60339bf2a578defda8a src/bar.c
100644 blob 76018072e09c5d31c8c6e3113b8aa0fe625195ca src/baz.c
100644 blob 257cc5642cb1a054f08cc83f2d943e56fd3ebe99 src/foo.c
tree of feature branch:
100644 blob f0f5c7c766cdb85df51efef2b02dff568788e0c0 bar.txt
100644 blob 76018072e09c5d31c8c6e3113b8aa0fe625195ca baz.txt
100644 blob 257cc5642cb1a054f08cc83f2d943e56fd3ebe99 foo.txt
100644 blob f0f5c7c766cdb85df51efef2b02dff568788e0c0 src/bar.c
100644 blob 76018072e09c5d31c8c6e3113b8aa0fe625195ca src/baz.c
100644 blob 257cc5642cb1a054f08cc83f2d943e56fd3ebe99 src/foo.c
##
## fixed/other.git
##
* commit e76a6a71d849b1f1716d20e7132bb9433648da67
| Author: msp <msp@localhost>
| Date: Sun Apr 26 18:54:27 2020 +0000
|
| Add more stuff to other files
|
| svn path=/branches/feature/other/; revision=8
|
| bar.py | 1 +
| 1 file changed, 1 insertion(+)
|
| * commit 87ff8db4f665a7a8637029b3d5cb5f4c777176e9
|/ Author: msp <msp@localhost>
| Date: Sun Apr 26 18:54:26 2020 +0000
|
| Add some stuff to other files
|
| svn path=/trunk/other/; revision=6
|
| bar.py | 1 +
| 1 file changed, 1 insertion(+)
|
* commit 604ca6dd5b379634d18e14f4d776e8d594a01075
Author: msp <msp@localhost>
Date: Sun Apr 26 18:54:25 2020 +0000
Add other files
svn path=/trunk/other/; revision=3
bar.py | 1 +
baz.py | 1 +
foo.py | 1 +
3 files changed, 3 insertions(+)
tree of master branch:
100644 blob ce781c47e1cab657df5ca60339bf2a578defda8a bar.py
100644 blob 76018072e09c5d31c8c6e3113b8aa0fe625195ca baz.py
100644 blob 257cc5642cb1a054f08cc83f2d943e56fd3ebe99 foo.py
tree of feature branch:
100644 blob f0f5c7c766cdb85df51efef2b02dff568788e0c0 bar.py
100644 blob 76018072e09c5d31c8c6e3113b8aa0fe625195ca baz.py
100644 blob 257cc5642cb1a054f08cc83f2d943e56fd3ebe99 foo.py
Having to maintain two different rule files makes things a little bit complicated (in particular, since my real repository is not as simple as my example) and I was asking myself whether there was a different way to solve my problem using a single rules file, which I might have missed?
If not, would it be a great effort @tnyblom to add a new 'export-and-recurse' action and would this make it possible to have all rules in a single file (as in the broken attempt) using something like the following?
match /branches/([^/]+)/$
repository broken/other.git
branch \1
action export-and-recurse
end match
I run svn-all-fast-export --rules standardlayout.rules --identity-map mapping.txt --svn-ignore --add-metadata --stats svn-repository
And it is processed up to latest revision 19018 according logs, and git bare repository size sounds correct.
But "master" reference is stuck to revision 10090, git log master
last commit shown is svn path=/trunk/; revision=10090
. Branches and tags created after that revision are not visible either.
If I run svn-all-fast-export
again, it starts back to revision 10091... Investigating git bare repository, I find out as many objects/pack/tmp_pack_XXXX
large files.
Do I guess right commits have not been "flushed" into repository ? What should I do to complete migration and get "recent" references visible ?
Thank you in advance for your help
Malformed rules shouldn't cause a core dump, it should just be an error, and the program should exit.
This was caused by a typo, in the rule set, instead of 'end match', I had 'emd match'
(gdb) bt
#0 0x00007f9fa3cbc1d7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007f9fa3cbd8c8 in __GI_abort () at abort.c:90
#2 0x00007f9fa4af57b4 in qt_message_output (msgType=msgType@entry=QtFatalMsg, buf=<optimized out>) at global/qglobal.cpp:2323
#3 0x00007f9fa4af5959 in qt_message(enum QtMsgType, const char *, typedef __va_list_tag __va_list_tag *) (msgType=msgType@entry=QtFatalMsg,
msg=msg@entry=0x4386a0 "Malformed line in rules file: line %d: %s", ap=ap@entry=0x7fff88320fb0) at global/qglobal.cpp:2369
#4 0x00007f9fa4af6261 in qFatal (msg=msg@entry=0x4386a0 "Malformed line in rules file: line %d: %s") at global/qglobal.cpp:2552
#5 0x000000000040d285 in Rules::load (this=0x1191aa0, filename=...) at ruleparser.cpp:317
#6 0x000000000040ff8b in load (this=<optimized out>) at ruleparser.cpp:124
#7 RulesList::load (this=this@entry=0x7fff88321b60) at ruleparser.cpp:39
#8 0x0000000000407209 in main (argc=6, argv=<optimized out>) at main.cpp:195
(gdb)
I don't have a clue why but for some SVN tags (aka branches), if you specify annotated true
within a matching rule, the tool die with : 'fatal: Can't tag an empty branch.'
It doesn't happened without annotated
.
I can't find anything with the failing tag that is different with those who works : could you lead to the path to understand what's happening ?
I'm using svn2git 1.0.11, svn 1.9.3 and git 2.15.1
Hi, I'm trying to convert an old svn repo and I'm getting this problem.
The crash report:
fast-import crash report:
fast-import process: 15204
parent process : 1
at 2019-08-20 09:28:46 +0000
fatal: Empty path component found in input
Most Recent Commands Before Crash
---------------------------------
data 2224
blob
mark :18446744073709551171
data 5330
blob
mark :18446744073709551170
data 11902
blob
mark :18446744073709551169
data 2693
blob
mark :18446744073709551168
data 1629
blob
mark :18446744073709551167
data 1733
blob
mark :18446744073709551166
data 2569
blob
mark :18446744073709551165
data 8825
blob
mark :18446744073709551164
data 1803
blob
mark :18446744073709551163
data 3467
blob
mark :18446744073709551162
data 3363
blob
mark :18446744073709551161
data 7116
blob
mark :18446744073709551160
data 4363
blob
mark :18446744073709551159
data 3269
blob
mark :18446744073709551158
data 3212
blob
mark :18446744073709551157
data 16104
blob
mark :18446744073709551156
data 5243
blob
mark :18446744073709551155
data 6632
blob
mark :18446744073709551154
data 23172
blob
mark :18446744073709551153
data 10553
blob
mark :18446744073709551152
data 7209
blob
mark :18446744073709551151
data 1092
blob
mark :18446744073709551150
data 4020
blob
mark :18446744073709551149
data 3713
blob
mark :18446744073709551148
data 4967
blob
mark :18446744073709551147
data 3157
blob
mark :18446744073709551146
data 3856
blob
mark :18446744073709551145
data 5849
blob
mark :18446744073709551144
data 4291
blob
mark :18446744073709551143
data 3751
blob
mark :18446744073709551142
data 7776
blob
mark :18446744073709551141
data 25184
commit refs/heads/master
mark :19
committer user <user@localhost> 1190996168 +0000
data 39
deleteall
* M 100644 :18446744073709551614 /.gitignore
Active Branch LRU
-----------------
active_branches = 1 cur, 5 max
pos clock name
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1) 0 refs/heads/master
Inactive Branches
-----------------
refs/heads/master:
status : active loaded dirty
tip commit : 9e09d0745d5af0c3978f948e10fd135c07783625
old tree : 0000000000000000000000000000000000000000
cur tree : 0000000000000000000000000000000000000000
commit clock: 0
last pack :
Marks
-----
exported to marks-REPO
-------------------
END OF CRASH REPORT
I am trying to export a Subversion repository that contains multiple releases of a product into a new git repository. Along the way I am also export code from a different system (Serena PVCS) by hand into this same repository. My goal has been to do the following:
I do the manual commits by cloning the bare repository svn2git created into a new folder. I check out master, add all my code from PVCS, commit, and push back into the bare repository.
However, when I run svn2git the second time to load the next batch of revisions, it actually completely wipes out my manual commits in the bare repository, as if they had never existed. They don't show up in the git log within the bare repository. When I "git pull" in my local copy I created for adding the code from PVCS, it reports divergence of the remote branch and does a commit for a merge.
Each time I run svn2git it seems to wipe out my commits that didn't come from subversion.
Am I doing the process incorrectly, or is this just not possible, to run svn2git into what is essentially an existing repository that has code from other commits?
I get this warning:
Exporting revision 1 .WARN: Branch "master" in repository "test1" doesn't exist at revision 1 -- did you resume from the wrong revision?
It seems this warning only appears when a new branch is created (for rev 1 there was no git created branch before). So I'm not sure if should care about this warning.
If this happens normally for every new branch I would feature request to not show this warning.
Git does not allow adding empty directories (see https://git.wiki.kernel.org/index.php/GitFaq#Can_I_add_empty_directories.3F). As this is possible in SVN it could break some repositories as it is in our case.
Therefore svn2git should add an empty .gitignore-file if it recognizes an empty directory, before converting it to git.
First of all, thank you for sharing this very flexible and highly configurable conversion tool.
I had however some nasty problems getting incremental conversions working, which I'd like to share because they were partially caused by a lack of documentation and partially by a design flaw:
The --resume-from
flag is poorly documented. The svn2git --help
output simply states
--resume-from start importing at svn revision number
--max-rev stop importing at svn revision number
which misguided me to think that the flags were meant to be used for incremental conversions. That is, if the previous svn2git run converted all revisions up to revision <n>
, I thought that I needed to call svn2git --resume-from <n+1> ...
for incrementally converting the newer revisions.
In the end, it turned out that the --resume-from
isn't needed at all, but the necessary information is collected automatically from the log files (see below). The --resume-from
flag seems to be meant to be used for discarding and rewriting some commits which were already committed. (I did not really check that in the source thoroughly.)
The log files (log-<repo>.git
) play a crucial role for the incremental conversions. Unfortunately, this important detail is completely undocumented. What makes things worse is the fact that the log-<repo>.git
is stored outside the Git repository, contrary to other important auxiliary files (branchNotes-<repo>.git
, marks-<repo>.git
).
.
├── <repo>.git
│ ├── HEAD
│ ├── branchNotes-<repo>.git
│ ├── branches
│ ├── config
│ ├── description
│ ├── hooks
│ ├── info
│ ├── marks-<repo>.git
│ ├── objects
│ └── refs
└── log-<repo>.git
Since the term "log file" suggests that the file just contains a transcript from a previous run, I failed to include them when I moved the converted Git repositories from my own machine to the production server (to run incremental conversions triggered by an SVN post-commit hook). The first run of the incremental update produced tons of warnings
Exporting revision <nnnn> ..WARN: Branch "master" in repository "<repo>.git" doesn't exist at revision <nnnn> -- did you resume from the wrong revision?
and left the Git repositories in a broken state. (A lot of files were missing from HEAD^{tree}
).
In view of my experiences, I would like to suggest the following enhancements:
--resume-from
and --max-rev
options, in particular how they relate to incremental conversions.log-<repo>.git
, branchNotes-.git,
marks-.git`) and how they relate to incremental conversions.log-<repo>.git
into the Git repository (in a backward-compatible way).Currently i don't see the ability to use the git subtrees (better alternative to the git submodules) to represent a piece of directory in the GIT repo. Currently i can create multiple GIT repos from a single or several SVN repos, but can not make a reference repository in the GIT.
At least it was not mentioned here: https://techbase.kde.org/Projects/MoveToGit/UsingSvn2Git
Do you have plans to make it?
Hello. I'm trying to use svn-all-fast-export to migrate our SVN project.
Out team used SVN Repository very not useful
they made a main repository, and put sub repositories to that main repository
so main repository is over 30000 revisions, and have lots of file size ...
Main Repository
|- Sub Repository 1
| |-branches
| |- tags
| |- trunk
|- Sub Repository 2
| |-branches
| |-tags
| |-trunk
....
By the way, I need to separate these sub repository to each git repositories.
(Sub)Repository 1
|-branches
|-tags
|-trunk
(Sub)Repository 2
|-branches
|-tags
|-trunk
...
I typed these things to Rule file, and svn2git program worked.
but, when passing 3000 revisions, Out of memory error happened.
I thought it using too much git repositories, I tried to convert just one repository to rule files,
but it rejected because of rule is not correct.
because, program may read other sub repository, because i didn't typed to rule file about it.
Error message is
fatal : Out of memory, malloc failed (tried to allocate 1468122 bytes)
warning: die() called many times. Recursion error or racy threaded death!
fatal: internal consistency error creating the index
is there any good way to solve this problem..? T_T
Title says it all... Easy for package distributors to provide a new version, as the current one is from 2013.
Hi! I'm getting compile error with 1.0.14 that I'm not getting with 1.0.13. Any ideas what's wrong?
x86_64-pc-linux-gnu-g++ -c -march=core2 -msse4.1 --param l1-cache-line-size=64 --param l1-cache-size=32 --param l2-cache-size=6144 -O2 -pipe -Wall -W -D_REENTRANT -fPIC -DVER="\"1.0.14\"" -DQT_NO_DEBUG -DQT_CORE_LIB -I. -I. -isystem /usr/include/subversion-1 -isystem /usr/include/apr-1 -isystem /usr/include/qt5 -isystem /usr/include/qt5/QtCore -I. -I/usr/lib64/qt5/mkspecs/linux-g++ -o repository.o repository.cpp
repository.cpp: In function ‘QDataStream& operator<<(QDataStream&, const FastImportRepository::AnnotatedTag&)’:
repository.cpp:263:9: error: ambiguous overload for ‘operator<<’ (operand types are ‘QDataStream’ and ‘quint64 {aka long long unsigned int}’)
out << annotatedTag.supportingRef
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<< annotatedTag.svnprefix
~~~~~~~~~~~~~~~~~~~~~~~~~
<< annotatedTag.author
~~~~~~~~~~~~~~~~~~~~~~
<< annotatedTag.log
~~~~~~~~~~~~~~~~~~~
<< (quint64) annotatedTag.dt
^~~~~~~~~~~~~~~~~~~~~~~~~~~~
(From trying to update the Gentoo package of it, hence the color and Gentoo bits)
svn2git-1.0.14-build-log.txt
Svn2git knows the "--empty-dirs" option to handle empty SVN directories. Unfortunately the option is only effective if the directory is created empty. If a directory is emptied while deleting all files and folders in it, the .gitignore file is not created.
It means that currently - if you have such cases - some kind of 'post-processing' after the svn2git-based migration is needed.
A lot of people are leaving github expecting that the microsoft take over will cause the site to become a mess in not too much time. As has happened with other such examples (like Skype).
I reserved the groups 'svn2git' and 'svn-all-fast-export' on gitlab, so hit me up if the current maintainers are interested in migrating and using one (or both) of those groups.
@alexgit2k implemented the ability to generate .gitignore
entries from svn:ignore
properties.
"Unfortunately" since SVN 1.8 there is also svn:global-ignores
which basically is the same, but works on all subdirectories, not only on the directory where it is specified, like /*.java
vs. *.java
in .gitignore
.
It would be great if svn2git would also support the svn:global-ignores
which it currently only warns about if you ask it to. @alexgit2k could you maybe give it a shot? :-)
@tnyblom it looks like something went wrong during release tagging today. The previous format was 1.0.xy and now we have 1.0xy, missing a dot. Any objections to re-tagging as 1.0.18?
Thanks and best, Sebastian
Entries to the log generated by svn2git should be time stamped. It is hard to know how long it has spent processing a particular commit.
Hello,
I want to migrate old SVN to git
the SVN repository is like:
http://svn/Sky/trunk/folder1/
http://svn/Sky/trunk/folder2/
http://svn/Sky/branches/0.1/folder1/
http://svn/Sky/branches/0.1/folder2/
...
http://svn/Sky/branches/X.Y.Z/folder1/
http://svn/Sky/branches/X.Y.Z/folder2/
but folder2 is an svn:external from http://svn/Moon/tags/18_10_2013/sources/folder2
project at a specific tag on trunk.
and
folder2 is an svn:external from http://svn/Moon/tags/01_10_2013/sources/folder2
project at a specific tag on /branches/0.1.
and so... for /branches/X.Y.Z/
How can i write the match rules in those cases?
BR
svn-all-fast-export (the latest version of the code in this repository as of 2015-12-01), installed using the svn2git port I recently added to MacPorts, crashes after doing a conversion, on OS X 10.11 El Capitan, built using Qt 5.5.1, Subversion 1.9.3 and APR 1.5.2 also installed by MacPorts.
The terminal output ends with:
svn-all-fast-export(14669,0x7fff71d09000) malloc: *** error for object 0x7fc2ca80d400: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
The crash log OS X writes says:
Crashed Thread: 0 Dispatch queue: com.apple.main-thread
Exception Type: EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY
Application Specific Information:
abort() called
*** error for object 0x7fc2ca80d400: pointer being freed was not allocated
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libsystem_kernel.dylib 0x00007fff8a619002 __pthread_kill + 10
1 libsystem_pthread.dylib 0x00007fff856ff5c5 pthread_kill + 90
2 libsystem_c.dylib 0x00007fff8efa46e7 abort + 129
3 libsystem_malloc.dylib 0x00007fff93ab0070 free + 425
4 libapr-1.0.dylib 0x0000000109b071b3 apr_allocator_destroy + 27
5 libapr-1.0.dylib 0x0000000109b07761 apr_pool_terminate + 36
6 libsystem_c.dylib 0x00007fff8efa546b __cxa_finalize_ranges + 345
7 libsystem_c.dylib 0x00007fff8efa576f exit + 55
8 svn-all-fast-export 0x0000000109a4b89b start + 59
I don't know if this is relevant, but this Subversion mailing list post says crashes in apr_pool_terminate
could be due to failing to clear an error with svn_error_clear
.
I receive the following error while attempting to run a migration - "fatal: Branch name doesn't conform to GIT standards: refs/tags/AbsoluteSafe 2.0 (v301)"
Any suggestions on what can be done about this - I'm not sure if this is a tool bug, or a limitation of the characters used in the name of the tag/branch.
I'm trying to perform what is admittedly a rather complex svn2git conversion, as documented here:
https://sourceforge.net/p/brlcad/code/HEAD/tree/brlcad/trunk/misc/svn2git/
When I try to do a partial conversion and resume it with the --resume-from option, the process gets a few commits in and then crashes. It generates a log, and this seems to be the key issue:
fatal: mark :0 not declared
looking at the mark file in the repo, it does indeed start with :1 with no :0 entry present. Is this a known issue?
I created a script to do a simple reproduction: https://github.com/kfsone/examples/tree/master/svn-all-fast-import
But it actually proves to be pretty simple. Command line arguments:
--identity-map --identity-domain unknown.unk
--rules
--add-metadata
--stats
--empty-dirs
--propcheck
--svn-ignore
--svn-branches
--max-rev
file://${REPOS_PATH}
. Create svn repos,
. Create a commit with the svn path of: /repos/Trunk/test1.txt
. Create a rule to map /repos/Trunk to master
. Attempt to convert
-> warning about "/repos/" not matching
. Add rule to ignore /repos/ after /repos/Trunk rule
. Attempt to convert
-> conversion appears to complete
*> fast_import_crash_NNN created, git repository is empty
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.