quick-clone on very large repository unacceptably slow (at least 15 hours to complete) about git-tfs

[spraints]

Out of curiosity, how long does it take to pull down a fresh copy of the repository to a TFS workspace?

Also, what version of git-tfs did you download or build?

IO (local file and TFS DB) is typically the bottleneck on clone or quick-clone. Depending on the version of git-tfs you're using, it could also be spinning up lots of subprocesses, which isn't fast. I may someday rewrite clone and/or quick-clone to address speed issues.

You might take a look at tfs2git to see if you can speed up your clone time. Once the repository is mirrored into git, git tfs fetch will be much more performant.

[JamesDunne]

Straight from TFS the whole repository shouldn't take more than 20 minutes I think. I haven't pulled down the entire project's repository in a long time; I usually just work within a specific branch and subproject which is about 10% the size of the entire repository, clocking in around 11,000 files I think.
git-tfs definitely needs a way to clone partial sections of a repository; we have lots of branches of code in our TFS project. I don't think I could do tfs2git since the repository has upwards of 123,000 changesets now, which is why quick-clone sounded attractive to me.
I built the git-tfs from your git repo just yesterday, commit id 44d6f40.

[JamesDunne]

I'm doing a git tfs fetch now and it takes about 12 seconds per changeset, consistently.

[spraints]

On larger projects, I generally don't clone the whole thing. I try to find the narrowest directory that works. e.g. if there is $/Project with "Docs", "Third Party", and "Source" in it, I just git tfs clone http://blah $/Project/Source. If you wanted Third Party and Source but not docs, you should be able to set the config value tfs-remotes.default.ignore-paths to "^Docs". You can also give this to git-tfs when you clone: git tfs clone --ignore-regex=^Docs http://blah $/Project/Source. I often use this flag for large files (like the MSDE installer) that are sometimes unnecessarily included in the source tree somewhere.

[JamesDunne]

Aha! Thanks for that. I'll definitely try that the next time I can. Thanks for your help and your quick responses!

[JamesDunne]

Hi Matt,
I was able to put together an effective ignore-regex and also checked out only the specific branch of code I needed. I ended up with 14,600 objects checked out at an average rate of 0.726 objects/sec download rate. This took 5 hours and 19 minutes to complete, according to your -d(ebug) mode. All of the time is being spent in the DownloadFile method, which is to be expected, but I'm not sure why it is so slow. I'm downloading over a VPN connection this time, whereas my last report was over wi-fi at the office. I'll do a checkout with vanilla TFS next and see if there's any significant difference.

[JamesDunne]

Okay, TFS is just slow. :P. I'm sure it's our completely-competent-in-every-way networking staff's fault.

[JamesDunne]

Closing this issue since the problem is solely contained within the DownloadFile API call to TFS.