Git vs Subversion Speed

Compiling, libraries, modules, coding guidelines and porting

Moderators: jesterKing, stiv

ldo
Posts: 544
Joined: Sun Nov 07, 2010 11:30 pm
Location: Hamilton, New Zealand

Git vs Subversion Speed

Postby ldo » Wed Feb 22, 2012 5:33 am

Interesting to compare relative speeds:

* Pulling down the entire history of trunk/blender from svn.blender.org (a total of about 24000 commits when I first did it) took about 5 hours using git-svn, as I recall. Pushing the resulting Git repo to my GitHub area took another 3 hours. Keeping that up-to-date only takes a few seconds each day.
* I am currently pulling down all the entries in tags, using the standard svn client. That’s a total of about 50 copies of the source tree. It’s been running for 22 hours, and done about half of them so far.

As another comparison, the entire size of my Git repo is currently 250MB. The checkout of the tags has getting close to 2GB, so it’ll probably be about 4GB when it’s finished, presumably sometime tomorrow.

To reinforce the point: a single checkout, equivalent to 50 commits, done with svn, is taking about an order of magnitude longer than a complete history dump of 24000 commits done via git-svn. Even if you try to argue that most of those commits only affected a small number of source lines, to the point where, say, those 50 code snapshots are really equivalent to 50000 commits, there still seems to be something of a disparity in speed of data transfer, don’t you think?

stiv
Posts: 3646
Joined: Tue Aug 05, 2003 7:58 am
Location: 45N 86W

Postby stiv » Wed Feb 22, 2012 5:38 am

took about 5 hours using git-svn


Show us the numbers using a plan svn client.

ldo
Posts: 544
Joined: Sun Nov 07, 2010 11:30 pm
Location: Hamilton, New Zealand

Postby ldo » Wed Feb 22, 2012 8:39 am

The complete uncompressed Blender trunk source tree is currently about 90MB. So the tags checkout is trying to download about 50 separate copies of this. Which just points out how inefficient the implementation of tags is in Subversion.

Subversionistas like to brag about “cheap copies”, where it’s very quick and easy to create a new branch with minimal overhead on the server. But that’s not much help when you try to copy that entire new branch onto the client. And each tag in Subversion involves just such a copy.

By contrast, Git tags are nothing but pointers to commits, not copies of the commits. So once you already have a copy of the commit, copying the tag across the network is very quick and cheap.

Git tags can also be signed. This is handy for marking “official” releases. And it doesn’t matter where you got the objects in your repo from, you know they’re genuine if the signature verifies.

stiv
Posts: 3646
Joined: Tue Aug 05, 2003 7:58 am
Location: 45N 86W

Postby stiv » Wed Feb 22, 2012 7:17 pm

So the tags checkout is trying to download about 50 separate copies of this.


Normally, an SVN user will simply check out trunk. I suspect what you are doing is downloading all the branches and tags. The branches are the expensive part.

Worse case, you are also getting the pre-built libraries for the various platforms.

ldo
Posts: 544
Joined: Sun Nov 07, 2010 11:30 pm
Location: Hamilton, New Zealand

Postby ldo » Sat Feb 25, 2012 3:17 am

It finally finished. There were in fact 60 different subdirectories in tags, totalling nearly 84GB, and they took about 87½ hours to download (just shy of 4 days).


Return to “Coding Blender”

Who is online

Users browsing this forum: No registered users and 0 guests