How to Clone Large Git Repository

Often when you try to clone large git repository, it may result in an error message. For some reason, git does not work properly, when you try to clone a large repository at one go. Apparently, there is a different way to clone big repositories. In this article, we will learn how to clone large repository to your local system. You can follow these steps if you are unable to clone large git repository.

How to Clone Large Git Repository

When you try to directly clone large repositories, you may see the following error message.

Cloning into ...
...
error: RPC failed; curl 18 transfer closed with outstanding read data remaining 
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed

The solution is to first raise the http.postBuffer variable as shown below.

$ git config --global http.postBuffer 524288000

Next, do a shallow clone, meaning clone only the last commit of your repository and not the ones before. Replace <url> with repository URL. This technique is known as shallow clone or partial clone. If you want to clone more commits, replace 1 with the number of commits you want to clone.

$ git clone <url> --depth 1

Shallow clone has certain limitations such as you cannot clone or fetch from it, nor push from or into it. But once you have the first clone, you can expand it to fetch previous commits from remote repository.

Here is an example, to fetch 100 previous commits.

$ git fetch --depth=100

In fact, you can run the above command multiple times to keep fetching 100 commits at a time, without error messages.

Once you have fetched sufficient number of commits, you can use the following command to fetch all the remaining commits.

$ git fetch --unshallow

This way you can clone a large repository. First you need to get a shallow clone of depth 1, then fetch a few more commits using ‘git fetch’ command. Finally, use –unshallow option with git fetch command to get all the remaining commits from repo.

Also read:

How to Clone Single Branch in Git
How to Ignore Git File Permission Changes
How to Configure Line Endings in Git Commit
Git Remove File From History
Bash Loop Through Files in Directory