Sunday, February 15, 2009

Synchronizing Git repositories without a server

This post describes a method for pushing changes between two repositories without using a server with network connections to both hosts having repositories. The solution is a reasonably straightforward application of local repositories, and I'm providing it to save the time of fellow developers with similar needs.

I first describe a solution using a USB stick to move data, which can be generalized to any other method for transferring files offline. Then I explain how to use this solution to deploy Rails applications using my tool, without a version control server. In case you're curious, I end the post explaining what prompted me to do this. Obviously, you can stop reading after you learned all you needed to know.

Pushing Changes with a USB Stick

I'm assuming your working directory is the place where you have your source repository. If not
cd /path/to/your/repository
Start up by creating a repository on the USB stick.
mkdir /path/to/usb/stick/repository.git
git clone --local --bare . /path/to/usb/stick/repository.git
Then register the repository on the USB stick as a remote repository, and push the desired branch to it (if you don't want to push master, substitute your desired branch).
git remote add usb file:///path/to/usb/stick/repository.git
git push usb master
In the future, you can treat the USB repository as any other remote repository. Just make sure it's mounted :) For instance, the following pushes new changes to the USB repository.
git push usb

On the receiving end, mount the USB stick, and use a file URL for the repository
file:///path/to/usb/stick/repository.git
A few handy commands:
  • cloning the repository on the USB stick



    git clone file:///path/to/usb/stick/repository.git
    
  • updating a repository cloned from the USB stick using the above command



    git pull origin
  • adding the USB stick repository as a remote for an existing repository


    git remote add usb file:///path/to/usb/stick/repository.git
  • updating from a remote repository configured using the above command



    git pull usb master
The technique above works for any offline data transfer method, as long as you can copy the git repository from one file-system to the other.

Deploy Rails Applications without a Version Control Server


rpwn assumes that the source code for Rails applications is under version control on a server. But, at least when using git, applications can be deployed from repositories on USB sticks, created as explained in the previous section. Install an application as follows.
sudo rpwn install file:///path/to/usb/stick/repository.git
Keep in mind that rpwn will use the same repository URL to update the application, so you will always have to use a USB stick that mounts to the same location. This limitation can be overcome by adding another indirection layer -- create a symbolic link to the remote repository, and use that symlink when installing.
ln -s /path/to/usb/stick/repository.git ~/victor/repository.git
sudo rpwn install file:///home/victor/repository.git
Note that the symlink preserves the repository name, so rpwn will still use the right name for the application.

Motivation

I wanted to help someone deploy a Rails application in a government setting, where the application server is not connected to the Internet (stupid security rules). Worker's computers are still connected to the Internet, so we wanted to host the Git repository on Github. This way, the application can be developed and tested anywhere.

So, the development machine is connected to the Internet, but the application server is not. I didn't want to push unversioned source code to the application server, and I was hoping I don't have to add extra logic to rpwn to handle this case. Git local repositories turned out to be an awesome, if not well-documented, solution.


I hope you found this post useful, and kindly ask that you share any better methods you may come up with to solve your own problem.

6 comments:

  1. Thanks very much for the clear instructions and explanation. I was able to make this work with an external volume served over afp for a local network.

    Harry

    ReplyDelete
  2. On Windows (using git inside of cygwin), the "--local" option unfortunately gives me this error:
    ----------------------------------------
    $ git clone --local --bare . /cygdrive/f/foobar.git
    Initialized empty Git repository in /cygdrive/f/foobar.git/
    fatal: failed to create link '/cygdrive/f/foobar.git/objects/00/0d9dbbb7293dd5f383e806b4919f8ebd9ac': Invalid cross-device link
    ----------------------------------------

    Without the "--local" option, I am able to successfully clone the repository to the USB stick.

    ReplyDelete
  3. Thankyou, nice post.

    @Jean Pierre

    You can use the git clone --no-hardlinks option to tell git to take a copy of the files rather than attempt to symlink over to the mounted USB drive.

    ReplyDelete
  4. Do you know about a way to maintain in the usb drive only the deltas rather the entire repository ? (our rep is 500 MB ) and we want to use very small USB drive .

    ReplyDelete
    Replies
    1. check git-annex http://git-annex.branchable.com/ watch the short video on bottom of page

      Delete