GitSuperRepoAdminGuide

Local mirrors

The modules in a Cactus tree typically constitute a large number of repositories in various version control systems from wide number of sources. While GetComponents makes it get everything from their original locations, it may be still desirable to create local mirrors of these modules for several reasons:

• Speed - Some sources may be on the other side of the world so having a nearby local mirror can dramatically reduce the time to access them.
• Privacy - You may want to develop a new feature privately within your group before publicly releasing it. If this feature is based on an existing module then having a local mirror means you can create local private development branch which can later be easily merged and pushed upstream.
• Convenience - With remote repositories in a variety of version control systems, it is easy to get confused. Having local mirrors of these in a single version control system (i.e. git) simplifies things so that only a single tool is needed. In the case where the remote repository is in SVN local git mirrors also provides the advantage of offering the additional features git provides over svn.

This section describes how to setup, host and administer local mirrors of remote repositories.

Hosting mirrors on a server

Rather than everyone creating their own local mirrors which they have to worry about keeping up-to-date, it makes sense to have a central server do so. In this section, we describe how to setup a server using Gitolite.

Optional Extras

Blocking pushes of unnecessary merge commits

While working on a change in a git repository, it often happens that while you make local commits someone else pushes their own changes to the central repository. If you then try to push your changes, your push will be rejected as a non-fast-forward commit. The easiest solution is to pull the new changes from the server before pushing. However, this will introduce a 'merge commit' in the process. These merge commits rapidly build up and clutter up the history of the repository. For this reason, instead of pulling and introducing a merge commit you should do one of two things:

• Rebase your changes on top of the central repository. This can be achieved using 'git pull --rebase'.
• If your changes are significant, you can create a new named branch for them and merge that branch into the branch from the central repository.

In an ideal world, everyone would always follow these rules. However, in practice it is easy to forget to do so.

A solution is to block these merge commits from ever being pushed to the server. This is easily achieved using a git hook on the server side. The file 'update.merge' in the git-module repository gives an example of how to so. Place this file in the hooks directory of the repository on your server, rename it to 'update' and make sure it is executable. It will then block these annoying merge commits while allowing through genuine merge commits from named branches.

Super repository

With repository mirrors created and all modules now available as git repositories, it is quite convenient to have a container repository to manage everything. This container can be implemented as a git super-repository with git submodules for each component. In this section, we describe how to setup and administer such a repository, both locally and using a central server.

Common tasks

Adding a submodule

Adding a new submodule is as simple as running:

 git module add <url-to-repository> <path-to-submodule> [<upstream-url>] [<upstream-type>] [<track-branch>]

checking the changes look OK, committing and then pushing. The optional arguments <upstream-url>, <upstream-type> and <upstream-branch> specify the URL and type of the upstream version of this repository and the branch which the super-repository should track. If not given, <upstream-url> is assumed to be the same as <url-to-repository> and <upstream-type> is assumed to be 'git'. Other possible options for <upstream-type> are 'hg' and 'svn'. If <track-branch> is not given it defaults to 'trunk' for git-svn repositories and 'master' for git and hg-git repositories.

Removing a submodule

To remove a submodule from the super repository, use:

 git module rm <submodule-path>

check the changes look OK, then commit and push.

Include here a guide to setting up a Git server with a Cactus super-repository

Have temporarily copy+pasted the README which is in note form - need to write a proper up-to-date guide.

###############################################################################
# CactusGit
###############################################################################

# Introduction

# This package allows you to check out Cactus using GetComponents from
# a CRL thornlist using git-svn instead of svn for the svn
# repositories.  This means that your local Cactus tree contains
# complete version information and history and can be managed using
# git rather than svn.

# Further, the package provides tools for creating a
# "super-repository" for the tree with pointers to the downloaded
# repositories as git submodules.  This means that the state of the
# tree can be identified by a single commit ID and can be cloned
# directly without having to use GetComponents or git-svn.

# Notes
# * To make this more convenient, small wrapper scripts can be written
#   to replace the recipes given below
#
# * GetComponents could be extended to support git-svn directly,
#   rather than using the wrapper-script approach that is currently
#   used

# Tutorial

# Add our tools to the path, including the svn wrapper which tricks
# GetComponents into making a git-svn clone of each svn repository.
# We also include a copy of GetComponents here for convenience.
export OLDPATH=$PATH PATH=$PWD/bin:$PATH

# Check out the tree into the Cactus directory (can take a very long
# time, ~60 minutes, and does not show progress for each repository
# until it has checked it out completely)
GetComponents -v -a WaveToy.th

# Now manually checkout any thorns which require authentication
git svn clone ...

# Add the git-svn repos to your server (also add the Cactus super-repo).
crlrepos WaveToy.th > gitsvn.list
cd <gitolite_dir>
vi conf/gitolite.conf # Add the repositories in gitsvn.list
git commit conf/gitolite.conf
git push

# Add a 'fetching and pushing' (fp) user
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

# Push as the fp user
scp Cactus fp@server:fetch/
scp gitsvn.list fp@server:fetch/
ssh fp@server
cd fetch
for i in `cat gitsvn.list`; do cd $i && git remote add bare cactus@localhost:$i && git config --unset remote.bare.fetch && git config remote.bare.push 'refs/remotes/*:refs/heads/*' && git push bare && cd -; done

# On the server, update the HEAD of the git-svn repos
ssh cactus@server
cd ~/repositories
for i in `cat ../fetch/gitsvn`; do cd $i.git && git symbolic-ref HEAD refs/heads/git-svn && cd -; done

# Generate a super-repository called CactusGit using all the
# repositories from Cactus listed in WaveToy.th
crltogit CactusGit cactus@server: Llama.CTGamma.ET.th

# Add non-svn repos
GetComponents Llama.CTGamma.ET.th
rm -rf repos
git submodule add ... repos/...

# Symlink flesh
ln -s flesh/CONTRIBUTORS
ln -s flesh/COPYRIGHT
ln -s flesh/Makefile
ln -s flesh/doc
ln -s flesh/lib

# Commit symlinks (as fp)
git add *
git commit ...
git push cactus@localhost:Cactus master:master

######### Administrators ########
# Updating the git-svn mirrors
for i in `cat gitsvn.list`; do cd $i; git svn fetch && git push bare; cd -; done

# Updating the hg-git mirrors
for i in `cat githg.list`; do cd $i; hg pull && hg push git; cd -; done

# Updating the git mirrors
for i in `cat git.list`; do cd $i; git fetch -a -u && git push bare --all; cd -; done

# Updating all submodules in the super-repo
cd Cactus
git pull
git submodule foreach git pull
git commit -m "Update submodules." -a

# Updating a single submodule in the super-repo
cd Cactus
cd <repo-path>
git pull
cd -
git commit -m "Update submodule <repo-path>." <repo-path>

# Automatic updates of super-repo
git clone --recursive cactus@localhost:Cactus
cd Cactus
git submodule foreach 'git remote set-url origin /var/cactus/repositories/$path cactus@git.barrywardell.net:/$path; echo OK'
git remote set-url origin /var/cactus/repositories/Cactus
git remote set-url --push origin cactus@git.barrywardell.net:Cactus
git checkout submodules
git pull origin submodules
git submodule foreach 'git fetch origin && git checkout origin'
git commit -m "Update submodules." -a
git push origin submodules

# Adding a git-svn repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi gitsvn.list # add <newrepo>
git svn clone -s <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare
cd ~/repositories/<newrepo>
git symbolic-ref HEAD refs/heads/trunk
cd ~/fetch/<newrepo>
git pull bare

# Adding a hg-git repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi githg.list # add <newrepo>
hg clone <url> <newrepo>
cd <newrepo>
vi .hg/hgrc # Add: 
            # [path]
            # git = git+ssh://localhost/<newrepo>
            # [git]
            # intree = 1
hg gexport # Can take quite a long time
hg push git

# Adding a git mirror repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi git.list # add <newrepo>
git clone --mirror <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git push bare --all

# Adding a submodule to the super-repo
git submodule add <url> <path>
git commit
git push

# Removing a submodule from the super-repo
vi .gitmodules
vi .git/config
git rm --cached <path>
git commit ...
git push

# Changing svn URL for a git-svn clone. WARNING: this is rewriting the history of 
# the git-svn branch of your mirror and will mess peoples checkout up.
cd fetch/<repo>
git status
git gc
git filter-branch --msg-filter 'sed "s/git-svn-id: http/git-svn-id: https/g"' $(cat .git/packed-refs | awk '// {print $2}' | grep -v 'pack-refs')
sed -i -e "s/http/https/g" .git/config
rm -rf .git/svn
git svn rebase
git push bare --force

# Changing svn URL without rewriting history (https://git.wiki.kernel.org/index.php/GitSvnSwitch):
Edit the svn-remote url URL in .git/config to point to the new domain name
Run git svn fetch - This needs to fetch at least one new revision from svn!
Change svn-remote url back to the original url
Run git svn rebase -l to do a local rebase (with the changes that came in with the last fetch operation)
Change svn-remote url back to the new url
Run git svn rebase should now work again!