GitSuperRepoAdminGuide
Local mirrors
The modules in a Cactus tree typically constitute a large number of repositories in various version control systems from wide number of sources. While GetComponents makes it get everything from their original locations, it may be still desirable to create local mirrors of these modules for several reasons:
- Speed - Some sources may be on the other side of the world so having a nearby local mirror can dramatically reduce the time to access them.
- Privacy - You may want to develop a new feature privately within your group before publicly releasing it. If this feature is based on an existing module then having a local mirror means you can create local private development branch which can later be easily merged and pushed upstream.
- Convenience - With remote repositories in a variety of version control systems, it is easy to get confused. Having local mirrors of these in a single version control system (i.e. git) simplifies things so that only a single tool is needed. In the case where the remote repository is in SVN local git mirrors also provides the advantage of offering the additional features git provides over svn.
This section describes how to setup, host and administer local mirrors of remote repositories.
Adding git-svn repository mirrors
git svn clone -s <url> <newrepo>
Hosting mirrors on a server
Rather than everyone creating their own local mirrors which they have to worry about keeping up-to-date, it makes sense to have a central server do so. In this section, we describe how to setup a server using Gitolite.
Setting up Gitolite
Follow the Gitolite installation instructions:
On the server (as root):
useradd -d /var/git -s /bin/bash -m git git clone git://github.com/sitaramc/gitolite cd gitolite src/gl-system-install su - git gl-setup /tmp/admin.pub
On your workstation:
git clone git@server:gitolite-admin
Adding git-svn repository mirrors
Edit conf/gitolite.conf and add repository.
git commit conf/gitolite.conf git push
Add a fetching repository on the server:
ssh server
add-git-svn-mirror arrangements/${arr}/${thorn} http://svn.cactuscode.org/arrangements/${arr}/${thorn}
Adding git mirrors
vi conf/gitolite.conf git commit conf/gitolite.conf git push
ssh fp@server cd fetch vi git.list # add <newrepo> git clone --mirror <url> <newrepo> cd <newrepo> git remote add bare cactus@localhost:<newrepo> git config --unset remote.bare.fetch git push bare --all
Removing repository mirrors
To remov a repository mirror:
rm -rf ~/repositories/<repo> rm -rf ~/fetch/<repo> vi ~/fetch/gitsvn.lisst # remove <repo>
Edit conf/gitolite.conf and remove the entry for the repository.
git commit conf/gitolite.conf git push
Permissions
Einstein Toolkit mirrors
The Einstein Toolkit mirrors are currently hosted on git.barrywardell.net. When a thorn gets added to the Einstein Toolkit, we have to do the following steps to handle it:
1. Add the thorn to the conf/gitolite.conf file in the gitolite-admin directory. Commit and push.
2. Run add-git-svn-mirror to add a mirror of the svn repository, e.g.
$ add-git-svn-mirror arrangements/arr/thorn https://svn.cactuscode.org/arrangements/arr/thorn
3. Add the new thorn to the Einstein Toolkit super-repository: Edit .gitmodules, run 'git module sync', commit and push.
4. Initialise the new submodule in the fetching repository on the server:
$ ssh et@barrywardell.net
$ cd fetch/EinsteinToolkit
$ git pull
$ git submodule update --init
Common Tasks
Moving to a new upstream url
Update the location on the gitolite server:
mv <old> <new> Edit gitolite permissions, commit and push
Update the fetching repository:
Edit gitsvn.list - change path and URL mv <old> <new> Edit .git/config - change svn-remote url and bare url Run git svn fetch - This needs to fetch at least one new revision from svn! Change svn-remote url back to the original url Run git svn rebase -l to do a local rebase (with the changes that came in with the last fetch operation) Change svn-remote url back to the new url Run git svn rebase should now work again!
Optional Extras
Blocking pushes of unnecessary merge commits
While working on a change in a git repository, it often happens that while you make local commits someone else pushes their own changes to the central repository. If you then try to push your changes, your push will be rejected as a non-fast-forward commit. The easiest solution is to pull the new changes from the server before pushing. However, this will introduce a 'merge commit' in the process. These merge commits rapidly build up and clutter up the history of the repository. For this reason, instead of pulling and introducing a merge commit you should do one of two things:
- Rebase your changes on top of the central repository. This can be achieved using 'git pull --rebase'.
- If your changes are significant, you can create a new named branch for them and merge that branch into the branch from the central repository.
In an ideal world, everyone would always follow these rules. However, in practice it is easy to forget to do so.
A solution is to block these merge commits from ever being pushed to the server. This is easily achieved using a git hook on the server side. The file 'update.merge' in the git-module repository gives an example of how to so. Place this file in the hooks directory of the repository on your server, rename it to 'update' and make sure it is executable. It will then block these annoying merge commits while allowing through genuine merge commits from named branches.
Super repository
With repository mirrors created and all modules now available as git repositories, it is quite convenient to have a container repository to manage everything. This container can be implemented as a git super-repository with git submodules for each component. In this section, we describe how to setup and administer such a repository, both locally and using a central server.
Creating a super repository
Common tasks
Adding a submodule
Adding a new submodule is as simple as running:
git module add <url-to-repository> <path-to-submodule> <upstream-url> <upstream-type> <track-branch>
checking the changes look OK, committing and then pushing. The arguments <upstream-url>, <upstream-type> and <upstream-branch> specify the URL and type of the upstream version of this repository and the branch which the super-repository should track.
Removing a submodule
To remove a submodule from the super repository, use:
git module rm <submodule-path>
check the changes look OK, then commit and push.
Moving a submodule
To move the location of a submodule in the super repository:
git module mv <module-path> <new-module-path>
check the changes look OK, then commit and push.
Include here a guide to setting up a Git server with a Cactus super-repository
Have temporarily copy+pasted the README which is in note form - need to write a proper up-to-date guide.
###############################################################################
# CactusGit
###############################################################################
# Introduction
# This package allows you to check out Cactus using GetComponents from
# a CRL thornlist using git-svn instead of svn for the svn
# repositories. This means that your local Cactus tree contains
# complete version information and history and can be managed using
# git rather than svn.
# Further, the package provides tools for creating a
# "super-repository" for the tree with pointers to the downloaded
# repositories as git submodules. This means that the state of the
# tree can be identified by a single commit ID and can be cloned
# directly without having to use GetComponents or git-svn.
# Notes
# * To make this more convenient, small wrapper scripts can be written
# to replace the recipes given below
#
# * GetComponents could be extended to support git-svn directly,
# rather than using the wrapper-script approach that is currently
# used
# Tutorial
# Add our tools to the path, including the svn wrapper which tricks
# GetComponents into making a git-svn clone of each svn repository.
# We also include a copy of GetComponents here for convenience.
export OLDPATH=$PATH PATH=$PWD/bin:$PATH
# Check out the tree into the Cactus directory (can take a very long
# time, ~60 minutes, and does not show progress for each repository
# until it has checked it out completely)
GetComponents -v -a WaveToy.th
# Now manually checkout any thorns which require authentication
git svn clone ...
# Add the git-svn repos to your server (also add the Cactus super-repo).
crlrepos WaveToy.th > gitsvn.list
cd <gitolite_dir>
vi conf/gitolite.conf # Add the repositories in gitsvn.list
git commit conf/gitolite.conf
git push
# Add a 'fetching and pushing' (fp) user
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push
# Push as the fp user
scp Cactus fp@server:fetch/
scp gitsvn.list fp@server:fetch/
ssh fp@server
cd fetch
for i in `cat gitsvn.list`; do cd $i && git remote add bare cactus@localhost:$i && git config --unset remote.bare.fetch && git config remote.bare.push 'refs/remotes/*:refs/heads/*' && git push bare && cd -; done
# On the server, update the HEAD of the git-svn repos
ssh cactus@server
cd ~/repositories
for i in `cat ../fetch/gitsvn`; do cd $i.git && git symbolic-ref HEAD refs/heads/git-svn && cd -; done
# Generate a super-repository called CactusGit using all the
# repositories from Cactus listed in WaveToy.th
crltogit CactusGit cactus@server: Llama.CTGamma.ET.th
# Add non-svn repos
GetComponents Llama.CTGamma.ET.th
rm -rf repos
git submodule add ... repos/...
# Symlink flesh
ln -s flesh/CONTRIBUTORS
ln -s flesh/COPYRIGHT
ln -s flesh/Makefile
ln -s flesh/doc
ln -s flesh/lib
# Commit symlinks (as fp)
git add *
git commit ...
git push cactus@localhost:Cactus master:master
######### Administrators ########
# Updating the git-svn mirrors
for i in `cat gitsvn.list`; do cd $i; git svn fetch && git push bare; cd -; done
# Updating the hg-git mirrors
for i in `cat githg.list`; do cd $i; hg pull && hg push git; cd -; done
# Updating the git mirrors
for i in `cat git.list`; do cd $i; git fetch -a -u && git push bare --all; cd -; done
# Updating all submodules in the super-repo
cd Cactus
git pull
git submodule foreach git pull
git commit -m "Update submodules." -a
# Updating a single submodule in the super-repo
cd Cactus
cd <repo-path>
git pull
cd -
git commit -m "Update submodule <repo-path>." <repo-path>
# Automatic updates of super-repo
git clone --recursive cactus@localhost:Cactus
cd Cactus
git submodule foreach 'git remote set-url origin /var/cactus/repositories/$path cactus@git.barrywardell.net:/$path; echo OK'
git remote set-url origin /var/cactus/repositories/Cactus
git remote set-url --push origin cactus@git.barrywardell.net:Cactus
git checkout submodules
git pull origin submodules
git submodule foreach 'git fetch origin && git checkout origin'
git commit -m "Update submodules." -a
git push origin submodules
# Adding a git-svn repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push
ssh fp@server
cd fetch
vi gitsvn.list # add <newrepo>
git svn clone -s <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare
cd ~/repositories/<newrepo>
git symbolic-ref HEAD refs/heads/trunk
cd ~/fetch/<newrepo>
git pull bare
# Adding a hg-git repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push
ssh fp@server
cd fetch
vi githg.list # add <newrepo>
hg clone <url> <newrepo>
cd <newrepo>
vi .hg/hgrc # Add:
# [path]
# git = git+ssh://localhost/<newrepo>
# [git]
# intree = 1
hg gexport # Can take quite a long time
hg push git
# Adding a git mirror repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push
ssh fp@server
cd fetch
vi git.list # add <newrepo>
git clone --mirror <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git push bare --all
# Adding a submodule to the super-repo
git submodule add <url> <path>
git commit
git push
# Removing a submodule from the super-repo
vi .gitmodules
vi .git/config
git rm --cached <path>
git commit ...
git push
# Changing svn URL for a git-svn clone. WARNING: this is rewriting the history of
# the git-svn branch of your mirror and will mess peoples checkout up.
cd fetch/<repo>
git status
git gc
git filter-branch --msg-filter 'sed "s/git-svn-id: http/git-svn-id: https/g"' $(cat .git/packed-refs | awk '// {print $2}' | grep -v 'pack-refs')
sed -i -e "s/http/https/g" .git/config
rm -rf .git/svn
git svn rebase
git push bare --force
# Changing svn URL without rewriting history (https://git.wiki.kernel.org/index.php/GitSvnSwitch):
Edit the svn-remote url URL in .git/config to point to the new domain name
Run git svn fetch - This needs to fetch at least one new revision from svn!
Change svn-remote url back to the original url
Run git svn rebase -l to do a local rebase (with the changes that came in with the last fetch operation)
Change svn-remote url back to the new url
Run git svn rebase should now work again!
Creating a new super-repo branch
- git clone git://git.barrywardell.net/EinsteinToolkit
- git checkout -b ET_2013_11
- (edit .gitmodules)
- git module sync
- git module update
- git module commit
